Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Things get weird with 115200 baud #1

Open
CoolNamesAllTaken opened this issue Mar 25, 2019 · 16 comments
Open

Things get weird with 115200 baud #1

CoolNamesAllTaken opened this issue Mar 25, 2019 · 16 comments

Comments

@CoolNamesAllTaken
Copy link
Collaborator

Commands work fine at 9600 baud, but something goes wonky at 115200 baud (none of the responses are received, and all I get is timeouts). My guess is that since the send and receive commands execute sequentially, the slight delay between send and receive at the higher baud rate causes something to get missed. It might be worth rearranging things slightly such that the script starts reading the serial port before the command is sent.

Thanks for making this script! Super helpful as both a tool and a reference :)

@CoolNamesAllTaken
Copy link
Collaborator Author

CoolNamesAllTaken commented Mar 25, 2019

Things started going south for me on 9600 baud as well, but this fix did the trick. Things work great on 115200 baud now, too. I think that python stops monitoring the serial port once you exit a "with" statement, so things were getting lost in the gap between "send" and "watch".

def watch_and_send(cmd, timeout=10, success=None, failure=None, echo_cmd=None):
    with serial.Serial(PORT, BAUD, timeout=1) as ser:
        ser.write(cmd.encode('utf-8') + CMD_LINEBREAK)
        t_start = time.time()
        reply = list()
        while True:
            if ser.in_waiting:
                line = ser.readline()
                echo = False
                if echo_cmd:
                    echo = line.decode('utf-8').strip().endswith(echo_cmd)
                if line != CMD_LINEBREAK and not echo:
                    line = line.decode('utf-8').strip()
                    reply.append('\t' + line)
                    if success and line.startswith(success):
                        return ("Success", reply, time.time()-t_start)
                    if failure and line.startswith(failure):
                        return ("Error", reply, time.time()-t_start)
            if (time.time()-t_start) > timeout:
                return ("Timeout", reply, time.time()-t_start)
            time.sleep(0.02)

@CoolNamesAllTaken
Copy link
Collaborator Author

I'm going to keep this thread going with some more things I've run into:

  • Restart function seems to crash things, at least for me (SIM7000A running 1351B04SIM7000A firmware).
  • I had to add a time.sleep(5) after every certificate load operation for things to work properly, at least at 115200 baud. It seems like the modem was choking on too many certificates being uploaded with no pause in between, and was returning "+CME Error: operation not permitted". Once the delay was added after each certificate upload, things worked fine.
  • The last command in the initialization sequence was looking at the wrong index in a reply string: cgcontdrp[1][0] should be cgcontdrp[1][1], since cgcontdrp[1][0] is the echo reply and contains no commas (was causing an issue with the split(",") command). This might not have been an issue before if replies were being missed on the serial port, but after implementing watch_and_send it began to act up.
  • I noticed that the SSL key conversion step worked with your key/certificate pair, but not the one that I generated. I dumped the output of our key files, and they seem identical except that yours starts with "BEGIN RSA PRIVATE KEY" and mine begins with "BEGIN PRIVATE KEY". As far as I can tell, both of our keys are rsa keys. Do you mind sharing what command you used to generate your keys? I used:
openssl req -newkey rsa:2048 -nodes -keyout key.pem -x509 -days 365 -out certificate.pem

@tmcadam
Copy link
Owner

tmcadam commented Mar 25, 2019

@CoolNamesAllTaken I followed the guide here on test.mosquitto.org to generate the certs and key.

@CoolNamesAllTaken
Copy link
Collaborator Author

CoolNamesAllTaken commented Mar 25, 2019

Ha, just saw your comment above. Will leave this here in case it's helpful to anyone.

Figured out the key problem. I was generating the wrong format of RSA Key, and then using that key as a certificate instead of using it to create a certificate request. The correct command sequence is below.

openssl genrsa -out azure.key
openssl req -out azure.csr -key azure.key -new

Then, use azure.csr with the mosquitto guide linked in the post above to get an actual client certificate.

Apologies for the spam--I'm picking this up for the first time, and hoping that if I generate some sort of documentation people might have a better time down the line. I'd be happy to do a PR with my code changes and maybe some readme improvements if you'd like @tmcadam .

@tmcadam
Copy link
Owner

tmcadam commented Mar 25, 2019

Yes please @CoolNamesAllTaken

@CoolNamesAllTaken
Copy link
Collaborator Author

CoolNamesAllTaken commented Mar 25, 2019

I'm still working through the code changes, but seem to have connections working to some degree for MQTT with only a remote server certificate (I put mqtt-bothcerts on the backburner for now since Azure doesn't actually have a nice key signing tool like mosquitto as far as I can tell).

@tmcadam dumb question: what is the <cid> that is referred to in all of the SIMCOMM manuals? The manual defines it as:

(PDP Context Identifier) a numeric parameter which
specifies a particular PDP context definition. The parameter
is local to the TE-MT interface and is used in other PDP
context-related commands. The range of permitted values
(minimum value=1) is returned by the test form of the
command.
 1…24 

But I don't really understand what it's saying. My best guess is that it's a connection ID value used for multiplexed connections with multiple IPs, but I'm not sure. So far I've set it to 0 or 1 and things work (as long as all the <cid> values are the same), but I'd like to understand what it's actually for.

@tmcadam
Copy link
Owner

tmcadam commented Mar 25, 2019

I'm not really sure to be honest. I guess you could test if you get a second IP using another index while having the first connected. I guess not all carriers would support that as well.

@tmcadam
Copy link
Owner

tmcadam commented Mar 25, 2019

I've found the SIMCOMM docs pretty useless and out of step with firmware updates on occasions. It's painful working with AT as well with no meaningful feed back from commands. Just all trial and error.

@CoolNamesAllTaken
Copy link
Collaborator Author

Yeah it's unfortunate how bad the documentation is. The worst part is that there seems to be 3 different ways of doing things for every command, and using any mix of the two methods causes things to break. I'm trying to integrate with the TinyGSM library on Arduino right now and things are blowing up royally.

Would you like me to fork your repo to add changes, or would you like to give me write access? I have a branch that I'm working on at dev/jm/mojo that I can push if you'd like to see any of my changes so far.

@tmcadam
Copy link
Owner

tmcadam commented Mar 26, 2019

@CoolNamesAllTaken Added you as collaborator, you should be able to push. Be aware our modules are on fairly different firmware versions (I'm Sim7000e @ B07), but we can add that as flag in the command line and handle the variations in the script if we need to.

@captFuture
Copy link

Yeah it's unfortunate how bad the documentation is. The worst part is that there seems to be 3 different ways of doing things for every command, and using any mix of the two methods causes things to break. I'm trying to integrate with the TinyGSM library on Arduino right now and things are blowing up royally.

I am trying to do this as well (based on an esp32 and the sim7000 connected) the initial setup in the current master of TinyGSM is a merge from my fork.
-> locally i have the most examples of TinyGSM running (but everything where certificates are used - even the basic https get example are not working at all)

Compared to the SIM800 stuff the AT+CIPSSL command is not present on the SIM7000 which would simply set TCP to use SSL function.

@CoolNamesAllTaken
Copy link
Collaborator Author

@captFuture One of the things that was confusing me about the TinyGSM code was that it was doing a very complicated setup for an "IP Bearer" configuration. It is my impression that a TLS connection is referred to in the AT manual as an "App", which uses an entirely different set of AT commands. For instance, AT+CAOPEN instead of AT+CIPSTART. I'm currently rewriting the Sim7000 TinyGSM code to use SSL only, and consequently use the AT+CA___ type commands. This corresponds to the set of commands that seems to work fine in @tmcadam's python script for mqtt over ssl with a ca cert only.

Is my understanding correct?

@JoshYang1977
Copy link

Thank very much for your thread. It answers my questions. Appreciate your experience-sharing.

@gduan2000
Copy link

thanks for the post here, it helped a lot. the code works for me except for the SSL part. my goal is to be able to connect to https://httpbin.org with AT+SHCONN. I downloaded the root cert from https://httpbin.org, named it "httpbin-ca.cer". It looks like:

-----BEGIN CERTIFICATE-----
MIIDQTCCAimgAwIBAgITBmyfz5m/jAo54vB4ikPmljZbyjANBgkqhkiG9w0BAQsF
ADA5MQswCQYDVQQGEwJVUzEPMA0GA1UEChMGQW1hem9uMRkwFwYDVQQDExBBbWF6
b24gUm9vdCBDQSAxMB4XDTE1MDUyNjAwMDAwMFoXDTM4MDExNzAwMDAwMFowOTEL
MAkGA1UEBhMCVVMxDzANBgNVBAoTBkFtYXpvbjEZMBcGA1UEAxMQQW1hem9uIFJv
b3QgQ0EgMTCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBALJ4gHHKeNXj
ca9HgFB0fW7Y14h29Jlo91ghYPl0hAEvrAIthtOgQ3pOsqTQNroBvo3bSMgHFzZM
9O6II8c+6zf1tRn4SWiw3te5djgdYZ6k/oI2peVKVuRF4fn9tBb6dNqcmzU5L/qw
IFAGbHrQgLKm+a/sRxmPUDgH3KKHOVj4utWp+UhnMJbulHheb4mjUcAwhmahRWa6
VOujw5H5SNz/0egwLX0tdHA114gk957EWW67c4cX8jJGKLhD+rcdqsq08p8kDi1L
93FcXmn/6pUCyziKrlA4b9v7LWIbxcceVOF34GfID5yHI9Y/QCB/IIDEgEw+OyQm
jgSubJrIqg0CAwEAAaNCMEAwDwYDVR0TAQH/BAUwAwEB/zAOBgNVHQ8BAf8EBAMC
AYYwHQYDVR0OBBYEFIQYzIU07LwMlJQuCFmcx7IQTgoIMA0GCSqGSIb3DQEBCwUA
A4IBAQCY8jdaQZChGsV2USggNiMOruYou6r4lK5IpDB/G/wkjUu0yKGX9rbxenDI
U5PMCCjjmCXPI6T53iHTfIUJrU6adTrCC2qJeHZERxhlbI1Bjjt/msv0tadQ1wUs
N+gDS63pYaACbvXy8MWy7Vu33PqUXHeeE6V/Uq2V8viTO96LXFvKWlJbYK8U90vv
o/ufQJVtMVT8QtPHRh8jrdkPSHCa2XV4cdFyQzR1bldZwgJcJmApzyMZFo6IQ6XU
5MsI+yMRQ+hDKXJioaldXgjUkK642M4UwtBV8ob2xJNDd2ZhwLnoQdeXeGADbkpy
rqXRfboQnoZsG4q5WTP468SQvvG5
-----END CERTIFICATE-----

Then I used python sim7000.py cert-load to upload the cert to the SIM7000A module, no issue. I am able to check it with QPST EFS explorer.

Finally I ran the mqtt-cacert function (updated some commands) and getting the following response, I tried running it through putty, SSCOMV5.13.1, I tried different cert, different sites. It gets ERROR on AT+SHCONN no matter what. am I missing some critical step? No issue with HTTP calls.

I appreciate anybody's help here, if you have a working root cert and matching site that you know is working, please share it. at this point, I don't know if this is a defective module. The firmware shows Revision:1351B04SIM7000A. thanks a lot!

...
...
...
++++++++++++++++++++ MQTT - CA Cert Only +++++++++++++++++++++

----------- AT+CNACT=1 -----------
Error (0.04secs):
AT+CNACT=1
+CME ERROR: operation not allowed

----------- AT+CNACT? -----------
Success (0.08secs):
AT+CNACT?
+CNACT: 1,"10.99.3.170"
OK

----------- AT+SHCONF="URL","https://httpbin.org" -----------
Success (0.05secs):
AT+SHCONF="URL","https://httpbin.org"
OK

----------- AT+CSSLCFG="sslversion",1,3 -----------
Success (0.04secs):
AT+CSSLCFG="sslversion",1,3
OK

----------- AT+CSSLCFG="convert",2,"httpbin-ca.cer" -----------
Success (0.03secs):
AT+CSSLCFG="convert",2,"httpbin-ca.cer"
OK

----------- AT+SHSSL=1, "httpbin-ca.cer" -----------
Success (0.04secs):
AT+SHSSL=1, "httpbin-ca.cer"
OK

----------- AT+SHSSL? -----------
Success (0.08secs):
AT+SHSSL?
+SHSSL: 1,"httpbin-ca.cer",""
OK

----------- AT+SHSTATE? -----------
Success (0.09secs):
AT+SHSTATE?
+SHSTATE: 0
OK

----------- AT+SHCONN -----------
Error (0.04secs):
AT+SHCONN
+CME ERROR: operation not allowed

...

@alibahba
Copy link

alibahba commented Apr 9, 2021

You can try:
AT+SHSSL=1,""
This will make a connection on HTTPS but don't validate the certificate.
I have the same problem and an outstanding question about this by SIMCOM.
My modem is a SIM7000G on firmware 1529B05SIM7000G.

Some of my examples:

AT+SHSSL=1,""
OK
AT+SHCONF="URL","https://google.nl"
OK
AT+SHCONF="BODYLEN",1024
OK
AT+SHCONF="HEADERLEN",350
OK
AT+SHCONN
OK

AT+SHSSL=1,""
OK
AT+SHCONF="URL","https://nu.nl"
OK
AT+SHCONF="BODYLEN",1024
OK
AT+SHCONF="HEADERLEN",350
OK
AT+SHCONN
ERROR

AT+SHSSL=1,""
OK
AT+SHCONF="URL","https://myqtthub.com"
OK
AT+SHCONF="BODYLEN",1024
OK
AT+SHCONF="HEADERLEN",350
OK
AT+SHCONN
OK

AT+CSSLCFG=convert,2,ca.crt
OK
AT+SHSSL=1,"ca.crt"
OK
AT+SHCONF="URL","https://myqtthub.com"
OK
AT+SHCONF="BODYLEN",1024
OK
AT+SHCONF="HEADERLEN",350
OK
AT+SHCONN
ERROR

@cshunt
Copy link

cshunt commented Feb 14, 2023

I had a similar issue when running code at 115200 and even some at 9600 baud. The beginning of strings were missing.
I believe same as @CoolNamesAllTaken second post on this, when "with" is used, it re-initiates the serial port and takes a moment and potentially drops some characters from the incoming stream.
One potential fix is below, the send and receive commands are initiated with the same "with" serial command. Another option would be to move the serial setup as part of a first run command.

Also note that I was able to remove all of the sleep commands and received data came through as expected!!

Thank you for sharing your code, I have adopted some to SIM7080, the AT command and return does work well.

#def send(data):
#    with serial.Serial(PORT, BAUD, timeout=1) as ser:
#        ser.write(data)

#def send_cmd(cmd):
#    send(cmd.encode('utf-8') + CMD_LINEBREAK)

#def send_cmd(cmd):
#    with serial.Serial(PORT, BAUD, timeout=1) as ser:
#        ser.write((cmd.encode('utf-8') + CMD_LINEBREAK))

def send_cmd(cmd, timeout=10, success=None, failure=None, echo_cmd=None):
#    with serial.Serial(PORT, BAUD, timeout=1) as ser:
#       ser.write((cmd.encode('utf-8') + CMD_LINEBREAK))

#def watch(timeout=10, success=None, failure=None, echo_cmd=None):
    with serial.Serial(PORT, BAUD, timeout=1) as ser:
        t_start = time.time()
        ser.write((cmd.encode('utf-8') + CMD_LINEBREAK))
        reply = list()
        while True:
            #time.sleep(.2)
            if ser.in_waiting:
                line = ser.readline()
                echo = False
                if echo_cmd:
                    echo = line.decode('utf-8').strip().endswith(echo_cmd)
                if line != CMD_LINEBREAK and not echo:
                    if line != "\r\n":
                        line = line.decode('utf-8').strip()
                        reply.append('\t' + line)
                    if success and line.startswith(success):
                        return ("Success", reply, time.time()-t_start)
                    if failure and line.startswith(failure):
                        return ("Error", reply, time.time()-t_start)
            if (time.time()-t_start) > timeout:
                return ("Timeout", reply, time.time()-t_start)
            #time.sleep(0.1)

def AT(cmd="", timeout=10, success="OK", failure="+CME ERROR"):
    cmd = 'AT' + cmd
    print("----------- ", cmd, " -----------")
    #send_cmd(cmd)
    #reply = watch(echo_cmd=cmd, timeout=timeout, success=success, failure=failure)
    reply = send_cmd(cmd, echo_cmd=cmd, timeout=timeout, success=success, failure=failure)
    print("{0} ({1:.2f}secs):".format(reply[0], reply[2]))
    print(*reply[1], sep='\n')
    print('')
    return reply

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

7 participants