Acquire and add SSL certificates to SIM7000 memory

[JGLJGL]

Nedd to add SSL certificates to SIM7000.
I want to access to https://www.atesens.com, how can I get and add the certificates to the device memory ?

[botletics]

You can add certificates to the SIM7000's EFS memory by using the EFS Explorer tool after installing the QPST tool from here.

[RobertMouncer]

@botletics Where can we download the tool from? That link doesn't exist anymore. Many thanks.

[botletics]

I just tried clicking on the link and it works.

[RobertMouncer]

@botletics Copy link address -> "https://www.dropbox.com/home/SIMCom%20Firmware%20Update%20Files" this isn't a sharing link - this is just your link to your dropbox. Please right click on the folder/file and click share to get a link :)
Thanks!

[botletics]

Oh sorry, my bad. You can actually find the correct link on the firmware update wiki or go there directly

[JGLJGL]

Ok thanks, I allready have EFS explorer. But how can I get the certificates needed?

[botletics]

In Chrome there's a little padlock icon next to the URL. Click that and view the certificate.

[JGLJGL]

This is what I get in the QPST EFS Explorer, the icons aren't shown.

[botletics]

Make sure your SIM7000 module is powered on with a LiPo battery and connected via USB (make sure the USB cable you're using has data wires, not just power).

[JGLJGL]

I have the battery and also the right USB, I get this error:
This port works fine with other applications like Realterm, AT Command Tester,...

[botletics]

Make sure all the USB drivers are installed from the device manager. I just tried with a random board and it detects it just fine.

[JGLJGL]

Ok, now it detects the port, I was trying to use the AT port instead of Diagnostics port

[JGLJGL]

Now the problem is, how do I add the certificates? I supposed something had to be in "customer" folder.

[botletics]

With the folder selected (like you have it in the screenshot above), right click on the right side and click "New" then "File..."

[JGLJGL]

Ok, now I have the certificates, but in .cer format, how could I convert them to .crt format ? I use AT+CSSLCFG="convert",2,"atesens.cer" but gives ERROR.

[botletics]

Try saving it as "atesens.crt" to begin with.

[sethivansh6]

Try saving it as "atesens.crt" to begin with.

@botletics Hey I tried all the things you stated in sim7000 issues and now I have copied .crt files to flash memory of SIM7000E and I followed the AT COMMANDS as yours in AT commands log 1 and 2. I tried all this on PUTTY. But while following those AT LOGS my Sim7000 got hang on AT+SHCONN for like 10-15 minutes or most of the times it gave an error.

Once it showed OK pr then it showed AT + SHSTATE SHSTATE 0 ......which should be 1 for making an https request right.

Can you tell me the probable reason for all this and any solution if possible.

[botletics]

Try these AT commands which worked for a user.

[jugkrit]

Ok, now it detects the port, I was trying to use the AT port instead of Diagnostics port

Hi, How did you change port from Diagnostics port to AT port in QPST?

[stbrie]

Hi, How did you change port from Diagnostics port to AT port in QPST?

Did you update all of the device drivers in device manager that are associated with the sim7000?

[botletics]

Hey guys, figured out how to connect with SSL without verifying certs! Please do the following:

• Set "SSL_FONA" to 1 in the .h file
• Download the latest .cpp file from GitHub
• Open the unedited LTE_Demo example sketch and change "http://dweet.io" to "https://dweet.io" on line 1035 (under the '2' option)
• Upload to the Arduino
• Run the 'G' command to enable data connection, then run '2' to send data to dweet.io using SSL. And voila! 😊

[baconcheese113]

@botletics can you point me to the commands you're using to connect without uploading client certs?

I'm unable to get past AT+SHCONN with HTTPS, works fine with HTTP

Here are my logs

AT
OK
AT+CFSINIT
OK
AT+CFSWFILE=3,"httpbin_root_ca.cer",0,1077,10000
DOWNLOAD

OK
AT+CFSTERM
OK
AT+CSSLCFG="convert",2,"httpbin_root_ca.cer"
OK
at+csslcfg="sslversion",1,3
OK
AT+SHSSL=1,"httpbin_root_ca.cer"
OK
AT+SHCONF="URL","https://httpbin.org"
OK
AT+SHCONF="BODYLEN",1024
OK
AT+SHCONF="HEADERLEN",350
OK
AT+SHCONN
ERROR

[botletics]

So the code has recently migrated to another repo for Arduino Library Manager compatibility (you can install the Botletics SIM7000 library directly from the library manager now). Set BOTLETICS_SSL to 1 in the .h file, then in the LTE_Demo change the HTTP_connect() URL to "https://", then comment out the lines like this (line 1026):

// #ifdef SIMCOM_7070 // Use this line if you have the SIM7000G because the 1529B01SIM7000G firmware doesn't seem to run the commands below well
#if defined(SIMCOM_7000) || defined(SIMCOM_7070) // Use this if you have SIM7000A, especially with SSL

and then run the 'G' command, followed by the '2' command. You will see it use AT+CNACT to enable data, then use AT+SHSSL=1,"" without using CFS to load the cert.

BUT I will say that for some reason it seems to work on the SIM7000A and SIM7070G, but not on the SIM7000G... Seems like a firmware thing because on SIM7000G, even AT+SHCONF="BODYLEN",1024 throws me an error. If you can figure it out on yours, let me know!

[baconcheese113]

I actually got HTTPS working on the SIM7000G last night!

; save this file as 'conn.ttl' or something and run it from tera term Control -> Macro
timeout = 5

sendln 'AT'
wait 'OK'
sendln 'AT+CCLK?'
wait 'OK'
sendln 'AT+CCLK="22/12/01,00:01:00-12"'
wait 'OK'
sendln 'AT+CNACT=1,"hologram"'
wait 'OK'
wait '+APP PDP'
sendln 'AT+CSSLCFG="sslversion",1,3'
wait 'OK'
sendln 'AT+SHSSL=1,""'
wait 'OK'
sendln 'AT+SHCONF="BODYLEN",1024'
wait 'OK'
sendln 'AT+SHCONF="HEADERLEN",350'
wait 'OK'
sendln 'AT+SHCONF="URL","https://httpbin.org"'
wait 'OK'
sendln 'AT+SHCONN'
wait 'OK' 'ERROR'
if result=2 then
    goto ERROR
endif
sendln 'AT+SHSTATE?'
wait 'OK'
sendln 'AT+SHCHEAD'
wait 'OK'
sendln 'AT+SHAHEAD="User-Agent","curl/7.47.0"'
wait 'OK'
sendln 'AT+SHAHEAD="Cache-control", "no-cache"'
wait 'OK'
sendln 'AT+SHAHEAD="Connection", "keep- alive"'
wait 'OK'
sendln 'AT+SHAHEAD="Accept", "*/*"'
wait 'OK'
sendln 'AT+SHREQ="/get?user=jack&password=123", 1'
wait 'OK'
wait '+SHREQ'
sendln 'AT+SHREAD=0, 300'
wait 'OK'
wait '+SHREAD'
sendln 'AT+SHDISC'
wait 'OK'
goto ERROR

:ERROR
    sendln 'AT+CNACT=0'
    wait 'OK'

The main issue is that the modem clock defaults to year 2080 and I believe this is used with verifying the server certs
You can do AT+CCLK? to check the time and `AT+CCLK="......" to set it, see this for more info.

I also finally heard back from SIMCOM about the firmware situation on the SIM7000G, it has different versions than the other variants. You can see the release notes here 1529SIM7000G Firmware Release Note.pdf I'm currently using 1529B06SIM7000G

The issue remaining is that I'm unable to connect to specific websites, including my own backend. I'm struggling to figure out why some sites can be connected to and others can't. Here's my kitchen sink macro file, you'll need to rename the extension to .ttl. You should be able to run it immediately after a cold boot.
conn.txt

[baconcheese113]

So even after setting the time, around 10% of sites would fail to connect through HTTPS, this is due to the servers hosting those domains having multiple certificates and/or the default certificate not being correct for the requested domain.

I recommend this video to understand the issue

To solve it you have to enable SNI for the domain you're connecting to before calling AT+SHCONN with this command

AT+CSSLCFG="sni",1,"example.com"

[baconcheese113]

If you don't care to check the time validity of the cert it can be ignored with

AT+CSSLCFG="ignorertctime",1,0

With this command it's no longer necessary to set the time with AT+CCLK=...

[Mark-Wills]

Guys, This is some really great information you are laying out here. Firstly, I want to thank you for solving this, and sharing the results with the community. Secondly, I feel like this should be written up somewhere because this issue will come up again and again. I wonder if someone can write a small program that demonstrates connecting to a web site using https and place it in a repo somewhere? I'll be sure to subscribe and star your repo. :-) With thanks Mark

…

On Thu, Dec 1, 2022 at 11:08 PM baconcheese113 ***@***.***> wrote: If you don't care to check the time validity of the cert it can be ignored with AT+CSSLCFG="ignorertctime",1,0 With this command it's no longer necessary to set the time with AT+CCLK=... — Reply to this email directly, view it on GitHub <#156 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AFAGDCXFV6ZP4VUHU7FAYW3WLEVXRANCNFSM4KMQDRFA> . You are receiving this because you are subscribed to this thread.Message ID: ***@***.***>

[baconcheese113]

@Mark-Wills How's this https://gist.github.com/baconcheese113/1f0264727fce3fa51a5bb06fa031aed2

[botletics]

I'll include this link in the wiki, thank you!

[Mark-Wills]

This looks fantastic! Only one question at the moment - which modem(s) does this sequence apply to? I couldn't see a reference in the text. Starred :-) Mark

…

On Sat, Dec 3, 2022 at 12:41 AM baconcheese113 ***@***.***> wrote: @Mark-Wills <https://github.com/Mark-Wills> How's this https://gist.github.com/baconcheese113/1f0264727fce3fa51a5bb06fa031aed2 — Reply to this email directly, view it on GitHub <#156 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AFAGDCRPVU7NQCD33YSQCXDWLKJM3ANCNFSM4KMQDRFA> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[baconcheese113]

@Mark-Wills I only have the SIM7000G to test with, and it should definitely work with all regional variants of the SIM7000. I'd also bet that it works for all SIM7xxxx modems, but I'm much less confident in that. I'm hoping that users with other SIMCOM modems will provide feedback and let me know!

[Mark-Wills]

Okay great - I've literally just ordered the SIM7000E board from the Botletics website (I'm in the UK) so we can perform some tests here in the UK using your technique. Thanks once again for your efforts on everyone's behalf with this issue. I've been on this mailing list for over a year, and have not ordered a board - not because there is anything wrong with it - on the contrary, it's a fantastic little design. My issue was that there didn't seem to be a solution for secure HTTP comms, either due to modem firmware issues, or just the really poor standard of the SIMCOM documentation. None was really sure. The work done here on this list is a tremendous step forward. Thanks again. Mark

…

On Sun, Dec 4, 2022 at 9:31 PM baconcheese113 ***@***.***> wrote: @Mark-Wills <https://github.com/Mark-Wills> I only have the SIM7000G to test with, and it should definitely work with all regional variants of the SIM7000. I'd also bet that it works for all SIM7xxxx modems, but I'm much less confident in that. I'm hoping that users with other SIMCOM modems will provide feedback and let me know! — Reply to this email directly, view it on GitHub <#156 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AFAGDCQG2WWXWHPKGDJXM5TWLUETJANCNFSM4KMQDRFA> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[jatayu86]

@baconcheese113 I have been trying secure comms through SIM7000C ,but in vain. Unfortunately there are some modem firmware issues I guess in their latest 1351B05SIM7000 firmware. AT+SHCONF="BODYLEN",1024 and HEADERLEN throws error, it cannot go past 60 in size. Maybe for that reason and due to incomplete header info ,my SHCONN is always ERROR. If anyone have figured a way out of this for SIM7000C, I will buy you a drink ! ;)

[baconcheese113]

@jatayu86 Not sure if it'd be a firmware issue, but can you try resetting the modem with ATZ and then following through the steps I outlined in my gist? It'd help to know exactly where your output starts to deviate. Also, you should try sending a plain HTTP request without specifying certificate data.

[jatayu86]

@baconcheese113 Plain http requests and mqtt works fine. Here are the steps as followed from the gist for https for SIm7000C Rev 1351B05SIM7000 (latest sent by simcom)

AT+CNACT=1,"airtelgprs.com"
<--- OK
<--- +APP PDP: ACTIVE

AT+CSSLCFG="sslversion",1,3
OK

AT+CSSLCFG="sni",1,"httpbin.org"
+CME ERROR: operation not allowed

AT+SHSSL=1,""
OK

AT+SHCONF="BODYLEN",1024
+CME ERROR: operation not allowed

AT+SHCONF="BODYLEN",64
OK

AT+SHCONF="HEADERLEN",350
+CME ERROR: operation not allowed

AT+SHCONF="HEADERLEN",64
OK

AT+SHCONF="URL","https://www.httpbin.org"
OK

AT+SHCONN
OK

AT+SHCHEAD
OK

AT+SHAHEAD="Content-type","application/json"
OK

AT+SHAHEAD="Connection","keep-alive"
+CME ERROR: operation not allowed. //going beyond headerlen configured

AT+SHBOD="{"query":"query getMySensors{hubViewer{sensors{serial}}}","variables":{}}",73
+CME ERROR: operation not allowed. //going beyond bodylen configured

AT+SHBOD="SIMCOMBODY",10
OK

AT+SHREQ="/",1
+CME ERROR: operation not allowed - Somehow my firmware doesn't support giving more than one argument for SMREQ.

AT+SHREQ=1
OK
+SHREQ: "GET",400,122. (probably because headers are incomplete)

AT+SHREAD=0,122

<title>400 Bad Request</title>

400 Bad Request

This seems consistent to the findings @botletics had as per below
#156 (comment)

https://github.com/botletics/SIM7000-LTE-Shield/blob/master/SIM7000%20Documentation/AT%20Command%20Logs/SIM7000_SSL_Log_1.txt

[Thusaraka2000]

AT+CFSINIT

ERROR
Command error..

Get the size of the file Device.crt in custapp

AT+CFSGFIS=0,"Device.crt"

+CFSGFIS: 1224

OK
AT+CFSRFILE=0,"Device.crt",0,1224,10000
Free the data buffer

+CFSRFILE: 1224
-----BEGIN CERTIFICATE-----
MIIDWjCCAkKgAwIBAgIVAMPrDSrvoyofjTSWegB09RhO/TLMA0GCSqGSIb3DQEB
CwUAME0xSzBJBgNVBAskFtYXpvbiXZWIgU2VydmljZXMgTz1BbWF6b24uY29t
IEluYy4gTD1TZW0dGxlIFNUPdhc2hpbmd0b24gQz1VUzAeFw0yNDA0MTAxMTQ
-----EN

OK

i uploaded the ssl certificate using the AT command tester i was able to load the Device .crt file using FS at commands(it loads sucessfully and shows me that the file exsist inside the custapp folder), but the problem is i am not able to config the ssl certificate and the following error occurs(for security purpose i have removed some parts of my ssl certicates in the comment).can you please help me with this?

AT+CSSLCFG="convert",2,"Device.crt"

ERROR
Command error..