Upload Failed - Caused by SSLError - SSLCertVerificationError - unable to get local issuer certificate

[KevWal]

Hi

I am using Horus Gui on my work laptop, that has an enforced proxy, and and I am getting an SSL upload error for Sondehub. Habhub is working as that is HTTP.

2022-10-27 14:53:29,656 DEBUG: Sondehub Amateur Uploader - Generated Packet: {'software_name': 'Horus-GUI', 'software_version': '0.3.9', 'uploader_callsign': 'G7PMO_Horus', 'uploader_position': [52.0, 0.0, 0.0], 'uploader_radio': 'Horus-GUI v0.3.9 ', 'uploader_antenna': '', 'time_received': '2022-10-27T13:53:29.656806Z', 'datetime': '2022-10-27T13:52:11.000000Z', 'payload_callsign': 'G7PMO-V2', 'frame': 463, 'lat': 52.32274627685547, 'lon': -0.7060413360595703, 'alt': 93, 'temp': 13, 'sats': 7, 'batt': 3.3529411764705883, 'snr': 0.6235790848731995, 'raw': '4D01CF010D340B7E4A514220BF34BF5D0000070DAB0110069E3F647BD2048B41', 'modulation': 'Horus Binary v2', 'baud_rate': 50, 'ascent_rate': 40.97, 'ext_temperature': -2508.2, 'ext_humidity': 63, 'ext_pressure': 3158.8}

2022-10-27 14:53:30,166 DEBUG: Sondehub Amateur Uploader - Pre-compression: 698 bytes, post: 428 bytes. 61.3 % compression ratio, in 0.0 s

2022-10-27 14:53:30,169 DEBUG: Starting new HTTPS connection (1): api.v2.sondehub.org:443

2022-10-27 14:53:30,296 ERROR: Sondehub Amateur Uploader - Upload Failed: HTTPSConnectionPool(host='api.v2.sondehub.org', port=443): Max retries exceeded with url: /amateur/telemetry (Caused by SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1091)')))

2022-10-27 14:53:30,502 DEBUG: Starting new HTTP connection (1): habitat.habhub.org:80

2022-10-27 14:53:30,746 DEBUG: http://habitat.habhub.org:80 "PUT /habitat/_design/payload_telemetry/_update/add_listener/79ffd1196c6400319526f08ff8a52f887ab229abd427f568b54d33e515056b1c HTTP/1.1" 201 2

2022-10-27 14:53:30,747 INFO: Habitat - Uploaded sentence: $$G7PMO-V2,463,13:52:11,52.32275,-0.70604,93,0,7,13,3.35,40.97,-2508.2,63,3158.8*4134

I assume this is an error due to that proxy, a normal browser works fine, but I do get issues with other programs sometimes and I have to input a manual proxy setting for them - but no option for Horus Gui?

Thanks very much
Kevin

[darksidelemm]

Ugh, this one is going to be a pain. I know handling proxies is possible (I've done it in radisonde_auto_rx), but it's not really something I thought would be required in Horus-GUI...

[KevWal]

Sorry! Thank you 😊! Any option for me setting sondehub to be HTTP as that doesn't seem to have the issue...

[darksidelemm]

I'm not even sure the SondeHub APIs respond on HTTP...

Also, if HTTP is working but HTTPS isn't, it's possible that the issue is not so much a proxy issue, but that your SSL connections are being man-in-the-middle'd by your work. In this case i'm not sure what the solution is for uploading, as dealing with that generally requires adding another root certificate somewhere.

I could run the requests with verify=False (which ignores SSL certificate errors), but I'm somewhat loathe to do that.

[KevWal]

Hi I'm not entirely sure what work are doing, I know that web browsers work fine to all but blocked sites, but other programs (like SkyDemon) that have a manual proxy entry setting need me to fill that in for them to.work. How about a run time command line flag option for disabling SSL verification? Thanks very much Kevin

On 29 October 2022 23:07:28 BST, Mark Jessop ***@***.***> wrote: I'm not even sure the SondeHub APIs respond on HTTP... Also, if HTTP is working but HTTPS isn't, it's possible that the issue is not so much a proxy issue, but that your SSL connections are being man-in-the-middle'd by your work. In this case i'm not sure what the solution is for uploading, as dealing with that generally requires adding another root certificate somewhere. I *could* run the requests with verify=False (which ignores SSL certificate errors), but I'm somewhat loathe to do that. -- Reply to this email directly or view it on GitHub: #27 (comment) You are receiving this because you authored the thread. Message ID: ***@***.***>

-- Sent from my Android device. Please excuse my brevity.

[darksidelemm]

Honestly, disabling SSL checking is not really something I want to add or encourage...

[darksidelemm]

It may be possible to get around this by installing your works proxy SSL root certificate (it sounds like this has been done for your web browser, else you would be getting continuous SSL warnings) in some location that the python requests library will find it, but I'm not entirely sure how to do this.

[darksidelemm]

Otherwise, the only way to handle this is to add verify=false into the following places in horusdemodlib:
https://github.com/projecthorus/horusdemodlib/blob/master/horusdemodlib/sondehubamateur.py#L313
and
https://github.com/projecthorus/horusdemodlib/blob/master/horusdemodlib/sondehubamateur.py#L386

You would then need to re-build the horus-gui binary against this modified version of horusdemodlib.

Disabling any form of SSL verification is not something I want to add into horusdemodlib, even as add an option.

[KevWal]

Thank-you, I'll give these options a go and come back - away for work this week but I will come back here.

[KevWal]

So I can recreate the error in 'raw' python:

Python 3.11.0 (main, Oct 24 2022, 18:26:48) [MSC v.1933 64 bit (AMD64)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> import requests
>>> requests.get('https://api.v2.sondehub.org/')
requests.exceptions.SSLError: HTTPSConnectionPool(host='api.v2.sondehub.org', port=443): Max retries exceeded with url: / (Caused by SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:992)')))

and apparently as Python doesnt use the Windows certificate system I need to give it the certificate chain through a pem file like this:

export REQUESTS_CA_BUNDLE="/usr/local/myproxy_info/cacert.pem"

But I can not find any way of getting the certificate chain into a single pem file for python :(

[KevWal]

I get:

set REQUESTS_CA_BUNDLE="C:\Users\xxx\Downloads\forcepoint-cloud-ops-ca-chain.pem"

C:\Users\xxx\Downloads>python
Python 3.11.0 (main, Oct 24 2022, 18:26:48) [MSC v.1933 64 bit (AMD64)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> import requests
>>> requests.get('https://api.v2.sondehub.org/')
OSError: Could not find a suitable TLS CA certificate bundle, invalid path: "C:\Users\xxx\Downloads\forcepoint-cloud-ops-ca-chain.pem

even though:

C:\Users\xxx\Downloads>dir C:\Users\xxx\Downloads\forcepoint-cloud-ops-ca-chain.pem
 Volume in drive C is System
 Volume Serial Number is 4E00-ABEA

 Directory of C:\Users\xxx\Downloads

30/10/2022  17:41             4,268 forcepoint-cloud-ops-ca-chain.pem
               1 File(s)          4,268 bytes
               0 Dir(s)  120,299,184,128 bytes free

[KevWal]

Small steps forwards, I found the default key file:

\Downloads>python -c "import certifi; print(certifi.where())"
C:\Program Files\Python311\Lib\site-packages\certifi\cacert.pem

edited it in notepad and added in the chain from the proxy key (which I exported from firefox):

and now I get no errors here:

Doesn't work in Horus gui yet, but getting closer...

[KevWal]

and working:

20:44:03 [INFO]  Habitat - Uploaded sentence: $$G7PMO-V2,1574,20:42:16,52.32269,-0.70620,65,0,4,22,5.00,40.97,-2508.2,63,3158.8*9629
20:44:05 [INFO]  Sondehub Amateur Uploader - Uploaded 1 telemetry packets to Sondehub Amateur in 1.0 seconds.

I'll do a clean set of instructions in a single post.

[KevWal]

Install Python, install requests module, find the default python certificate file:

C:\Users\xxx>python -c "import certifi; print(certifi.where())"
C:\Program Files\Python311\Lib\site-packages\certifi\cacert.pem

Install Firefox, browse to https://api.v2.sondehub.org/, find the name of the furthest right Certificate used:

In Firefox click Settings, Privacy & Security, Certificates, click View Certificates:

Find the certificate that matches the furthest right Certificate used:

and click Download PEM (chain):

Open the file it downloads in Notepad and copy the certificates from that file. Open the default Python certificate file (that you found above) in Notepad and paste your additional certificates at the beginning of that file.

Run a command prompt and tell horus gui where to find the certificate file:

C:\Apps\Horus-Gui\>set REQUESTS_CA_BUNDLE=C:\Program Files\Python311\Lib\site-packages\certifi\cacert.pem
C:\Apps\Horus-Gui\>horus-gui.exe

It is important to note the set command has no single nor double quotes, they break it!

Assuming that works, you can put the above in a .bat file so it is set every time!

Thanks very much
Kev

[KevWal]

Confirmed this is still working, so closing. It is probably a suitable rare issue just to let google direct people here, unless you want a line adding to the end of the Readme?