Async operations gets a certificate error (while sync operation works fine)

[obrigg]

Python version installed
3.10

Meraki library version installed
meraki==1.37.2

Have you reproduced the issue with the latest version of this library? And with the latest version of Python?
Yes

OS Platform
MacOS 13.5.2

Describe the bug
On some computers (3 people reported this so far - two of them are Cisco folks with standard Cisco IT Macs.), the async operation for getOrganizationAdmins results in a self-sign certificate error (?!) while the equivalent sync operation works fine.

How can we replicate the problem you're reporting?
Run the snippet below on an affected computer.

Expected behavior
A clear and concise description of what you expected to happen instead, and why.

Code snippets

import asyncio
import meraki
import meraki.aio

async def main():
    async with meraki.aio.AsyncDashboardAPI(
        api_key=api_key,
        output_log=False,
        suppress_logging=False,
        maximum_concurrent_requests=5,
        nginx_429_retry_wait_time=2,
        wait_on_rate_limit=True,
        maximum_retries=100,
    ) as aiomeraki:
        try:
            org_admins_async = await aiomeraki.organizations.getOrganizationAdmins(org_id)
            print(f"Async admins: {len(org_admins_async)}")
        except Exception as e:
            print(f"Some other ERROR: {e}")


org_id = 1215707
api_key="75dd5334bef4d2bc96f26138c163c0a3fa0b5ca6"

print(50*"*", "\nSync Request\n")
dashboard = meraki.DashboardAPI(api_key=api_key, suppress_logging=False)
org_admins_sync = dashboard.organizations.getOrganizationAdmins(org_id)
print(f"Sync admins: {len(org_admins_sync)}")

print("\n\n", 50*"*", "\nAsync Request\n")
loop = asyncio.new_event_loop()
loop = asyncio.get_event_loop()
loop.run_until_complete(main())

[obrigg]

The output:

/Users/apiskuno/PycharmProjects/meraki-health-check/venv/bin/python /Users/apiskuno/PycharmProjects/meraki-health-check/test.py 
2023-09-12 18:35:07       meraki:     INFO > Meraki dashboard API session initialized with these parameters: {'version': '1.37.2', 'api_key': '************************************5ca6', 'base_url': 'https://api.meraki.com/api/v1', 'single_request_timeout': 60, 'certificate_path': '', 'requests_proxy': '', 'wait_on_rate_limit': True, 'nginx_429_retry_wait_time': 60, 'action_batch_retry_wait_time': 60, 'network_delete_retry_wait_time': 180, 'retry_4xx_error': False, 'retry_4xx_error_wait_time': 60, 'maximum_retries': 2, 'simulate': False, 'be_geo_id': None, 'caller': None, 'use_iterator_for_get_pages': False}
2023-09-12 18:35:07       meraki:     INFO > GET https://api.meraki.com/api/v1/organizations/1215707/admins
************************************************** 
Sync Request

2023-09-12 18:35:08       meraki:     INFO > organizations, getOrganizationAdmins - 200 OK
/Users/apiskuno/PycharmProjects/meraki-health-check/test.py:32: DeprecationWarning: There is no current event loop
  loop = asyncio.get_event_loop()
2023-09-12 18:35:08   meraki.aio:     INFO > Meraki dashboard API session initialized with these parameters: {'version': '1.37.2', 'api_key': '************************************5ca6', 'base_url': 'https://api.meraki.com/api/v1', 'single_request_timeout': 60, 'certificate_path': '', 'requests_proxy': '', 'wait_on_rate_limit': True, 'nginx_429_retry_wait_time': 2, 'action_batch_retry_wait_time': 60, 'network_delete_retry_wait_time': 180, 'retry_4xx_error': False, 'retry_4xx_error_wait_time': 60, 'maximum_retries': 100, 'simulate': False, 'be_geo_id': None, 'caller': None, 'use_iterator_for_get_pages': False, 'maximum_concurrent_requests': 5, 'version_warning_string': 'This library requires Python 3.7 at minimum. Python versions 3.6 and below are end of life and end of support per the Python maintainers, and your interpreter version details are: \nplatform.python_version_tuple()[0] = 3\nplatform.python_version_tuple()[1] = 10\nplatform.python_version is 3.10.6\nPlease consult the readme at your convenience: https://github.com/meraki/dashboard-api-python'}
2023-09-12 18:35:08   meraki.aio:     INFO > GET https://api.meraki.com/api/v1/organizations/1215707/admins
Sync admins: 2


 ************************************************** 
Async Request

2023-09-12 18:35:10   meraki.aio:  WARNING > organizations, getOrganizationAdmins > https://api.meraki.com/api/v1/organizations/1215707/admins - Cannot connect to host api.meraki.com:443 ssl:True [SSLCertVerificationError: (1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: self signed certificate in certificate chain (_ssl.c:997)')], retrying in 1 second
2023-09-12 18:35:11   meraki.aio:     INFO > GET https://api.meraki.com/api/v1/organizations/1215707/admins
2023-09-12 18:35:12   meraki.aio:  WARNING > organizations, getOrganizationAdmins > https://api.meraki.com/api/v1/organizations/1215707/admins - Cannot connect to host api.meraki.com:443 ssl:True [SSLCertVerificationError: (1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: self signed certificate in certificate chain (_ssl.c:997)')], retrying in 1 second
2023-09-12 18:35:13   meraki.aio:     INFO > GET https://api.meraki.com/api/v1/organizations/1215707/admins
2023-09-12 18:35:15   meraki.aio:  WARNING > organizations, getOrganizationAdmins > https://api.meraki.com/api/v1/organizations/1215707/admins - Cannot connect to host api.meraki.com:443 ssl:True [SSLCertVerificationError: (1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: self signed certificate in certificate chain (_ssl.c:997)')], retrying in 1 second
2023-09-12 18:35:16   meraki.aio:     INFO > GET https://api.meraki.com/api/v1/organizations/1215707/admins
2023-09-12 18:35:18   meraki.aio:  WARNING > organizations, getOrganizationAdmins > https://api.meraki.com/api/v1/organizations/1215707/admins - Cannot connect to host api.meraki.com:443 ssl:True [SSLCertVerificationError: (1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: self signed certificate in certificate chain (_ssl.c:997)')], retrying in 1 second
2023-09-12 18:35:19   meraki.aio:     INFO > GET https://api.meraki.com/api/v1/organizations/1215707/admins
2023-09-12 18:35:20   meraki.aio:  WARNING > organizations, getOrganizationAdmins > https://api.meraki.com/api/v1/organizations/1215707/admins - Cannot connect to host api.meraki.com:443 ssl:True [SSLCertVerificationError: (1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: self signed certificate in certificate chain (_ssl.c:997)')], retrying in 1 second
2023-09-12 18:35:21   meraki.aio:     INFO > GET https://api.meraki.com/api/v1/organizations/1215707/admins
2023-09-12 18:35:23   meraki.aio:  WARNING > organizations, getOrganizationAdmins > https://api.meraki.com/api/v1/organizations/1215707/admins - Cannot connect to host api.meraki.com:443 ssl:True [SSLCertVerificationError: (1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: self signed certificate in certificate chain (_ssl.c:997)')], retrying in 1 second

......

Some other ERROR: organizations, getOrganizationAdmins - None None, Reached retry limit: None

Process finished with exit code 0

[TKIPisalegacycipher]

Hi @obrigg are you still having this issue? If so are you able to test from different physical locations (e.g. not behind a firewall that might be doing SSL decryption)?

[obrigg]

It still happens from various locations.
Sync works, async does not (due to a certificate error).

[TKIPisalegacycipher]

Having trouble recreating this in Python 3.11; script succeeds:

2023-10-10 16:12:38       meraki:     INFO > Meraki dashboard API session initialized with these parameters: {'version': '1.38.0', 'api_key': '************************************5ca6', 'base_url': 'https://api.meraki.com/api/v1', 'single_request_timeout': 60, 'certificate_path': '', 'requests_proxy': '', 'wait_on_rate_limit': True, 'nginx_429_retry_wait_time': 60, 'action_batch_retry_wait_time': 60, 'network_delete_retry_wait_time': 240, 'retry_4xx_error': False, 'retry_4xx_error_wait_time': 60, 'maximum_retries': 2, 'simulate': False, 'be_geo_id': None, 'caller': None, 'use_iterator_for_get_pages': False}
2023-10-10 16:12:38       meraki:     INFO > GET https://api.meraki.com/api/v1/organizations/1215707/admins
************************************************** 
Sync Request

Sync admins: 2


 ************************************************** 
Async Request

2023-10-10 16:12:39       meraki:     INFO > organizations, getOrganizationAdmins - 200 OK
2023-10-10 16:12:39   meraki.aio:     INFO > Meraki dashboard API session initialized with these parameters: {'version': '1.38.0', 'api_key': '************************************5ca6', 'base_url': 'https://api.meraki.com/api/v1', 'single_request_timeout': 60, 'certificate_path': '', 'requests_proxy': '', 'wait_on_rate_limit': True, 'nginx_429_retry_wait_time': 2, 'action_batch_retry_wait_time': 60, 'network_delete_retry_wait_time': 240, 'retry_4xx_error': False, 'retry_4xx_error_wait_time': 60, 'maximum_retries': 100, 'simulate': False, 'be_geo_id': None, 'caller': None, 'use_iterator_for_get_pages': False, 'maximum_concurrent_requests': 5}
2023-10-10 16:12:39   meraki.aio:     INFO > GET https://api.meraki.com/api/v1/organizations/1215707/admins
Async admins: 2
2023-10-10 16:12:40   meraki.aio:     INFO > organizations, getOrganizationAdmins > https://api.meraki.com/api/v1/organizations/1215707/admins - 200 OK

Process finished with exit code -1073741819 (0xC0000005)

[LeChat19]

Hi.

I have this issue as well. I can workaround it by setting ssl verification to False. (Meraki -> aio -> rest_session.py)

Make the HTTP request to the API endpoint

            try:
                if self._logger:
                    self._logger.info(f"{method} {abs_url}")
                response = await self._req_session.request(
                    method, abs_url, ssl = False, **kwargs
                )

I assume that the issue is with a certificate. If I test SSL certificate I see this:

openssl s_client -showcerts -connect api.meraki.com:443 -brief
CONNECTION ESTABLISHED
Protocol version: TLSv1.3
Ciphersuite: TLS_AES_256_GCM_SHA384
Peer certificate: C = US, ST = California, L = San Francisco, O = Meraki LLC, CN = ios.meraki.com
Hash used: SHA256
Signature type: RSA-PSS
Verification: OK
Server Temp Key: X25519, 253 bits

CN is ios.meraki.com. I think it should be api.meraki.com.

Can you do the same test from a working location to check CN ?

[TKIPisalegacycipher]

Hi @LeChat19 thanks for sharing this. I see similar output from those commands, but still successful script execution.

From Ubuntu:

openssl s_client -showcerts -connect api.meraki.com:443 -brief
CONNECTION ESTABLISHED
Protocol version: TLSv1.3
Ciphersuite: TLS_AES_256_GCM_SHA384
Peer certificate: C = US, ST = California, L = San Francisco, O = Meraki LLC, CN = ios.meraki.com
Hash used: SHA256
Signature type: RSA-PSS
Verification: OK
Server Temp Key: X25519, 253 bits

And from Windows:

~#@❯ Get-RemoteCertificate api.meraki.com -As X509Certificate                                                    ❮  

Thumbprint                                Subject
----------                                -------
609E9C9990408C58630A59B1F92D8B6AA1881936  CN=ios.meraki.com, O=Meraki LLC, L=San Francisco, S=California, C=US

[LeChat19]

I'm still confused why we are using certificate signed for ios.meraki.com while requests are sent to api.meraki.com.

However looking to this issue I see that it's a common one for Mac. Issue is with access to MacOS root certificates. It can be fixed by installing root certificates:

% cd /Applications/Python\ 3.10

(can be different depending on version)

% ./Install\ Certificates.command

I was able to run the script.

[TKIPisalegacycipher]

Thank you @LeChat19. My test above was from a Windows machine. I am also left wondering why connections via aiohttp would have an issue but not the requests library. If anyone would like to propose a CR that mitigates this issue we should evaluate it and consider adding it into the library code. I wonder broadly if we should bump the required version of the packages the async library uses, but I'm not confident yet that would fix this particular issue.

For now though, I'm not seeing any definitive evidence that this is a library-side issue.

1. The same test script works from other locations/host OSes and in multiple versions of Python.
2. There's no specific recent change to the async library that would explain a certificate issue (keep me honest!).
3. The CN on the certificate doesn't cause issues on its own; it isn't the reason the Mac hosts experiencing the issue have the cert error. Separately, if there are any questions about the cert, Meraki Support can best answer those.

With that said I've marked this ticket 'not able to reproduce' and welcome further guidance and feedback from the community.

[LeChat19]

I think this is the official information regarding this issue.

https://bugs.python.org/issue43404

Can we update documentation that we need to run the "Install Certificates.command" on Macos or disable SSL certificate verification ?

Disabling SSL verification is a workaround.

[TKIPisalegacycipher]

Thanks for your help, @LeChat19. I've updated the readme to cover this macOS limitation. I'll close the issue.