SSO does not honor custom ca-bundle configuration

[kjluedke]

Describe the bug

Executing an sso subcommand does not honor the --ca-bundle option or ca-bundle configuration. The value provided to the CLI is ignored in favor of the default certificate bundle.

Expected Behavior

I expected my specified bundle to be used.

Current Behavior

The default bundle at C:\Program Files\Amazon\AWSCLIV2\awscli\botocore\cacert.pem is used.

Reproduction Steps

Execute the following command:

aws sso login --ca-bundle C:\Users\myUser\aws-ca-oidc-bundle.pem --debug

Observe the following line in the debug output:
2023-01-17 09:53:20,292 - MainThread - botocore.httpsession - DEBUG - Certificate path: C:\Program Files\Amazon\AWSCLIV2\awscli\botocore\cacert.pem

Possible Solution

Our temporary workaround has been to replace the contents of C:\Program Files\Amazon\AWSCLIV2\awscli\botocore\cacert.pem with our custom bundle.

Additional Information/Context

This defect might be related to aws/aws-cdk#21328

CLI version used

aws-cli/2.9.15 Python/3.9.11

Environment details (OS name and version, etc.)

Windows/10 exe/AMD64

[tim-finnigan]

Thanks @kjluedke for reporting this. The team confirmed the issue and will need to investigate it further. We will share any updates here.

[esmcelroy]

Reporting that I'm having the same issue. Version: aws-cli/2.11.0 Python/3.11.2 Windows/10 exe/AMD64
Pulling the bottom-most traceback after running aws configure sso --no-verify-ssl --debug:

Traceback (most recent call last):
  File "awscli\clidriver.py", line 460, in main
  File "awscli\customizations\commands.py", line 151, in __call__
  File "awscli\customizations\commands.py", line 205, in __call__
  File "awscli\customizations\configure\sso.py", line 516, in _run_main
  File "awscli\customizations\sso\utils.py", line 72, in do_sso_login
  File "awscli\botocore\utils.py", line 3052, in fetch_token
  File "awscli\botocore\utils.py", line 3037, in _token
  File "awscli\botocore\utils.py", line 2947, in _poll_for_token
  File "awscli\botocore\utils.py", line 2918, in _registration
  File "awscli\botocore\utils.py", line 2875, in _register_client
  File "awscli\botocore\client.py", line 341, in _api_call
  File "awscli\botocore\client.py", line 683, in _make_api_call
  File "awscli\botocore\client.py", line 703, in _make_request
  File "awscli\botocore\endpoint.py", line 101, in make_request
  File "awscli\botocore\endpoint.py", line 155, in _send_request
  File "awscli\botocore\endpoint.py", line 199, in _do_get_response
  File "awscli\botocore\endpoint.py", line 271, in _send
  File "awscli\botocore\httpsession.py", line 475, in send
botocore.exceptions.SSLError: SSL validation failed for https://oidc.us-west-2.amazonaws.com/client/register [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: self signed certificate in certificate chain (_ssl.c:992)

Which suggests that at no point in the creation of the SSO request session is the session made aware of the verify global variable. Throughout the various customization files, we can see different sessions being created with different contexts (and usually with reference to the verify value):
For configservice.subscribe: https://github.com/aws/aws-cli/blob/2.11.0/awscli/customizations/configservice/subscribe.py#L120-L130
For codedeploy.register: https://github.com/aws/aws-cli/blob/2.11.0/awscli/customizations/codedeploy/register.py#L76-L81

Then, for configure.sso: https://github.com/aws/aws-cli/blob/2.11.0/awscli/customizations/configure/sso.py#L528-L529

This may not be the issue, but it's clear that the verify option isn't being correctly parsed during token-get operations, sso configure operations and other SSO-based operations. It's possible the issue is more widespread, but currently it's a clear issue with SSO. We've also worked around this issue by updating the cacerts.pem bundle, but this isn't a sustainable or maintainable fix.

[jboeshart]

Still seeing this as an issue as well. Running aws-cli/2.13.0 Python/3.11.4 Windows/10 exe/AMD64 prompt/off. Since this has been an issue since at least January, any chance the CLI documentation could be updated to reflect that neither the config file profile setting nor command line parameter work in the latest version? It does seem to work when specifying through environment variable though, both in Windows CMD and Windows PowerShell. Might save some others from burning time trying to troubleshoot this, if it's not something that is going to be fixed soon.

[jboeshart]

Also looks like this was reported here #7552 as well.

[jboeshart]

I just upgraded to 2.13.34 and it seems to be honoring the ca_bundle configuration file setting, haven't tried the other options. I don't see anything in the release notes that indicates there was a fix released, so I'm not sure if it's fully fixed or not.

[jboeshart]

Strike that, looks like the certs must have been cached or something, as it's no longer honoring the config file and I've had to manually add the cert of our forward proxy to "C:\Program Files\Amazon\AWSCLIV2\awscli\botocore\cacert.pem" in order for it to work. Underlying issue still looks to be there.

[esmcelroy]

I was gonna say, I didn't see any code changes in the last ~year to the SSO functions, aside from a docs update!

If I can assemble myself a testing environment with a self-signed certificate injection, I'll put together the fix - it shouldn't be too bad... It looks like when it creates the session it's just not receiving the parsed globals.

[Farmbuyer]

Any progress on this by chance? We're having to use workarounds here for overly intrusive SSL inspection; until the IT department implements the proper fixes documented elsewhere, the individual programmers need to use --no-verify-ssl for all commands.

Unfortunately, when SSO ignores the CA bundle and the option, there's no way to make progress.

[esmcelroy]

There's likely more to it (or at least I hope there is...), and I don't have a good test environment with a self-signed cert in the chain, but if I had to guess, this is the diff of the change against the latest release tag to support global verify_ssl settings:

diff --git a/awscli/customizations/configure/sso.py b/awscli/customizations/configure/sso.py
--- awscli/customizations/configure/sso.py
+++ awscli/customizations/configure/sso.py
@@ -524,9 +524,9 @@
         client_config = Config(
             signature_version=UNSIGNED,
             region_name=sso_registration_args['sso_region'],
         )
-        sso = self._session.create_client('sso', config=client_config)
+        sso = self._session.create_client('sso', config=client_config, verify=parsed_globals.verify_ssl)
 
         sso_account_id, sso_role_name = self._prompt_for_sso_account_and_role(
             sso, sso_token
         )

Did a build with this change included in 2.15.7 on my local, and it hasn't broken anything afaict, but I can't confirm if it's using the global settings as expected. If I have some time this week, I'll setup a test environment and see if this in fact works.

[joaocc]

Hi. Is there any progress on this? aws-cli/2.15.18 still exhibits this behaviour. Thanks

[lambda-science]

Any update ? Facing the same issue with Zscaler, it's terrible