config, remote: Made S3 CA bundle customizable #6018
Conversation
botocore allows a path to a custom CA bundle either by passing a path to the CA bundle file into the verify argument of boto3.session.Session.client or passing None (the default) which will fall back to the AWS config. Previously, the DVC config only accepted a boolean into the ssl_verify option in the remote S3 config. This changes the DVC config to accept both string and None in addition to boolean and defaults to None. I also changed the default for ssl_verfiy to None in BaseS3FileSystem. Thus, if ssl_verify is not provided, botocore will fall back to the AWS config. Testing Unit tests to cover the changes to the config schema and addition ssl_verify types that will be passed into S3FileSystem. Also, ran dvc push -r object-store data/cifar-10-python.tar.gz in my work environment that has a private S3 endpoint that requires a custom CA bundle, both with and without ssl_verify specified in the config. This was successful, showing that communication could be established. And I ran dvc remote modify object-store ssl_verify "$HOME/.aws/cabundle.pem" and confirmed that the custom CA bundle path was added to the config. Fixes iterative#6012
|
@rgvanwesep this PR will require a docs update - in the See https://dvc.org/doc/user-guide/contributing/docs#submitting-changes for info on submitting the docs PR, the file you need to update is: https://github.com/iterative/dvc.org/blob/master/content/docs/command-reference/remote/modify.md |
|
@pmrowla Thanks for pointing that out. I should be able to put up the docs PR today. |
iterative/dvc#6018 implements the ability to set `ssl_verify` in the S3 remote config to a path to a custom CA bundle file in addition to setting true/false. It also makes the default the same as the `botocore` default, which is to read the CA bundle path from the AWS config. This updates the docs to reflect those changes.
|
The doc PR: |
dvc/config_schema.py
Outdated
| @@ -145,7 +145,7 @@ class RelPath(str): | |||
| "session_token": str, | |||
| Optional("listobjects", default=False): Bool, # obsoleted | |||
| Optional("use_ssl", default=True): Bool, | |||
| Optional("ssl_verify", default=True): Bool, | |||
| Optional("ssl_verify", default=None): Any(Bool, str, None), | |||
There was a problem hiding this comment.
I guess we could just
| Optional("ssl_verify", default=None): Any(Bool, str, None), | |
| "ssl_verify": Any(Bool, str, None), |
since they are optional by default.
There was a problem hiding this comment.
Yeah, that makes sense. And at that point I think I can get rid of the None so that it is "ssl_verify": Any(Bool, str),. I'm testing the change now and should be able to push soon.
Responding to PR comment, removed the Optional, default None on ssl_verify since the config keys are optional by default. Rather than a missing ssl_verify producing a None that eventually gets filtered, it doesn't appear in the parsed config in the first place.
* Updated S3 ssl_verify documentation iterative/dvc#6018 implements the ability to set `ssl_verify` in the S3 remote config to a path to a custom CA bundle file in addition to setting true/false. It also makes the default the same as the `botocore` default, which is to read the CA bundle path from the AWS config. This updates the docs to reflect those changes. * Update content/docs/command-reference/remote/modify.md * Update content/docs/command-reference/remote/modify.md * Apply suggestions from code review Co-authored-by: Jorge Orpinel <jorgeorpinel@users.noreply.github.com>
|
I thought this added support for http(s) as well. Looking into it. :) |
botocoreallows a path to a custom CA bundle either by passing a path to the CA bundle file into the verify argument ofboto3.session.Session.client(see https://boto3.amazonaws.com/v1/documentation/api/latest/reference/core/session.html) or passingNone(the default) which will fall back to the AWS config. Previously, the DVC config only accepted aboolean into the
ssl_verifyoption in the remote S3 config. This changes the DVC config to accept both string andNonein addition to boolean and defaults toNone. I also changed the default forssl_verfiytoNoneinBaseS3FileSystem. Thus, ifssl_verifyis not provided,botocorewill fall back to the AWS config.Testing
Unit tests to cover the changes to the config schema and additional
ssl_verifytypes that will be passed intoS3FileSystem. Also, ranin my work environment that has a private S3 endpoint that requires a custom CA bundle, both with and without
ssl_verifyspecified in the DVC config. This was successful, showing that communication could be established. And I ranand confirmed that the custom CA bundle path was added to the config.
Fixes #6012
❗ I have followed the Contributing to DVC checklist.
📖 If this PR requires documentation updates, I have created a separate PR (or issue, at least) in dvc.org and linked it here.
config, remote: Made S3 CA bundle customizable #6018
Thank you for the contribution - we'll try to review it as soon as possible. 🙏