dvc push: unable to validate self-signed certificate from S3 service

[JoaoDeDynam]

Bug Report

Description

Attempting dvc push to s3 remote which uses SSL termination. Getting certificate validation error since we are using self-signed certificates

Reproduce

1. dvc init
2. modify remote to s3 endpoint with https scheme
3. copy certificate bundle from remote se to local directory
4. update AWS_CA_BUNDLE environment variable to point to local certificate bundle
5. dvc add
6. dvc push

Expected

data sent to s3 remote

Environment information

MacOs
Homebrew DVC installation

Output of dvc doctor:
DVC version: 2.0.18 (brew)

Platform: Python 3.9.4 on macOS-10.15.7-x86_64-i386-64bit
Supports: azure, gdrive, gs, http, https, s3, ssh, oss, webdav, webdavs
Cache types: reflink, hardlink, symlink
Cache directory: apfs on /dev/disk1s1
Caches: local
Remotes: s3
Workspace directory: apfs on /dev/disk1s1
Repo: dvc, git

$ dvc doctor

Additional Information (if any):
I placed the output of dvc push -v below along with relevant environment variables

johnathan@Johnathans-MacBook-Pro repo % dvc push -v
2021-05-05 13:56:31,064 DEBUG: Check for update is enabled.
+------------------------------------------+
|                                          |
|     Update available 2.0.18 -> 2.1.0     |
|          Run `brew upgrade dvc`          |
|                                          |
+------------------------------------------+

2021-05-05 13:56:31,329 DEBUG: Preparing to upload data to 's3://internal-test-files'
2021-05-05 13:56:31,330 DEBUG: Preparing to collect status from s3://internal-test-files
2021-05-05 13:56:31,330 DEBUG: Collecting information from local cache...
2021-05-05 13:56:31,332 DEBUG: Collecting information from remote cache...                                                                                                            
2021-05-05 13:56:31,332 DEBUG: Querying 1 hashes via object_exists
2021-05-05 13:56:43,907 ERROR: unexpected error - SSL validation failed for https://remote:443/internal-test-files?prefix=a5%2Fb8b049b6da9ad14c1a6b77ad0f49b1.dir&encoding-type=url [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: self signed certificate in certificate chain (_ssl.c:1123)
------------------------------------------------------------
Traceback (most recent call last):
  File "/usr/local/Cellar/dvc/2.0.18/libexec/lib/python3.9/site-packages/urllib3/connectionpool.py", line 699, in urlopen
    httplib_response = self._make_request(
  File "/usr/local/Cellar/dvc/2.0.18/libexec/lib/python3.9/site-packages/urllib3/connectionpool.py", line 382, in _make_request
    self._validate_conn(conn)
  File "/usr/local/Cellar/dvc/2.0.18/libexec/lib/python3.9/site-packages/urllib3/connectionpool.py", line 1010, in _validate_conn
    conn.connect()
  File "/usr/local/Cellar/dvc/2.0.18/libexec/lib/python3.9/site-packages/urllib3/connection.py", line 411, in connect
    self.sock = ssl_wrap_socket(
  File "/usr/local/Cellar/dvc/2.0.18/libexec/lib/python3.9/site-packages/urllib3/util/ssl_.py", line 428, in ssl_wrap_socket
    ssl_sock = _ssl_wrap_socket_impl(
  File "/usr/local/Cellar/dvc/2.0.18/libexec/lib/python3.9/site-packages/urllib3/util/ssl_.py", line 472, in _ssl_wrap_socket_impl
    return ssl_context.wrap_socket(sock, server_hostname=server_hostname)
  File "/usr/local/Cellar/python@3.9/3.9.4/Frameworks/Python.framework/Versions/3.9/lib/python3.9/ssl.py", line 500, in wrap_socket
    return self.sslsocket_class._create(
  File "/usr/local/Cellar/python@3.9/3.9.4/Frameworks/Python.framework/Versions/3.9/lib/python3.9/ssl.py", line 1040, in _create
    self.do_handshake()
  File "/usr/local/Cellar/python@3.9/3.9.4/Frameworks/Python.framework/Versions/3.9/lib/python3.9/ssl.py", line 1309, in do_handshake
    self._sslobj.do_handshake()
ssl.SSLCertVerificationError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: self signed certificate in certificate chain (_ssl.c:1123)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/Cellar/dvc/2.0.18/libexec/lib/python3.9/site-packages/botocore/httpsession.py", line 314, in send
    urllib_response = conn.urlopen(
  File "/usr/local/Cellar/dvc/2.0.18/libexec/lib/python3.9/site-packages/urllib3/connectionpool.py", line 755, in urlopen
    retries = retries.increment(
  File "/usr/local/Cellar/dvc/2.0.18/libexec/lib/python3.9/site-packages/urllib3/util/retry.py", line 507, in increment
    raise six.reraise(type(error), error, _stacktrace)
  File "/usr/local/Cellar/dvc/2.0.18/libexec/lib/python3.9/site-packages/urllib3/packages/six.py", line 734, in reraise
    raise value.with_traceback(tb)
  File "/usr/local/Cellar/dvc/2.0.18/libexec/lib/python3.9/site-packages/urllib3/connectionpool.py", line 699, in urlopen
    httplib_response = self._make_request(
  File "/usr/local/Cellar/dvc/2.0.18/libexec/lib/python3.9/site-packages/urllib3/connectionpool.py", line 382, in _make_request
    self._validate_conn(conn)
  File "/usr/local/Cellar/dvc/2.0.18/libexec/lib/python3.9/site-packages/urllib3/connectionpool.py", line 1010, in _validate_conn
    conn.connect()
  File "/usr/local/Cellar/dvc/2.0.18/libexec/lib/python3.9/site-packages/urllib3/connection.py", line 411, in connect
    self.sock = ssl_wrap_socket(
  File "/usr/local/Cellar/dvc/2.0.18/libexec/lib/python3.9/site-packages/urllib3/util/ssl_.py", line 428, in ssl_wrap_socket
    ssl_sock = _ssl_wrap_socket_impl(
  File "/usr/local/Cellar/dvc/2.0.18/libexec/lib/python3.9/site-packages/urllib3/util/ssl_.py", line 472, in _ssl_wrap_socket_impl
    return ssl_context.wrap_socket(sock, server_hostname=server_hostname)
  File "/usr/local/Cellar/python@3.9/3.9.4/Frameworks/Python.framework/Versions/3.9/lib/python3.9/ssl.py", line 500, in wrap_socket
    return self.sslsocket_class._create(
  File "/usr/local/Cellar/python@3.9/3.9.4/Frameworks/Python.framework/Versions/3.9/lib/python3.9/ssl.py", line 1040, in _create
    self.do_handshake()
  File "/usr/local/Cellar/python@3.9/3.9.4/Frameworks/Python.framework/Versions/3.9/lib/python3.9/ssl.py", line 1309, in do_handshake
    self._sslobj.do_handshake()
urllib3.exceptions.SSLError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: self signed certificate in certificate chain (_ssl.c:1123)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/Cellar/dvc/2.0.18/libexec/lib/python3.9/site-packages/dvc/main.py", line 55, in main
    ret = cmd.run()
  File "/usr/local/Cellar/dvc/2.0.18/libexec/lib/python3.9/site-packages/dvc/command/data_sync.py", line 56, in run
    processed_files_count = self.repo.push(
  File "/usr/local/Cellar/dvc/2.0.18/libexec/lib/python3.9/site-packages/dvc/repo/__init__.py", line 49, in wrapper
    return f(repo, *args, **kwargs)
  File "/usr/local/Cellar/dvc/2.0.18/libexec/lib/python3.9/site-packages/dvc/repo/push.py", line 41, in push
    return len(used_run_cache) + self.cloud.push(used, jobs, remote=remote)
  File "/usr/local/Cellar/dvc/2.0.18/libexec/lib/python3.9/site-packages/dvc/data_cloud.py", line 66, in push
    return remote.push(
  File "/usr/local/Cellar/dvc/2.0.18/libexec/lib/python3.9/site-packages/dvc/remote/base.py", line 56, in wrapper
    return f(obj, *args, **kwargs)
  File "/usr/local/Cellar/dvc/2.0.18/libexec/lib/python3.9/site-packages/dvc/remote/base.py", line 466, in push
    ret = self._process(
  File "/usr/local/Cellar/dvc/2.0.18/libexec/lib/python3.9/site-packages/dvc/remote/base.py", line 323, in _process
    dir_status, file_status, dir_contents = self._status(
  File "/usr/local/Cellar/dvc/2.0.18/libexec/lib/python3.9/site-packages/dvc/remote/base.py", line 169, in _status
    remote_exists.update(
  File "/usr/local/Cellar/dvc/2.0.18/libexec/lib/python3.9/site-packages/dvc/remote/base.py", line 242, in _indexed_dir_hashes
    dir_exists.update(self.odb.list_hashes_exists(dir_md5s - dir_exists))
  File "/usr/local/Cellar/dvc/2.0.18/libexec/lib/python3.9/site-packages/dvc/objects/db/base.py", line 338, in list_hashes_exists
    ret = list(itertools.compress(hashes, in_remote))
  File "/usr/local/Cellar/python@3.9/3.9.4/Frameworks/Python.framework/Versions/3.9/lib/python3.9/concurrent/futures/_base.py", line 608, in result_iterator
    yield fs.pop().result()
  File "/usr/local/Cellar/python@3.9/3.9.4/Frameworks/Python.framework/Versions/3.9/lib/python3.9/concurrent/futures/_base.py", line 445, in result
    return self.__get_result()
  File "/usr/local/Cellar/python@3.9/3.9.4/Frameworks/Python.framework/Versions/3.9/lib/python3.9/concurrent/futures/_base.py", line 390, in __get_result
    raise self._exception
  File "/usr/local/Cellar/python@3.9/3.9.4/Frameworks/Python.framework/Versions/3.9/lib/python3.9/concurrent/futures/thread.py", line 52, in run
    result = self.fn(*self.args, **self.kwargs)
  File "/usr/local/Cellar/dvc/2.0.18/libexec/lib/python3.9/site-packages/dvc/objects/db/base.py", line 329, in exists_with_progress
    ret = self.fs.exists(path_info)
  File "/usr/local/Cellar/dvc/2.0.18/libexec/lib/python3.9/site-packages/dvc/fs/s3.py", line 233, in exists
    return self.isfile(path_info) or self.isdir(path_info)
  File "/usr/local/Cellar/dvc/2.0.18/libexec/lib/python3.9/site-packages/dvc/fs/s3.py", line 263, in isfile
    return path_info.path in self._list_paths(path_info)
  File "/usr/local/Cellar/dvc/2.0.18/libexec/lib/python3.9/site-packages/dvc/fs/s3.py", line 284, in _list_paths
    for obj_summary in obj_summaries:
  File "/usr/local/Cellar/dvc/2.0.18/libexec/lib/python3.9/site-packages/boto3/resources/collection.py", line 83, in __iter__
    for page in self.pages():
  File "/usr/local/Cellar/dvc/2.0.18/libexec/lib/python3.9/site-packages/boto3/resources/collection.py", line 166, in pages
    for page in pages:
  File "/usr/local/Cellar/dvc/2.0.18/libexec/lib/python3.9/site-packages/botocore/paginate.py", line 255, in __iter__
    response = self._make_request(current_kwargs)
  File "/usr/local/Cellar/dvc/2.0.18/libexec/lib/python3.9/site-packages/botocore/paginate.py", line 332, in _make_request
    return self._method(**current_kwargs)
  File "/usr/local/Cellar/dvc/2.0.18/libexec/lib/python3.9/site-packages/botocore/client.py", line 357, in _api_call
    return self._make_api_call(operation_name, kwargs)
  File "/usr/local/Cellar/dvc/2.0.18/libexec/lib/python3.9/site-packages/botocore/client.py", line 662, in _make_api_call
    http, parsed_response = self._make_request(
  File "/usr/local/Cellar/dvc/2.0.18/libexec/lib/python3.9/site-packages/botocore/client.py", line 682, in _make_request
    return self._endpoint.make_request(operation_model, request_dict)
  File "/usr/local/Cellar/dvc/2.0.18/libexec/lib/python3.9/site-packages/botocore/endpoint.py", line 102, in make_request
    return self._send_request(request_dict, operation_model)
  File "/usr/local/Cellar/dvc/2.0.18/libexec/lib/python3.9/site-packages/botocore/endpoint.py", line 136, in _send_request
    while self._needs_retry(attempts, operation_model, request_dict,
  File "/usr/local/Cellar/dvc/2.0.18/libexec/lib/python3.9/site-packages/botocore/endpoint.py", line 253, in _needs_retry
    responses = self._event_emitter.emit(
  File "/usr/local/Cellar/dvc/2.0.18/libexec/lib/python3.9/site-packages/botocore/hooks.py", line 356, in emit
    return self._emitter.emit(aliased_event_name, **kwargs)
  File "/usr/local/Cellar/dvc/2.0.18/libexec/lib/python3.9/site-packages/botocore/hooks.py", line 228, in emit
    return self._emit(event_name, kwargs)
  File "/usr/local/Cellar/dvc/2.0.18/libexec/lib/python3.9/site-packages/botocore/hooks.py", line 211, in _emit
    response = handler(**kwargs)
  File "/usr/local/Cellar/dvc/2.0.18/libexec/lib/python3.9/site-packages/botocore/retryhandler.py", line 183, in __call__
    if self._checker(attempts, response, caught_exception):
  File "/usr/local/Cellar/dvc/2.0.18/libexec/lib/python3.9/site-packages/botocore/retryhandler.py", line 250, in __call__
    should_retry = self._should_retry(attempt_number, response,
  File "/usr/local/Cellar/dvc/2.0.18/libexec/lib/python3.9/site-packages/botocore/retryhandler.py", line 277, in _should_retry
    return self._checker(attempt_number, response, caught_exception)
  File "/usr/local/Cellar/dvc/2.0.18/libexec/lib/python3.9/site-packages/botocore/retryhandler.py", line 316, in __call__
    checker_response = checker(attempt_number, response,
  File "/usr/local/Cellar/dvc/2.0.18/libexec/lib/python3.9/site-packages/botocore/retryhandler.py", line 222, in __call__
    return self._check_caught_exception(
  File "/usr/local/Cellar/dvc/2.0.18/libexec/lib/python3.9/site-packages/botocore/retryhandler.py", line 359, in _check_caught_exception
    raise caught_exception
  File "/usr/local/Cellar/dvc/2.0.18/libexec/lib/python3.9/site-packages/botocore/endpoint.py", line 200, in _do_get_response
    http_response = self._send(request)
  File "/usr/local/Cellar/dvc/2.0.18/libexec/lib/python3.9/site-packages/botocore/endpoint.py", line 269, in _send
    return self.http_session.send(request)
  File "/usr/local/Cellar/dvc/2.0.18/libexec/lib/python3.9/site-packages/botocore/httpsession.py", line 341, in send
    raise SSLError(endpoint_url=request.url, error=e)
botocore.exceptions.SSLError: SSL validation failed for https://remote:443/internal-test-files?prefix=a5%2Fb8b049b6da9ad14c1a6b77ad0f49b1.dir&encoding-type=url [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: self signed certificate in certificate chain (_ssl.c:1123)
------------------------------------------------------------
2021-05-05 13:56:44,478 DEBUG: Version info for developers:
DVC version: 2.0.18 (brew)
---------------------------------
Platform: Python 3.9.4 on macOS-10.15.7-x86_64-i386-64bit
Supports: azure, gdrive, gs, http, https, s3, ssh, oss, webdav, webdavs
Cache types: reflink, hardlink, symlink
Cache directory: apfs on /dev/disk1s1
Caches: local
Remotes: s3
Workspace directory: apfs on /dev/disk1s1
Repo: dvc, git

Having any troubles? Hit us up at https://dvc.org/support, we are always happy to help!
2021-05-05 13:56:44,479 DEBUG: Analytics is enabled.
2021-05-05 13:56:44,717 DEBUG: Trying to spawn '['daemon', '-q', 'analytics', '/var/folders/9m/xx24sxdx2fvb9s5q5rt1jxy00000gn/T/tmpo1qv_wjs']'
2021-05-05 13:56:44,722 DEBUG: Spawned '['daemon', '-q', 'analytics', '/var/folders/9m/xx24sxdx2fvb9s5q5rt1jxy00000gn/T/tmpo1qv_wjs']'

johnathan@Johnathans-MacBook-Pro repo % printenv
USER=johnathan
PATH=/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin:/Library/Apple/usr/bin
...
TERM=xterm-256color
AWS_CA_BUNDLE=</path/to/bundle.crt>
REQUESTS_CA_BUNDLE=REQUESTS_CA_BUNDLE=</path/to/bundle.crt>
_=/usr/bin/printenv
johnathan@Johnathans-MacBook-Pro repo % cat cat /Users/johnathan/.bash_profile             
cat: cat: No such file or directory
export PATH="/usr/local/bin:$PATH"
export PATH="/usr/local/opt/python@3.7/bin:$PATH"
export GODEBUG="x509ignoreCN=0"
export AWS_CA_BUNDLE="</path/to/bundle.crt>"
export REQUESTS_CA_BUNDLE="</path/to/bundle.crt>"
export SSL_CERT_FILE="</path/to/bundle.crt>"

[isidentical]

@JoaoDeDynam can you try to install dvc from the master, pip install "dvc[s3] @ git+https://github.com/iterative/dvc and try again? We recently rewrote all our s3 code, so it would be nice if you could check that version out and let us know whether you still experience an odd behavior.

[JoaoDeDynam]

after setting up a virtualenv and installing with pip and pip3, I run into the same issues I was having on this issue <#5900 (comment)>

…

On Thu, May 6, 2021 at 4:14 AM Batuhan Taskaya ***@***.***> wrote: @JoaoDeDynam <https://github.com/JoaoDeDynam> can you try to install dvc from the master, pip install "dvc[s3] @ git+ https://github.com/iterative/dvc and try again? We recently rewrote all our s3 code, so it would be nice if you could check that version out and let us know whether you still experience an odd behavior. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#5972 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AT4QSBM7RXQW53CDUP46PSDTMJ2ZFANCNFSM44F2BCPQ> .

-- This e-mail message, including any attachments, is for the sole use of the intended recipient(s) and may contain confidential and privileged information. Unauthorized review, use, disclosure or distribution is prohibited. If you are not the intended recipient, please contact the sender by reply e-mail and destroy all copies of the original message. Thank you.

[dolfinus]

There is an investigation of the issue:
boto/botocore#2339 (comment)

boto3 library reads AWS_CA_BUNDLE/REQUESTS_CA_BUNDLE environment variables to get a path of certificate bundle which is then used to verify SSL handshake.

It is also possible to pass a verify option explicitly into the boto3 client class.
If it is passed, False means disabling SSL check and some str means to use this value as a path to certificates bundle.
But True means not only to enable SSL check but also to completely ignore value of both AWS_CA_BUNDLE and REQUESTS_CA_BUNDLE environment variables.
They are being parsed only if version option value is None or if this option has not been passed into a client at all.

But dvc uses the same config value ssl_verify for both HTTP and S3 storage types, and the default value for this option is True.

This unpexpected behavior also caused an error in MLflow project: mlflow/mlflow#4047

[3265]

I have a similar problem when i push my data to s3.
Following message is the error message in my case.

$ dvc push
ERROR: unexpected error - Cannot connect to host xxx.s3.amazonaws.com:443 ssl:default [[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:852)]: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:852)

Having any troubles? Hit us up at https://dvc.org/support, we are always happy to help!
dvc push  4.14s user 0.32s system 24% cpu 18.118 total

my dvc version was 2.3.0

$ dvc --version
2.3.0

[mike@mike-pc /run/media/mike/70ecef3b-5ec4-4400-8028-7955b30f3a46/bush-dataset][10:58:40] !+(master)[O][I][%][λ]> dvc doctor
DVC version: 2.3.0 (snap)
---------------------------------
Platform: Python 3.6.9 on Linux-5.10.41-1-MANJARO-x86_64-with-ManjaroLinux-21.0.6-Ornara
Supports: azure, gdrive, gs, http, https, s3, ssh
Cache types: hardlink, symlink
Cache directory: ext4 on /dev/sdb1
Caches: local
Remotes: s3
Workspace directory: ext4 on /dev/sdb1
Repo: dvc, git

[pmrowla]

This issue should be resolved in 2.3.0

In 2.3.0 ssl_verify for S3 remotes can be used to point directly to a custom CA bundle, the same way you would use verify with boto. If ssl_verify is unset in your remote config, DVC will fall back (via boto) to the CA bundle specified in your ~/.aws config (previously leaving ssl_verify unset would force the default value to True)

[3265]

@pmrowla thanks! After setup use_ssl = false in dvc config. SSL error was not displayed !

[core]
    remote = main
['remote "main"']
    url = s3://xxx
    use_ssl = false

[pmrowla]

@mgoldchild just so you are aware, that will disable SSL (HTTPS) entirely. If you were looking to use SSL with a self-signed certificate, you should be using the ssl_verify config option, and not disabling use_ssl.

[justusschock]

@pmrowla I experience the same issue with dvc 2.5.0. I setup a custom s3 storage using minio with a self-signed certificate. with setting ssl_verify to false I get past the first SSL-Error (probably for authentication?) but for each file I try to pull I still get this error. Disabling ssl at all with use_ssl does not help either.

The output from dvc doctor is:

➜ dvc doctor
DVC version: 2.5.0 (brew)
---------------------------------
Platform: Python 3.9.6 on macOS-11.4-x86_64-i386-64bit
Supports: azure, gdrive, gs, http, https, s3, ssh, oss, webdav, webdavs
Cache types: reflink, hardlink, symlink
Cache directory: apfs on /dev/disk1s5s1
Caches: local
Remotes: s3
Workspace directory: apfs on /dev/disk1s5s1
Repo: dvc, git

[skshetry]

@justusschock, did you set the endpoint_url? Could you please try posting the error that you get ? Please try adding the -v flag to the command that you are using.

[justusschock]

@skshetry Yes, I set the endpoint_url to my internal domain (replaced with MYURL below).

I get the following error:

2021-07-08 14:08:45,104 DEBUG: Uploading '.dvc/cache/d1/a1f686dc2f807b33a81bc6c02e5c57' to 's3://dvc/Dens/d1/a1f686dc2f807b33a81bc6c02e5c57'
2021-07-08 14:08:47,309 ERROR: failed to upload '.dvc/cache/ac/13f137d80741105dc543f08858c4ae' to 's3://dvc/Dens/ac/13f137d80741105dc543f08858c4ae' - SSL validation failed for https://MYURL:9000/dvc/Dens/ac/13f137d80741105dc543f08858c4ae [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: self signed certificate (_ssl.c:1129)
------------------------------------------------------------
Traceback (most recent call last):
  File "/usr/local/Cellar/dvc/2.5.0/libexec/lib/python3.9/site-packages/urllib3/connectionpool.py", line 699, in urlopen
    httplib_response = self._make_request(
  File "/usr/local/Cellar/dvc/2.5.0/libexec/lib/python3.9/site-packages/urllib3/connectionpool.py", line 382, in _make_request
    self._validate_conn(conn)
  File "/usr/local/Cellar/dvc/2.5.0/libexec/lib/python3.9/site-packages/urllib3/connectionpool.py", line 1010, in _validate_conn
    conn.connect()
  File "/usr/local/Cellar/dvc/2.5.0/libexec/lib/python3.9/site-packages/urllib3/connection.py", line 411, in connect
    self.sock = ssl_wrap_socket(
  File "/usr/local/Cellar/dvc/2.5.0/libexec/lib/python3.9/site-packages/urllib3/util/ssl_.py", line 449, in ssl_wrap_socket
    ssl_sock = _ssl_wrap_socket_impl(
  File "/usr/local/Cellar/dvc/2.5.0/libexec/lib/python3.9/site-packages/urllib3/util/ssl_.py", line 493, in _ssl_wrap_socket_impl
    return ssl_context.wrap_socket(sock, server_hostname=server_hostname)
  File "/usr/local/Cellar/python@3.9/3.9.6/Frameworks/Python.framework/Versions/3.9/lib/python3.9/ssl.py", line 500, in wrap_socket
    return self.sslsocket_class._create(
  File "/usr/local/Cellar/python@3.9/3.9.6/Frameworks/Python.framework/Versions/3.9/lib/python3.9/ssl.py", line 1040, in _create
    self.do_handshake()
  File "/usr/local/Cellar/python@3.9/3.9.6/Frameworks/Python.framework/Versions/3.9/lib/python3.9/ssl.py", line 1309, in do_handshake
    self._sslobj.do_handshake()
ssl.SSLCertVerificationError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: self signed certificate (_ssl.c:1129)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/Cellar/dvc/2.5.0/libexec/lib/python3.9/site-packages/botocore/httpsession.py", line 314, in send
    urllib_response = conn.urlopen(
  File "/usr/local/Cellar/dvc/2.5.0/libexec/lib/python3.9/site-packages/urllib3/connectionpool.py", line 755, in urlopen
    retries = retries.increment(
  File "/usr/local/Cellar/dvc/2.5.0/libexec/lib/python3.9/site-packages/urllib3/util/retry.py", line 507, in increment
    raise six.reraise(type(error), error, _stacktrace)
  File "/usr/local/Cellar/dvc/2.5.0/libexec/lib/python3.9/site-packages/urllib3/packages/six.py", line 769, in reraise
    raise value.with_traceback(tb)
  File "/usr/local/Cellar/dvc/2.5.0/libexec/lib/python3.9/site-packages/urllib3/connectionpool.py", line 699, in urlopen
    httplib_response = self._make_request(
  File "/usr/local/Cellar/dvc/2.5.0/libexec/lib/python3.9/site-packages/urllib3/connectionpool.py", line 382, in _make_request
    self._validate_conn(conn)
  File "/usr/local/Cellar/dvc/2.5.0/libexec/lib/python3.9/site-packages/urllib3/connectionpool.py", line 1010, in _validate_conn
    conn.connect()
  File "/usr/local/Cellar/dvc/2.5.0/libexec/lib/python3.9/site-packages/urllib3/connection.py", line 411, in connect
    self.sock = ssl_wrap_socket(
  File "/usr/local/Cellar/dvc/2.5.0/libexec/lib/python3.9/site-packages/urllib3/util/ssl_.py", line 449, in ssl_wrap_socket
    ssl_sock = _ssl_wrap_socket_impl(
  File "/usr/local/Cellar/dvc/2.5.0/libexec/lib/python3.9/site-packages/urllib3/util/ssl_.py", line 493, in _ssl_wrap_socket_impl
    return ssl_context.wrap_socket(sock, server_hostname=server_hostname)
  File "/usr/local/Cellar/python@3.9/3.9.6/Frameworks/Python.framework/Versions/3.9/lib/python3.9/ssl.py", line 500, in wrap_socket
    return self.sslsocket_class._create(
  File "/usr/local/Cellar/python@3.9/3.9.6/Frameworks/Python.framework/Versions/3.9/lib/python3.9/ssl.py", line 1040, in _create
    self.do_handshake()
  File "/usr/local/Cellar/python@3.9/3.9.6/Frameworks/Python.framework/Versions/3.9/lib/python3.9/ssl.py", line 1309, in do_handshake
    self._sslobj.do_handshake()
urllib3.exceptions.SSLError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: self signed certificate (_ssl.c:1129)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/Cellar/dvc/2.5.0/libexec/lib/python3.9/site-packages/dvc/remote/base.py", line 36, in wrapper
    func(from_info, to_info, *args, **kwargs)
  File "/usr/local/Cellar/dvc/2.5.0/libexec/lib/python3.9/site-packages/dvc/fs/base.py", line 233, in upload
    self._upload(  # noqa, pylint: disable=no-member
  File "/usr/local/Cellar/dvc/2.5.0/libexec/lib/python3.9/site-packages/dvc/fs/s3.py", line 160, in wrapper
    raise translate_boto_error(exc)
  File "/usr/local/Cellar/dvc/2.5.0/libexec/lib/python3.9/site-packages/dvc/fs/s3.py", line 156, in wrapper
    func(*args, **kwargs)
  File "/usr/local/Cellar/dvc/2.5.0/libexec/lib/python3.9/site-packages/dvc/fs/s3.py", line 210, in _upload
    obj.upload_file(
  File "/usr/local/Cellar/dvc/2.5.0/libexec/lib/python3.9/site-packages/boto3/s3/inject.py", line 278, in object_upload_file
    return self.meta.client.upload_file(
  File "/usr/local/Cellar/dvc/2.5.0/libexec/lib/python3.9/site-packages/boto3/s3/inject.py", line 129, in upload_file
    return transfer.upload_file(
  File "/usr/local/Cellar/dvc/2.5.0/libexec/lib/python3.9/site-packages/boto3/s3/transfer.py", line 279, in upload_file
    future.result()
  File "/usr/local/Cellar/dvc/2.5.0/libexec/lib/python3.9/site-packages/s3transfer/futures.py", line 106, in result
    return self._coordinator.result()
  File "/usr/local/Cellar/dvc/2.5.0/libexec/lib/python3.9/site-packages/s3transfer/futures.py", line 265, in result
    raise self._exception
  File "/usr/local/Cellar/dvc/2.5.0/libexec/lib/python3.9/site-packages/s3transfer/tasks.py", line 126, in __call__
    return self._execute_main(kwargs)
  File "/usr/local/Cellar/dvc/2.5.0/libexec/lib/python3.9/site-packages/s3transfer/tasks.py", line 150, in _execute_main
    return_value = self._main(**kwargs)
  File "/usr/local/Cellar/dvc/2.5.0/libexec/lib/python3.9/site-packages/s3transfer/upload.py", line 692, in _main
    client.put_object(Bucket=bucket, Key=key, Body=body, **extra_args)
  File "/usr/local/Cellar/dvc/2.5.0/libexec/lib/python3.9/site-packages/botocore/client.py", line 357, in _api_call
    return self._make_api_call(operation_name, kwargs)
  File "/usr/local/Cellar/dvc/2.5.0/libexec/lib/python3.9/site-packages/botocore/client.py", line 662, in _make_api_call
    http, parsed_response = self._make_request(
  File "/usr/local/Cellar/dvc/2.5.0/libexec/lib/python3.9/site-packages/botocore/client.py", line 682, in _make_request
    return self._endpoint.make_request(operation_model, request_dict)
  File "/usr/local/Cellar/dvc/2.5.0/libexec/lib/python3.9/site-packages/botocore/endpoint.py", line 102, in make_request
    return self._send_request(request_dict, operation_model)
  File "/usr/local/Cellar/dvc/2.5.0/libexec/lib/python3.9/site-packages/botocore/endpoint.py", line 136, in _send_request
    while self._needs_retry(attempts, operation_model, request_dict,
  File "/usr/local/Cellar/dvc/2.5.0/libexec/lib/python3.9/site-packages/botocore/endpoint.py", line 253, in _needs_retry
    responses = self._event_emitter.emit(
  File "/usr/local/Cellar/dvc/2.5.0/libexec/lib/python3.9/site-packages/botocore/hooks.py", line 356, in emit
    return self._emitter.emit(aliased_event_name, **kwargs)
  File "/usr/local/Cellar/dvc/2.5.0/libexec/lib/python3.9/site-packages/botocore/hooks.py", line 228, in emit
    return self._emit(event_name, kwargs)
  File "/usr/local/Cellar/dvc/2.5.0/libexec/lib/python3.9/site-packages/botocore/hooks.py", line 211, in _emit
    response = handler(**kwargs)
  File "/usr/local/Cellar/dvc/2.5.0/libexec/lib/python3.9/site-packages/botocore/retryhandler.py", line 183, in __call__
    if self._checker(attempts, response, caught_exception):
  File "/usr/local/Cellar/dvc/2.5.0/libexec/lib/python3.9/site-packages/botocore/retryhandler.py", line 250, in __call__
    should_retry = self._should_retry(attempt_number, response,
  File "/usr/local/Cellar/dvc/2.5.0/libexec/lib/python3.9/site-packages/botocore/retryhandler.py", line 277, in _should_retry
    return self._checker(attempt_number, response, caught_exception)
  File "/usr/local/Cellar/dvc/2.5.0/libexec/lib/python3.9/site-packages/botocore/retryhandler.py", line 316, in __call__
    checker_response = checker(attempt_number, response,
  File "/usr/local/Cellar/dvc/2.5.0/libexec/lib/python3.9/site-packages/botocore/retryhandler.py", line 222, in __call__
    return self._check_caught_exception(
  File "/usr/local/Cellar/dvc/2.5.0/libexec/lib/python3.9/site-packages/botocore/retryhandler.py", line 359, in _check_caught_exception
    raise caught_exception
  File "/usr/local/Cellar/dvc/2.5.0/libexec/lib/python3.9/site-packages/botocore/endpoint.py", line 200, in _do_get_response
    http_response = self._send(request)
  File "/usr/local/Cellar/dvc/2.5.0/libexec/lib/python3.9/site-packages/botocore/endpoint.py", line 269, in _send
    return self.http_session.send(request)
  File "/usr/local/Cellar/dvc/2.5.0/libexec/lib/python3.9/site-packages/botocore/httpsession.py", line 341, in send
    raise SSLError(endpoint_url=request.url, error=e)
botocore.exceptions.SSLError: SSL validation failed for https://MYURL:9000/dvc/Dens/ac/13f137d80741105dc543f08858c4ae [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: self signed certificate (_ssl.c:1129)

My remote config looks like this:

['remote "truenas"']
    url = s3://dvc/Dens
    jobs = 8
    endpointurl = https://MYURL:9000/
    ssl_verify = false
    use_ssl = false