CI Failure (ssl failure wihle connecting to vectorized-public s3 bucket) in ClusterConfigLegacyDefaultTest.test_legacy_default_explicit_after_upgrade

[vbotbuildovich]

https://buildkite.com/redpanda/vtools/builds/13901

Module: rptest.tests.cluster_config_test
Class: ClusterConfigLegacyDefaultTest
Method: test_legacy_default_explicit_after_upgrade
Arguments: {
    "wipe_cache": false
}

test_id:    ClusterConfigLegacyDefaultTest.test_legacy_default_explicit_after_upgrade
status:     FAIL
run time:   7.162 seconds

SSLError(MaxRetryError("HTTPSConnectionPool(host='vectorized-public.s3.us-west-2.amazonaws.com', port=443): Max retries exceeded with url: /releases/redpanda/23.1.21/redpanda-23.1.21-amd64.tar.gz (Caused by SSLError(SSLError(1, '[SSL] EVP lib (_ssl.c:1007)')))"))
Traceback (most recent call last):
  File "/opt/.ducktape-venv/lib/python3.10/site-packages/urllib3/connectionpool.py", line 670, in urlopen
    httplib_response = self._make_request(
  File "/opt/.ducktape-venv/lib/python3.10/site-packages/urllib3/connectionpool.py", line 381, in _make_request
    self._validate_conn(conn)
  File "/opt/.ducktape-venv/lib/python3.10/site-packages/urllib3/connectionpool.py", line 978, in _validate_conn
    conn.connect()
  File "/opt/.ducktape-venv/lib/python3.10/site-packages/urllib3/connection.py", line 362, in connect
    self.sock = ssl_wrap_socket(
  File "/opt/.ducktape-venv/lib/python3.10/site-packages/urllib3/util/ssl_.py", line 386, in ssl_wrap_socket
    return context.wrap_socket(sock, server_hostname=server_hostname)
  File "/usr/lib/python3.10/ssl.py", line 513, in wrap_socket
    return self.sslsocket_class._create(
  File "/usr/lib/python3.10/ssl.py", line 1104, in _create
    self.do_handshake()
  File "/usr/lib/python3.10/ssl.py", line 1375, in do_handshake
    self._sslobj.do_handshake()
ssl.SSLError: [SSL] EVP lib (_ssl.c:1007)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/opt/.ducktape-venv/lib/python3.10/site-packages/requests/adapters.py", line 439, in send
    resp = conn.urlopen(
  File "/opt/.ducktape-venv/lib/python3.10/site-packages/urllib3/connectionpool.py", line 726, in urlopen
    retries = retries.increment(
  File "/opt/.ducktape-venv/lib/python3.10/site-packages/urllib3/util/retry.py", line 446, in increment
    raise MaxRetryError(_pool, url, error or ResponseError(cause))
urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='vectorized-public.s3.us-west-2.amazonaws.com', port=443): Max retries exceeded with url: /releases/redpanda/23.1.21/redpanda-23.1.21-amd64.tar.gz (Caused by SSLError(SSLError(1, '[SSL] EVP lib (_ssl.c:1007)')))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/opt/.ducktape-venv/lib/python3.10/site-packages/ducktape/tests/runner_client.py", line 184, in _do_run
    data = self.run_test()
  File "/opt/.ducktape-venv/lib/python3.10/site-packages/ducktape/tests/runner_client.py", line 276, in run_test
    return self.test_context.function(self.test)
  File "/opt/.ducktape-venv/lib/python3.10/site-packages/ducktape/mark/_mark.py", line 535, in wrapper
    return functools.partial(f, *args, **kwargs)(*w_args, **w_kwargs)
  File "/home/ubuntu/redpanda/tests/rptest/services/cluster.py", line 103, in wrapped
    r = f(self, *args, **kwargs)
  File "/home/ubuntu/redpanda/tests/rptest/tests/cluster_config_test.py", line 2014, in test_legacy_default_explicit_after_upgrade
    versions = self.load_version_range(self.legacy_version)
  File "/home/ubuntu/redpanda/tests/rptest/tests/redpanda_test.py", line 143, in load_version_range
    v = self.redpanda._installer.highest_from_prior_feature_version(v)
  File "/home/ubuntu/redpanda/tests/rptest/services/redpanda_installer.py", line 511, in highest_from_prior_feature_version
    if not self._avail_for_download(v) and skip_versions > 0:
  File "/home/ubuntu/redpanda/tests/rptest/services/redpanda_installer.py", line 446, in _avail_for_download
    r = requests.head(self._version_package_url(version))
  File "/opt/.ducktape-venv/lib/python3.10/site-packages/requests/api.py", line 104, in head
    return request('head', url, **kwargs)
  File "/opt/.ducktape-venv/lib/python3.10/site-packages/requests/api.py", line 61, in request
    return session.request(method=method, url=url, **kwargs)
  File "/opt/.ducktape-venv/lib/python3.10/site-packages/requests/sessions.py", line 530, in request
    resp = self.send(prep, **send_kwargs)
  File "/opt/.ducktape-venv/lib/python3.10/site-packages/requests/sessions.py", line 643, in send
    r = adapter.send(request, **kwargs)
  File "/opt/.ducktape-venv/lib/python3.10/site-packages/requests/adapters.py", line 514, in send
    raise SSLError(e, request=request)
requests.exceptions.SSLError: HTTPSConnectionPool(host='vectorized-public.s3.us-west-2.amazonaws.com', port=443): Max retries exceeded with url: /releases/redpanda/23.1.21/redpanda-23.1.21-amd64.tar.gz (Caused by SSLError(SSLError(1, '[SSL] EVP lib (_ssl.c:1007)')))

JIRA Link: CORE-3051

[vbotbuildovich]

*https://buildkite.com/redpanda/vtools/builds/14148

[vbotbuildovich]

*https://buildkite.com/redpanda/vtools/builds/14148

[travisdowns]

Probably a duplicate of #18252 ?

[travisdowns]

Perhaps S3 is just rejecting the request due to rate limits or whatever and this is how the error is surfaced?