Latest release of requests causes urllib3 to throw an error

[Rach81]

A previously working lambda function started throwing this error.

{
"errorMessage": "Unable to import module 'app': cannot import name 'DEFAULT_CIPHERS' from 'urllib3.util.ssl_' (/var/task/urllib3/util/ssl_.py)",
"errorType": "Runtime.ImportModuleError",
"requestId": "30bf7245-a58f-4192-81e3-5d122fb31d11",
"stackTrace": []
}

While trying to debug the issue, I narrowed it down to being a problem with requests (new release yesterday 5/3/2023). I then tried referencing the prior release which fixed the problem and my function worked as expected again.

To reproduce the error.

requirements,txt > requests- uses the latest release (2.30.0) and causes the lambda function to throw the error above.
requirements,txt > requests==2.29.0 - uses the prior release (2.29.0). With this release the error above no longer occurs.

[nateprewitt]

Hi @Rach81,

This is unrelated to Requests. You're installing a version of urllib3 that's incompatible with the version of Boto3 bundled in your runtime. You'll either need to pin your urllib3 dependency to urllib3<2 or rely on the one provided by Lambda.

[Rach81]

I'm not directly installing urllib3 anywhere. I fixed the issue by reverting to the prior version of requests from 4/26 . That was the only change I made.

[nateprewitt]

Upgrading Requests to 2.30.0 allows for the use of previously released urllib3 2.0. If you're using something like pip download to create your Lambda package, it will eagerly pull the latest available version (urllib3 2.0.3).

Prior to Requests 2.30.0, we actively blocked the use of this urllib3 version. Now that this is no longer happening, you may need to constrain your dependencies for certain environments such as Lambda until they update to a newer version of Boto3.

[jsondai]

+1

[amir-mks]

Same issue - worked in the morning, stopped working in the afternoon. could be related to vs-code update.

Any help would be greatly appreciated

[nateprewitt]

I'll try to further clarify the comments above since there's some confusion. The solution if you rely on one of the breaking changes with urllib3 2.0's release is to pin your dependencies to urllib3<2 as noted in the release notes. Long term this will be less hassle than trying to pin an older version of Requests.

Depending on how you deploy/install your code, this may look something like:

pip installation

python -m pip install requests "urllib3<2"

requirements.txt

requests
urllib3<2

pyproject.toml

[project]
name = "myproject"
...
dependencies = [
    "requests",
    "urllib3<2",
]

poetry dependencies

[tool.poetry.dependencies]
requests = "^2.30.0"
urllib3 = "^1.26"

[Rach81]

I'll try to further clarify the comments above since there's some confusion. The solution if you rely on one of the breaking changes with urllib3 2.0's release is to pin your dependencies to urllib3<2 as noted in the release notes. Long term this will be less hassle than trying to pin an older version of Requests.

Depending on how you deploy/install your code, this may look something like:

pip installation

python -m pip install requests "urllib3<2"

requirements.txt

requests
urllib3<2

pyproject.toml

[project]
name = "myproject"
...
dependencies = [
    "requests",
    "urllib3<2",
]

poetry dependencies

[tool.poetry.dependencies]
requests = "^2.30.0"
urllib3 = "^1.26"

I see your point, but ultimately the solution is to pin an outdated version of something. Whether that's requests or urllib3... Unless I am misunderstanding something?

[nateprewitt]

urllib3 1.26.x is continuing to receive security updates for the near future. Pinning at the Requests level will block you from receiving security patches. The underlying issue is at the urllib3 level which ideally is where it's constrained.

Otherwise, you're relying on Requests to implicitly be the barrier from other dependencies bringing in urllib3 2.0.

[apfromiit]

boto/botocore#2926

[emma125dgggd]

File "/home/appuser/venv/lib/python3.9/site-packages/streamlit/runtime/scriptrunner/script_runner.py", line 565, in _run_script
exec(code, module.dict)
File "main.py", line 35, in
import pyrebase
File "/home/appuser/venv/lib/python3.9/site-packages/pyrebase/init.py", line 1, in
from .pyrebase import initialize_app
File "/home/appuser/venv/lib/python3.9/site-packages/pyrebase/pyrebase.py", line 19, in
from requests.packages.urllib3.contrib.appengine import is_appengine_sandbox

[deepak-joshi1-nutanix]

downgrading to V2.29.0 fixed the issue as of now.

[mase-git]

Hi guys any news about the requests and urllib dependencies?
I have a lot of lambda with the latest version of requests that are affected by the same CIPHER dependencies bug.
When I build my lambda, the requirements.txt got the latest version available (not explicit version are defined, so I suppose that the latest version is triggered)

[nateprewitt]

@mase-git There isn't anything to be solved in Requests or urllib3. Requests 2.31.0 works with both major versions of urllib3.

If you're using dependencies that don't support the breaking changes in urllib3 2.0, you can pin to urllib3 1.x as detailed when this issue was first opened. That will resolve any incompatibilities.

[eighthave]

I have an issue that doesn't look directly related based on this thread, but it is also fixed by forcing urllib3<2. This doesn't seem to match the urllib3 porting guide, so I'm including it here:

https://gitlab.com/fdroid/sdkmanager/-/jobs/4372611166

Traceback (most recent call last):
  File "/usr/local/lib/python3.9/dist-packages/urllib3/response.py", line 705, in _error_catcher
    yield
  File "/usr/local/lib/python3.9/dist-packages/urllib3/response.py", line 830, in _raw_read
    raise IncompleteRead(self._fp_bytes_read, self.length_remaining)
urllib3.exceptions.IncompleteRead: IncompleteRead(23392014 bytes read, -11696007 more expected)

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/local/lib/python3.9/dist-packages/requests/models.py", line 816, in generate
    yield from self.raw.stream(chunk_size, decode_content=True)
  File "/usr/local/lib/python3.9/dist-packages/urllib3/response.py", line 935, in stream
    data = self.read(amt=amt, decode_content=decode_content)
  File "/usr/local/lib/python3.9/dist-packages/urllib3/response.py", line 906, in read
    data = self._raw_read(amt)
  File "/usr/local/lib/python3.9/dist-packages/urllib3/response.py", line 830, in _raw_read
    raise IncompleteRead(self._fp_bytes_read, self.length_remaining)
  File "/usr/lib/python3.9/contextlib.py", line 135, in __exit__
    self.gen.throw(type, value, traceback)
  File "/usr/local/lib/python3.9/dist-packages/urllib3/response.py", line 722, in _error_catcher
    raise ProtocolError(f"Connection broken: {e!r}", e) from e
urllib3.exceptions.ProtocolError: ('Connection broken: IncompleteRead(23392014 bytes read, -11696007 more expected)', IncompleteRead(23392014 bytes read, -11696007 more expected))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/lib/python3.9/unittest/mock.py", line 1769, in _inner
    return f(*args, **kw)
  File "/builds/fdroid/sdkmanager/./test_sdkmanager.py", line 282, in test_install_and_rerun
    sdkmanager.main()
  File "/builds/fdroid/sdkmanager/sdkmanager.py", line 1199, in main
    method(args.packages)
  File "/builds/fdroid/sdkmanager/sdkmanager.py", line 1034, in install
    download_file(url, zipball)
  File "/builds/fdroid/sdkmanager/sdkmanager.py", line 593, in download_file
    r = requests.get(url, stream=True, allow_redirects=True, headers=HTTP_HEADERS)
  File "/usr/local/lib/python3.9/dist-packages/requests/api.py", line 73, in get
    return request("get", url, params=params, **kwargs)
  File "/usr/local/lib/python3.9/dist-packages/requests/api.py", line 59, in request
    return session.request(method=method, url=url, **kwargs)
  File "/usr/local/lib/python3.9/dist-packages/requests_cache/session.py", line 159, in request
    return super().request(method, url, *args, headers=headers, **kwargs)  # type: ignore
  File "/usr/local/lib/python3.9/dist-packages/requests/sessions.py", line 587, in request
    resp = self.send(prep, **send_kwargs)
  File "/usr/local/lib/python3.9/dist-packages/requests_cache/session.py", line 206, in send
    response = self._send_and_cache(request, actions, cached_response, **kwargs)
  File "/usr/local/lib/python3.9/dist-packages/requests_cache/session.py", line 234, in _send_and_cache
    self.cache.save_response(response, actions.cache_key, actions.expires)
  File "/usr/local/lib/python3.9/dist-packages/requests_cache/backends/base.py", line 89, in save_response
    cached_response = CachedResponse.from_response(response, expires=expires)
  File "/usr/local/lib/python3.9/dist-packages/requests_cache/models/response.py", line 102, in from_response
    obj.raw = CachedHTTPResponse.from_response(response)
  File "/usr/local/lib/python3.9/dist-packages/requests_cache/models/raw_response.py", line 60, in from_response
    response.content  # This property reads, decodes, and stores response content
  File "/usr/local/lib/python3.9/dist-packages/requests/models.py", line 899, in content
    self._content = b"".join(self.iter_content(CONTENT_CHUNK_SIZE)) or b""
  File "/usr/local/lib/python3.9/dist-packages/requests/models.py", line 818, in generate
    raise ChunkedEncodingError(e)
requests.exceptions.ChunkedEncodingError: ('Connection broken: IncompleteRead(23392014 bytes read, -11696007 more expected)', IncompleteRead(23392014 bytes read, -11696007 more expected))

[mase-git]

@mase-git There isn't anything to be solved in Requests or urllib3. Requests 2.31.0 works with both major versions of urllib3.

If you're using dependencies that don't support the breaking changes in urllib3 2.0, you can pin to urllib3 1.x as detailed when this issue was first opened. That will resolve any incompatibilities.

{ "errorMessage": "Unable to import module 'X: cannot import name 'DEFAULT_CIPHERS' from 'urllib3.util.ssl_' (/var/task/urllib3/util/ssl_.py)", "errorType": "Runtime.ImportModuleError", "requestId": "XXXX", "stackTrace": [] }

I am using boto3-1.26.140 and botocore-1.29.140 with python 3.9, no requirements.txt are defined.
I continue to have this error, any idea about the cause of the problem? Is it any correlation about external libraries such as boto3 as specified above?

In addition, if no requirements.txt was specified and dependencies didn't solve it automatically with no forcing versioning, the problem should be the library dependencies binding. So, it should be a problem of the library.

Generally, industrial applications trust on libraries for the stable of the latest versions, what happened with requests and urllib compatibles? There is no excuse about old dependencies ends to be compatible due to the "unstable" old version. Am I wrong?

[Kongstad]

I don't know if this has any value to the content of this thread. But I just wanted to say that today, as of june 6th my code suddenly stopped working. Your solution of:
python -m pip install requests "urllib3<2"
Did the job for me.

[Sky-Syed]

I am still getting this error when running lambda.
cannot import name 'DEFAULT_CIPHERS' from 'urllib3.util.ssl_' (/opt/python/urllib3/util/ssl_.py)",

[tool.poetry.dependencies]
python = "^3.9"
requests = "^2.30.0"
urllib3 = "^1.26.12"

[vumdao]

I got this error today after re-deploying my lambda function

• Python runtime 3.9
• Requests=2.31.0

Error

[ERROR] Runtime.ImportModuleError: Unable to import module 'index': cannot import name 'DEFAULT_CIPHERS' from 'urllib3.util.ssl_' (/var/task/urllib3/util/ssl_.py)
Traceback (most recent call last):

[Sky-Syed]

I have managed to resolve this issue.

This can be resolved in two ways:
1- Use the existing lambda layer that is working in your AWS environment.
2- Use the urllib3==1.26.16 before your request layer in lambda.

[sebbbrbrr]

Hi @Rach81,

This is unrelated to Requests. You're installing a version of urllib3 that's incompatible with the version of Boto3 bundled in your runtime. You'll either need to pin your urllib3 dependency to urllib3<2 or rely on the one provided by Lambda.

could you please explain me on discord a way to pin your dependancies,since i'm in need of a program that gives me this specific error(and i need it to get my google drive backup of whatsapp since i lost everything)

[i] Searching...

Requesting access to Google...
Traceback (most recent call last):
File "C:\Users\Sebbi\Desktop\whapa-master\libs\whagodri.py", line 553, in
wa_backup = WaBackup(**getConfigs())
^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\Sebbi\Desktop\whapa-master\libs\whagodri.py", line 43, in init
token = gpsoauth.perform_master_login(email=gmail, password=password, android_id=android_id)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\Sebbi\Desktop\whapa-master\libs\gpsoauth_init_.py", line 138, in perform_master_login
return perform_auth_request(data, proxy)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\Sebbi\Desktop\whapa-master\libs\gpsoauth_init.py", line 76, in perform_auth_request
session.mount(AUTH_URL, AuthHTTPAdapter())
^^^^^^^^^^^^^^^^^
File "C:\Users\Sebbi\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCache\local-packages\Python311\site-packages\requests\adapters.py", line 155, in init
self.init_poolmanager(pool_connections, pool_maxsize, block=pool_block)
File "C:\Users\Sebbi\Desktop\whapa-master\libs\gpsoauth_init.py", line 66, in init_poolmanager
context.set_ciphers(ssl_.DEFAULT_CIPHERS)
^^^^^^^^^^^^^^^^^^^^
AttributeError: module 'urllib3.util.ssl_' has no attribute 'DEFAULT_CIPHERS'

[merwok]

You should use pip-tools or poetry to record dependencies.
Or see https://packaging.python.org/en/latest/tutorials/managing-dependencies/

[nateprewitt]

Resolving along with #6432 and locking it to avoid losing context with further comments. Please read #6432 (comment) if you still have questions after reading this thread.

Of particular note here:

• Requests 2.30.0+ supports urllib3 2.0, however awscli/boto3/botocore do not currently. Please follow the steps linked above to resolve any conflicts.
• Upgrading to the latest version of the awscli/boto3/botocore will resolve the DEFAULT_CIPHERS error which was patched in April 2023. If you're receiving this error, you're either using an outdated version of one of these libraries or another library using DEFAULT_CIPHERS. This issue is unrelated to Requests.