Chardet is used, when it is available, not when `[use-chardet-on-py3]`-extra is installed

[sedrubal]

First: IMHO this is related to but not a duplicate of #5871 #7222 #7219

requests tries to use chardet when it can be imported. However, IMHO it should only use it, if requests was installed with the [use-chardet-on-py3]-extra.

As reported in #7219, requests shows a warning, when the version of chardet does not fulfill the requirements. However, package managers like uv won't respect the version requirement of the dependency of requests on chardet, when the extra is not installed (see astral-sh/uv#18179). If another project dependency depends on chardet, it will be installed, but the version of chardet might not fulfill the requirement of requests.

The same applies to charset_normalizer or any other optional dependency.

Expected Result

chardet should not be used, when requests is not installed with [use-chardet-on-py3]-extra.

Actual Result

requests tries to use chardet and generates a log message complaining about an incompatible version.

Reproduction Steps

uv init
# I use fixed versions to make the example more reproducible:
uv add tox==4.44.0 requests==2.32.5

cat > test.py <<EOF
import logging

LOGGER = logging.getLogger()
LOGGER.setLevel(logging.DEBUG)

import chardet
import requests
import tox

print("tox", tox.__version__)
print("requests", requests.__version__)
print("chardet", chardet.__version__)
EOF
uv run test.py

Output:

.venv/lib/python3.12/site-packages/requests/__init__.py:113: RequestsDependencyWarning: urllib3 (2.6.3) or chardet (6.0.0.post1)/charset_normalizer (3.4.4) doesn't match a supported version!
  warnings.warn(
tox 4.44.0
requests 2.32.5
chardet 6.0.0.post1

System Information

$ python -m requests.help

{
  "chardet": {
    "version": "6.0.0.post1"
  },
  "charset_normalizer": {
    "version": "3.4.4"
  },
  "cryptography": {
    "version": ""
  },
  "idna": {
    "version": "3.11"
  },
  "implementation": {
    "name": "CPython",
    "version": "3.12.3"
  },
  "platform": {
    "release": "6.17.0-14-generic",
    "system": "Linux"
  },
  "pyOpenSSL": {
    "openssl_version": "",
    "version": null
  },
  "requests": {
    "version": "2.32.5"
  },
  "system_ssl": {
    "version": "300000d0"
  },
  "urllib3": {
    "version": "2.6.3"
  },
  "using_charset_normalizer": false,
  "using_pyopenssl": false
}

[sigmavirus24]

There is no way to know what extra was installed, last I had checked so the only way to know what to use is whether it is importable or not.

[mahek2016]

Thanks for the clarification that installation extras can't be detected at runtime.

Given that limitation, would it make sense to improve the warning message to clarify that chardet is being used simply because it is importable in the environment, not because the extra was explicitly installed?

This might help reduce confusion for users working with alternative package managers like uv.

[sigmavirus24]

It has nothing to do with package managers... What about the message is unclear today or implied it's the fault if the package manager?

[sedrubal]

My problem is, that requests complains about unsupported versions of dependencies and my first thought was, why did my package manager fail to resolve them correctly.

[sigmavirus24]

Package managers (all of them I've used in python) do not protect you from doing an install today, and an upgrade later that violates a constraint of something already installed. To do that would be slow, even for uv. This is why pipx came into being and many other tools.

This is also why some projects are defensive - we can only do so much to prevent breakage but some end users disregard that. The warning is our best ability to potentially warn you that something is unexpected in your system.

[sedrubal]

The checking and warning itself is not the problem. I think, the problem is, that noone did any mistake:

• the package manager did resolve all dependencies correctly. They follow all PEPs how to install packages.
• I, as developer, specified the dependencies correctly and I didn't even mess up with the environment
• requests tries the best as it can to find out if it can use chardet or not.

I was not aware of the shortcoming of python packaging that packages can't recognize the installed extras. I did some short research and realized that you're right. There seems to be some ongoing work on this problem. But I'm not sure if there is any solution yet and if you say so, I believe you that there is no solution yet.

Given this situation, the only things that come into my mind and could be done are:

• the message could be more verbose and a bit clearer and maybe link to some background information. I was not aware of the licensing problem and that there is even an extra for requests. And still, I don't know if I need that extra to requests working properly.
• requests could be more strict: if the versions of the requirements are not fulfilled, it could assume that the extra is not installed and chardet can't be used at all.
• Developers could just install this extra. Then, package manager can resolve the dependencies correctly. However, developers must be aware that this extra exists (I was not).
• requests can of course support the new versions of chardet but I'm sure this will happen sooner or later. And the problem is that this underlaying issue is not solved and might happen any time again.

In any case, thanks for your work and the explanations. If you think, you can't do anything, you can close this issue.

[sigmavirus24]

To be clear, we already raised the cap. We need to perform a release but that will eliminate this warning.

[maitrey]

Dear Maintainers,

We use conan and are also affected since chardet got released : https://github.com/conan-io/conan/blob/develop2/conans/requirements.txt#L1. We are aware of the complexities of open source development, but from our perspective having an estimate of when a fix might be released, would be incredibly valuable for our internal planning. Are there any thoughts on a potential timeline, even though a rough one, that could perhaps be shared?

We are happy to assist with testing or provide further details regarding the issue if that helps in any way.

Thanks for your effort on requests.

[sigmavirus24]

We don't work on this full time, and all work is best effort. So a reasonable estimate is somewhere between in the next month to the next 5 years. I can't be more specific than that

[nateprewitt]

Sigmavirus24 is correct that we cannot reasonably detect this. We've merged a new PR bringing in chardet 7.0 in #7239. Assuming this goes well, this resolves Apache's original licensing concerns that strong-armed charset_normalizer into Requests. At that point, we will likely move back to a single character detection library, our original dependency chardet. That will remove any ambiguity with installs and this extra will become a no-op.

As for release timelines, we never commit to anything publicly that we cannot uphold. I do think we can reasonably say there will be a release of Requests this quarter though.