make `urllib3` an optional dependency?

[jameslamb]

Description

urllib3 is a required dependency of distributed:

distributed/pyproject.toml

Line 43 in c813e6d

"urllib3 >= 1.26.5",

As far as I can tell, the only direct use of it is for --preload-ing scripts stored at a remote URL (described at https://docs.dask.org/en/stable/customize-initialization.html):

distributed/distributed/preloading.py

Lines 132 to 134 in c813e6d

	# This is the only place where urrllib3 is used and it is a relatively heavy
	# import. Do lazy import to reduce import time
	import urllib3

Would you consider making it an optional dependency of distributed?

Benefits of this work

• reduced risk of dependency conflicts between distributed and other projects
• reduced risk of distributed transitively being the source of findings from CVE scanners (libraries like urrlib3 involved in passing data over networks are popular targets for security researchers)
• faster installs of distributed

I'll note that none of distributed's other strong dependencies appear to pull in urllib3, so making it optional here really would have all these benefits for some environments.

$ docker run --rm -it python:3.14 bash
$ pip install pipdeptree 'distributed>=2026.3.0'
$ pipdeptree
distributed==2026.3.0
├── click [required: >=8.0, installed: 8.4.0]
├── cloudpickle [required: >=3.0.0, installed: 3.1.2]
├── dask [required: >=2026.3.0,<2026.3.1, installed: 2026.3.0]
│   ├── click [required: >=8.1, installed: 8.4.0]
│   ├── cloudpickle [required: >=3.0.0, installed: 3.1.2]
│   ├── fsspec [required: >=2021.09.0, installed: 2026.4.0]
│   ├── packaging [required: >=20.0, installed: 26.2]
│   ├── partd [required: >=1.4.0, installed: 1.4.2]
│   │   ├── locket [required: Any, installed: 1.0.0]
│   │   └── toolz [required: Any, installed: 1.1.0]
│   ├── PyYAML [required: >=5.3.1, installed: 6.0.3]
│   └── toolz [required: >=0.12.0, installed: 1.1.0]
├── Jinja2 [required: >=2.10.3, installed: 3.1.6]
│   └── MarkupSafe [required: >=2.0, installed: 3.0.3]
├── locket [required: >=1.0.0, installed: 1.0.0]
├── msgpack [required: >=1.0.2, installed: 1.1.2]
├── packaging [required: >=20.0, installed: 26.2]
├── psutil [required: >=5.8.0, installed: 7.2.2]
├── PyYAML [required: >=5.4.1, installed: 6.0.3]
├── sortedcontainers [required: >=2.0.5, installed: 2.4.0]
├── tblib [required: >=1.6.0,!=3.2.1,!=3.2.0, installed: 3.2.2]
├── toolz [required: >=0.12.0, installed: 1.1.0]
├── tornado [required: >=6.2.0, installed: 6.5.5]
├── urllib3 [required: >=1.26.5, installed: 2.7.0]
└── zict [required: >=3.0.0, installed: 3.0.0]
pip==25.3
pipdeptree==2.35.3
└── packaging [required: >=26, installed: 26.2]

Acceptance Criteria

• distributed does not have a hard dependency on urllib3
• preloading from a URL still works (if maintainers decide that's worth preserving) if urllib3 is installed, and raises an informative error otherwise

Approach

N/A - will defer to maintainers

This would be a breaking change for users who rely on --preload-ing from a URL, but I think it could be acceptable in exchange for the benefits I listed above.

Notes

I'd be happy to put together a PR for this if there is interest.

Thanks for your time and consideration.

[TomAugspurger]

#3755, for a bit of background.

IMO, making it optional is worth the breakage. We can raise with an informative error message when we reach that spot and find that urllib3 isn't present.

[jrbourbeau]

No strong thoughts on this topic from me. Though alternatively we could replace the urllib3 code with something that only uses the Python standard library. Something like

from urllib.error import HTTPError
from urllib.request import Request, urlopen
import time

retry_statuses = {429, 502, 503, 504}
backoff_factor = 0.2
max_retries = 10

request = Request(url, method="GET")

for attempt in range(max_retries + 1):
    try:
        with urlopen(request) as response:
            source = response.read()
        break
    except HTTPError as exc:
        if exc.code not in retry_statuses or attempt == max_retries:
            raise

        delay = backoff_factor * (2 ** attempt)
        time.sleep(delay)

[jameslamb]

I like that suggestion @jrbourbeau

I'd be happy to put up a PR if you all are open to it.

[prince8273]

Hey everyone,
I like the stdlib direction here too. Replacing the current urllib3 usage with urllib.request seems like a cleaner approach than introducing optional dependency handling, especially if this is the only direct usage today.

If nobody has started working on it yet, I’d be interested in putting together a PR for this.

[jameslamb]

I will put up a PR.

[jameslamb]

I put up #9273, with changes very similar to what @jrbourbeau suggested.

[crusaderky]

Worth noting that urllib3 remains an optional dependency of fsspec (through requests)