Skip to content

fix(deps): raise tldextract floor to >=5.3.0 to avoid AttributeError on same-domain enqueue#2031

Merged
vdusek merged 1 commit into
apify:masterfrom
anxkhn:fix/tldextract-min-version
Jul 7, 2026
Merged

fix(deps): raise tldextract floor to >=5.3.0 to avoid AttributeError on same-domain enqueue#2031
vdusek merged 1 commit into
apify:masterfrom
anxkhn:fix/tldextract-min-version

Conversation

@anxkhn

@anxkhn anxkhn commented Jul 7, 2026

Copy link
Copy Markdown
Contributor

_domain_under_public_suffix in src/crawlee/_utils/urls.py reads the
top_domain_under_public_suffix property of tldextract's ExtractResult:

return _get_tld_extractor().extract_str(host).top_domain_under_public_suffix

That property was added in tldextract 5.3.0 (2025-04-21), but the dependency
constraint in pyproject.toml is still tldextract>=5.1.0. Any environment that
resolves tldextract into the permitted 5.1.0-5.2.x range therefore raises

AttributeError: 'ExtractResult' object has no attribute 'top_domain_under_public_suffix'

the first time registrable-domain matching runs, i.e. on the same-domain
enqueue strategy (_matches_enqueue_strategy). The bundled uv.lock pins a
newer tldextract, so CI always runs a version that has the property and never
exercises the older versions the metadata still allows.

This bumps the floor to >=5.3.0 to match the code. top_domain_under_public_suffix
is the only tldextract result property used anywhere in src/, so 5.3.0 is
exactly the version that introduced the API the code relies on. It is a
metadata-only change; the currently resolved tldextract already satisfies the
new floor, so uv.lock needs no re-resolution beyond the mirrored specifier
line.

Issues

  • Not applicable (no existing issue).

Testing

  • uv lock --check passes (lockfile stays consistent with the new floor).
  • uv run pytest tests/unit/_utils/test_urls.py -> 28 passed, including the
    same-domain parametrizations that exercise the property.
  • uv run poe lint passes (ruff format check + ruff check).
  • No new test: the change is pure dependency metadata and cannot be driven
    red->green against the resolved tldextract (which already has the property);
    the same-domain path is already covered by the existing URL filter tests.

Checklist

  • CI passed

`_domain_under_public_suffix` in `src/crawlee/_utils/urls.py` reads the
`top_domain_under_public_suffix` property of tldextract's `ExtractResult`,
which was only added in tldextract 5.3.0. The dependency constraint still
allowed `tldextract>=5.1.0`, so installing a resolved tldextract in the
5.1.0-5.2.x range raises `AttributeError: 'ExtractResult' object has no
attribute 'top_domain_under_public_suffix'` on the `same-domain` enqueue
strategy (and anywhere else registrable-domain matching runs). The bundled
lockfile resolves a newer tldextract, so CI never exercises the old versions
the metadata permits. Bump the floor to `>=5.3.0` to match the code.

Signed-off-by: Anas Khan <83116240+anxkhn@users.noreply.github.com>
@vdusek vdusek changed the title fix(deps): raise tldextract floor to >=5.3.0 fix(deps): raise tldextract floor to >=5.3.0 to avoid AttributeError on same-domain enqueue Jul 7, 2026

@vdusek vdusek left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@vdusek vdusek merged commit 8850a86 into apify:master Jul 7, 2026
35 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants