Skip to content

Add exact-match uri filter to GET /assets endpoint#69489

Open
seanmuth wants to merge 1 commit into
apache:mainfrom
seanmuth:restore-asset-uri-exact-lookup
Open

Add exact-match uri filter to GET /assets endpoint#69489
seanmuth wants to merge 1 commit into
apache:mainfrom
seanmuth:restore-asset-uri-exact-lookup

Conversation

@seanmuth

@seanmuth seanmuth commented Jul 6, 2026

Copy link
Copy Markdown
Contributor

What

Adds a uri query parameter to GET /api/v2/assets that matches an asset by its exact URI, backed by a new single-column index on asset.uri.

Why

Today, resolving a single asset when you already know its full URI (for example, discovering an asset_id before posting to /assets/{asset_id}/events for cross-deployment dependencies) requires uri_pattern. That parameter compiles to ILIKE '%...%', which cannot use a B-tree index and degrades to a full table scan on large asset tables — the API docs themselves warn about this.

The Airflow 2 REST API offered a fast exact lookup via GET /datasets/{uri} (later GET /assets/{uri}), but it was not carried forward when the endpoint migrated to FastAPI under AIP-84 — the single-resource route became GET /assets/{asset_id} (integer PK only). This PR restores the exact-URI lookup as a query parameter.

Passing the URI as a query parameter rather than a path segment also avoids the URI-encoding problems path parameters have with the /, :, and %2F characters that asset URIs contain.

What changed

  • New uri exact-match filter on GET /assets (equality comparison via the existing filter_param_factory).
  • New non-unique index idx_asset_uri on asset.uri (the existing unique index leads with name and cannot serve uri-only lookups), with an Alembic migration.
  • Regenerated OpenAPI spec + UI client, updated REVISION_HEADS_MAP and migration ref doc.
  • Tests: exact-match hits plus negative cases confirming a substring does not match (unlike uri_pattern).

Was generative AI tooling used to co-author this PR?
  • Yes — Claude Code (Opus 4.8, 1M context)

Generated-by: Claude Code (Opus 4.8, 1M context) following the guidelines


Drafted-by: Claude Code (Opus 4.8, 1M context) (no human review before posting)


Important

🛠️ Maintainer triage note for @seanmuth · by @potiuk · 2026-07-08 15:38 UTC

Helpful heads-up from the maintainers — please address before this PR can be reviewed:

  • Pre-commit / static checks failing (CI image checks / Static checks). See the contributor guide.

Full criteria: Pull Request quality criteria.

The ball is in your court — you've been assigned to this PR. Fix the above, then mark it Ready for review.

Automated triage — may be imperfect; a maintainer takes the next look.

@boring-cyborg boring-cyborg Bot added area:API Airflow's REST/HTTP API area:db-migrations PRs with DB migration area:UI Related to UI/UX. For Frontend Developers. kind:documentation labels Jul 6, 2026
@seanmuth seanmuth force-pushed the restore-asset-uri-exact-lookup branch from d549e9b to 5416302 Compare July 6, 2026 18:54

@pierrejeambrun pierrejeambrun left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the PR, makes sense!

A few suggestions bellow :)

Comment thread airflow-core/docs/migrations-ref.rst Outdated
Comment thread airflow-core/newsfragments/69489.improvement.rst Outdated
Comment thread airflow-core/src/airflow/api_fastapi/common/parameters.py
Comment thread airflow-core/src/airflow/api_fastapi/common/parameters.py
Comment thread airflow-core/src/airflow/models/asset.py Outdated
@pierrejeambrun pierrejeambrun added this to the Airflow 3.4.0 milestone Jul 7, 2026
@seanmuth

seanmuth commented Jul 7, 2026

Copy link
Copy Markdown
Contributor Author

Thanks for the review! Pushed 3c0560208b addressing all the points:

  • Target version — retargeted the migration to 3.4.0: renamed to 0124_3_4_0_…, airflow_version="3.4.0", restored the 3.3.0 head and added a 3.4.0 entry in REVISION_HEADS_MAP, and updated the migration ref doc.
  • Newsfragment — removed; agreed this isn't a behaviour change that needs a user warning.
  • Description — trimmed to just the exact-match / indexed-equality note, dropped the comparison to the search params (they have their own docs).
  • Multiple valuesuri now accepts repeated params (?uri=a&uri=b) via FilterParam[list[str]] + ANY_EQUAL + default_factory=list, following the QueryTIStateFilter pattern. Added a test covering it. Agreed a list endpoint accepting multiple values is the right shape here.
  • ORM comment — removed the API-layer reference from the model; kept the index (thanks for confirming 2.x had one).

Regenerated the OpenAPI spec + UI client. Locally the asset filter tests (14) and a fresh-DB up → down → up migration check both pass.


Drafted-by: Claude Code (Opus 4.8, 1M context) (no human review before posting)

@seanmuth seanmuth force-pushed the restore-asset-uri-exact-lookup branch from 3c05602 to 24b2e92 Compare July 7, 2026 15:56
@seanmuth

seanmuth commented Jul 7, 2026

Copy link
Copy Markdown
Contributor Author

Heads-up on CI: the only red check is CI image checks / Static checks, and it's unrelated to this PR. The failing hook is update-providers-build-files, which regenerates provider packaging files (apache/kafka, apache/flink, github, zendesk, microsoft/mssql, …) — this branch touches zero provider files (it's scoped to airflow-core assets API + a migration). It's pre-existing drift on main that any airflow-core PR triggering the --all-files static run inherits.

Everything substantive is green: the asset filter tests, mypy, ruff, migration/revision-heads checks, and the full test matrix all pass (138 passing). I also validated the change end-to-end on a local 3.2.1 env and a real Astro deployment (Runtime 3.2-4): exact ?uri= returns the single matching asset, repeated ?uri=a&uri=b returns both, a partial value correctly returns nothing, and uri_pattern still does its substring match.

This is ready for another look. Thanks again for the review!


Drafted-by: Claude Code (Opus 4.8, 1M context) (no human review before posting)

@pierrejeambrun pierrejeambrun added type:improvement Changelog: Improvements backport-to-v3-3-test Backport to v3-3-test and removed type:improvement Changelog: Improvements labels Jul 8, 2026
@pierrejeambrun pierrejeambrun added type:improvement Changelog: Improvements and removed backport-to-v3-3-test Backport to v3-3-test labels Jul 8, 2026

@pierrejeambrun pierrejeambrun left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice, thanks

LGTM, needs a rebase to fix conflicts, unrelated CI failures should go away too.

Resolving a single asset by its full URI currently requires uri_pattern,
which compiles to ILIKE '%...%' and cannot use a database index, so it
degrades to a full table scan on large asset tables. This restores the
fast, index-backed exact-URI lookup that the Airflow 2 REST API exposed
via GET /datasets/{uri} but that was not carried forward when the endpoint
moved to FastAPI under AIP-84.

The uri query parameter matches assets by exact URI using an equality
comparison and accepts repeated values (?uri=a&uri=b) to resolve several
assets in one call, backed by a new single-column index on asset.uri (the
existing unique index leads with name and cannot serve uri-only lookups).
Passing it as a query parameter rather than a path segment avoids the
URI-encoding problems that path parameters have with the '/' and ':'
characters in asset URIs.
@seanmuth seanmuth force-pushed the restore-asset-uri-exact-lookup branch from 24b2e92 to ff3f5ec Compare July 8, 2026 18:07
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area:API Airflow's REST/HTTP API area:db-migrations PRs with DB migration area:UI Related to UI/UX. For Frontend Developers. kind:documentation type:improvement Changelog: Improvements

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants