Allow FQN-based lineage APIs#28599
Conversation
github-actions
Bot
added
backend
safe to test
![gitar-bot[bot]](https://avatars.githubusercontent.com/in/827041?s=60&v=4){kind=small}
There was a problem hiding this comment.
Pull request overview
Enables AddLineageRequest (and related lineage endpoints) to accept references by fullyQualifiedName in addition to id. The server resolves references via Entity.getEntityReference, new FQN-suffixed routes are added for get/patch/delete/delete-by-source, the Python ingestion and SDK build FQN-based lineage payloads, and the global EntityReference JSON schema is relaxed to no longer require id. UI generated TypeScript types are regenerated to reflect id?: string.
Changes:
- Backend: relax
EntityReferenceschema, addresolveLineageReferencehelper, add FQN-based lineage routes (/getLineageEdge/name/..., PATCH/DELETE name routes, delete-by-source-by-name). - Ingestion:
OMetaLineageMixinnow keys cache and builds edge paths by id or FQN; SQL lineage emits FQN-only references; SDK exposes*_by_namevariants. - Tests: new
LineageRepositoryTestcases for resolve-by-FQN and missing-id+FQN; new IT cases covering FQN-only add, id precedence, FQN-based patch/delete, and delete-by-source-by-name.
Reviewed changes
Copilot reviewed 9 out of 233 changed files in this pull request and generated 2 comments.
Show a summary per file
| File | Description |
|---|---|
| openmetadata-spec/.../entityReference.json | Drops id from required so FQN-only refs validate. |
| openmetadata-service/.../LineageRepository.java | Adds resolve/validate helpers, FQN variants of getEdge/patchEdge/deleteBySource. |
| openmetadata-service/.../LineageResource.java | New FQN routes and shared authorizeLineageReference; modifies existing delete-by-name path. |
| openmetadata-service/.../LineageRepositoryTest.java | Unit tests for FQN resolution + missing-ref error. |
| openmetadata-integration-tests/.../LineageResourceIT.java | IT coverage for FQN add/patch/delete and delete-by-source. |
| ingestion/.../lineage_mixin.py | Cache keys/paths support id or FQN; add/patch/delete handle both forms. |
| ingestion/.../metadata_rest.py | Dispatch delete-by-source by id or FQN based on reference. |
| ingestion/.../sql_lineage.py | SQL lineage now uses FQN instead of id when building edges. |
| ingestion/src/metadata/sdk/api/lineage.py | New add_lineage_by_name / delete_lineage_by_name / delete_lineage_by_source_by_name (sync+async). |
| openmetadata-ui/.../generated/**/* (many) | Codegen updates: id: string → id?: string on EntityReference. |
Jest test CoverageUI tests summary
|
🔴 Playwright Results — 3 failure(s), 16 flaky✅ 4253 passed · ❌ 3 failed · 🟡 16 flaky · ⏭️ 88 skipped
Genuine Failures (failed on all attempts)❌
|
harshach
changed the title
|
Code Review ✅ Approved 5 resolved / 5 findingsAdds path-based FQN lineage APIs to enable direct lineage operations without UUID resolution. This update resolves several issues, including silent lineage failures, inconsistent identifier handling, and crashes during ingestion. ✅ 5 resolved✅ Edge Case: Mixed id/FQN references cause silent lineage operation failures
✅ Quality: SDK lineage.py uses excessive None assignments for EntityReference
✅ Edge Case: _delete_lineage_by_source_reference silently skips if no identifier
✅ Bug: write_fqn_lineage crashes with AttributeError when lineage returns None
✅ Bug: add_lineage_by_name returns None on success when get_lineage_by_name fails
OptionsDisplay: compact → Showing less information. Comment with these commands to change:
Was this helpful? React with 👍 / 👎 | Gitar |
|
Describe your changes:
Issue number was not provided in the PR request.
Adds path-based FQN lineage APIs so callers can add, fetch, patch, delete, and delete-by-source lineage without resolving entity UUIDs first. The global
EntityReferenceschema remains unchanged andPUT /v1/lineagecontinues to use id-basedAddLineageRequestreferences.Python SQL lineage ingestion now carries table FQNs through
OMetaFQNLineageRequestand writes lineage through the name-based API, avoiding extraget_by_namecalls just to build UUID-backed lineage edges.Type of change:
High-level design:
Path-based APIs are explicit and have no id/FQN precedence ambiguity:
PUT /v1/lineage/{fromEntity}/name/{fromFQN}/{toEntity}/name/{toFQN}GET /v1/lineage/getLineageEdge/{fromEntity}/name/{fromFQN}/{toEntity}/name/{toFQN}PATCH /v1/lineage/{fromEntity}/name/{fromFQN}/{toEntity}/name/{toFQN}DELETE /v1/lineage/{fromEntity}/name/{fromFQN}/{toEntity}/name/{toFQN}DELETE /v1/lineage/source/name/{entityType}/{entityFQN}/type/{lineageSource}The server resolves path FQNs with existing entity name resolution, authorizes lineage edits against the resolved entities, and keeps body-based
AddLineageRequestbehavior id-based.Tests:
Use cases covered
EntityReferenceschema and generated UI types are not relaxed.Unit tests
openmetadata-service/src/test/java/org/openmetadata/service/jdbi3/LineageRepositoryTest.java, ingestion unit tests for FQN lineage request generation.Backend integration tests
openmetadata-integration-tests/for new/changed API endpoints.openmetadata-integration-tests/src/test/java/org/openmetadata/it/tests/LineageResourceIT.java.Ingestion integration tests
Playwright (UI) tests
Manual testing performed
py_compilefor touched ingestion files.mvn -pl openmetadata-service spotless:applyandmvn -pl openmetadata-service spotless:check.mvn -T 1C -pl openmetadata-service -Dtest=LineageRepositoryTest test.UI screen recording / screenshots:
Not applicable; no UI behavior changes.
Checklist:
Fixes <issue-number>: <short explanation>Fixes #<issue-number>above.