fix migration: update legacy relatedTerms in glossaryTerm version history after the glossary term realtion changes

[sonika-shah]

Summary

Fixes #27802 .Extends PR #26586. That fix cleaned glossary_term_entity of legacy relatedTerms data but missed the version history in entity_extension, so GET /glossaryTerms/{id}/versions/{v} still 500s on any pre-1.13 term with non-empty relatedTerms:

UnrecognizedPropertyException: Unrecognized field "id" (class TermRelation)

What this PR does

Adds a Java migration migrateGlossaryTermVersionRelatedTermsToTermRelation (v1130) that transforms each legacy EntityReference[] item into the new TermRelation[] shape — { "term": <ref>, "relationType": "relatedTo" } — at:

1. the top-level relatedTerms field of each entity_extension version history, and
2. the JSON value strings inside changeDescription.fieldsAdded[*].newValue and fieldsDeleted[*].oldValue, so version-history diffs continue to render strikethrough/green for related-term changes.

The earlier draft of this PR did a SQL strip; that has been replaced with the Java migration. The SQL files now contain a comment pointer to the Java migration.

Why transform instead of strip

Version reads (EntityRepository.getVersion) deserialize the snapshot JSON directly — no entity_relationship rehydration, no setFields repopulation. A strip would silently lose every related-term that ever existed at any historical version. The transform preserves that history exactly.

The live-row strip in glossary_term_entity (from #26586) stays as-is; that read path does rehydrate from entity_relationship.

Scale

Designed for tables with 10M+ rows:

• Pre-filters at SQL using extension_index + JSON-side legacy-shape predicate (relatedTerms[0].id exists in mysql / jsonb_exists in postgres). Continuous-migration cost on a fully-transformed install: a single short-circuiting index scan, no JSON evaluation, no Java parsing.
• Keyset pagination on PK (id, extension) — no OFFSET cost.
• PreparedBatch updates (1 round-trip per 500 rows).
• Per-row try/catch isolates malformed snapshots; cursor advances even on caught errors so a poison-pill row can't loop forever.
• Idempotent: re-runs on already-transformed rows do shape-detection-then-skip with zero DB writes.

Test plan

• Seeded a 1.12.7 install with 500 glossary terms across glossary → root → child → grandchild levels, 3 update passes each, ~1500 version history(s) in legacy EntityReference[] shape (script: scripts/seed_legacy_related_terms_v1130.py).
• Upgraded to 1.13.0; migration ran cleanly: pages=3 transformed=1285 skipped=0 with zero failures.
• Verified zero data loss in DB: items per snapshot match pre-migration counts; every wrapped item has term + relationType: "relatedTo"; term sub-objects preserve id, name, fullyQualifiedName, type, description, displayName, deleted.
• GET /glossaryTerms/{id}/versions/{v} returns 200 with TermRelation[] shape across root, child, and grandchild terms.
• Version-history UI renders correctly with strikethrough/green diff annotations for related-term changes on covered snapshots.

[Copilot]

Pull request overview

Extends the existing 1.13.0 post-data-migration to also remove legacy relatedTerms data from glossary term version snapshots stored in entity_extension, preventing Jackson deserialization failures (500s) when requesting historical term versions.

Changes:

• Add a Postgres migration step to strip relatedTerms from entity_extension rows with extension matching glossaryTerm.version.%.
• Add a MySQL migration step to strip relatedTerms from the same entity_extension version snapshot rows.

Reviewed changes

Copilot reviewed 2 out of 2 changed files in this pull request and generated 2 comments.

File	Description
bootstrap/sql/migrations/native/1.13.0/postgres/postDataMigrationSQLScript.sql	Removes relatedTerms from glossary term version snapshot JSON in entity_extension for Postgres.
bootstrap/sql/migrations/native/1.13.0/mysql/postDataMigrationSQLScript.sql	Removes relatedTerms from glossary term version snapshot JSON in entity_extension for MySQL.

[github-actions]

🟡 Playwright Results — all passed (11 flaky)

✅ 3963 passed · ❌ 0 failed · 🟡 11 flaky · ⏭️ 86 skipped

Shard	Passed	Failed	Flaky	Skipped
🟡 Shard 1	298	0	1	4
🟡 Shard 2	739	0	5	8
🟡 Shard 3	747	0	1	7
✅ Shard 4	759	0	0	18
✅ Shard 5	687	0	0	41
🟡 Shard 6	733	0	4	8

🟡 11 flaky test(s) (passed on retry)

• Pages/UserCreationWithPersona.spec.ts › Create user with persona and verify on profile (shard 1, 1 retry)
• Features/ActivityAPI.spec.ts › Activity event is created when description is updated (shard 2, 1 retry)
• Features/ActivityAPI.spec.ts › Activity event shows the actor who made the change (shard 2, 1 retry)
• Features/DataQuality/ColumnLevelTests.spec.ts › Column Value Median To Be Between (shard 2, 1 retry)
• Features/DomainFilterQueryFilter.spec.ts › Domain filter should persist across page navigation (shard 2, 1 retry)
• Features/DomainFilterQueryFilter.spec.ts › Domain filter should use exact match and prefix with dot to prevent false positives (shard 2, 1 retry)
• Features/RTL.spec.ts › Verify Following widget functionality (shard 3, 1 retry)
• Pages/Lineage/DataAssetLineage.spec.ts › Column lineage for mlModel -> container (shard 6, 1 retry)
• Pages/Lineage/LineageFilters.spec.ts › Verify lineage schema filter selection (shard 6, 1 retry)
• Pages/ServiceEntity.spec.ts › Tier Add, Update and Remove (shard 6, 1 retry)
• Pages/UserDetails.spec.ts › Create team with domain and verify visibility of inherited domain in user profile after team removal (shard 6, 1 retry)

📦 Download artifacts

How to debug locally

# Download playwright-test-results-<shard> artifact and unzip
npx playwright show-trace path/to/trace.zip    # view trace

[Copilot]

Pull request overview

Copilot reviewed 5 out of 5 changed files in this pull request and generated 2 comments.

[gitar-bot]

Code Review ✅ Approved 1 resolved / 1 findings

Updates the migration to transform legacy relatedTerms in glossaryTerm version snapshots, resolving the MySQL SQL stripping issue that previously defeated the Java transformation.

✅ 1 resolved

✅ Bug: MySQL SQL still strips relatedTerms, defeating the Java transform

📄 bootstrap/sql/migrations/native/1.13.0/mysql/postDataMigrationSQLScript.sql:85-90
The MySQL postDataMigrationSQLScript.sql still contains the old SQL UPDATE entity_extension SET json = JSON_REMOVE(json, '$.relatedTerms') at lines 87-90. This SQL runs before the Java migration (runDataMigration is called after SQL scripts), so it will strip all relatedTerms from version snapshots before the Java code ever gets a chance to transform them.

The Postgres file correctly removed the equivalent SQL (visible in the diff as deleted lines), but the MySQL file only added the comment without removing the old SQL. This means:

• On MySQL: related-term history is silently lost (the exact outcome the PR aims to prevent).
• On Postgres: works as intended (transform preserves history).

Remove lines 85-90 from the MySQL SQL file, matching what was done for Postgres.

Options

Display: compact → Showing less information.

Comment with these commands to change:

Compact

gitar display:verbose

_{Was this helpful? React with 👍 / 👎 | Gitar}

[sonarqubecloud]

Quality Gate passed for 'open-metadata-ingestion'

Issues
0 New issues
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube Cloud

[sonika-shah]

Cherrypicked to 1.13