Skip to content

Conversation

Copy link
Contributor

Copilot AI commented Nov 21, 2025

Describe your changes:

Fixes issue with PostgreSQL logical replication failures

PostgreSQL logical replication requires tables to have REPLICA IDENTITY configured for UPDATE/DELETE operations. Without it, migrations fail with:

ERROR: cannot update table "tag" because it does not have a replica identity and publishes updates

This affects installations using AWS RDS with read replicas or any PostgreSQL setup with logical replication enabled.

Changes made:

  • Base schema (postgres.sql): Added REPLICA IDENTITY for 45 tables using existing unique indexes
  • Flyway v001 migration: Added REPLICA IDENTITY for 27 tables immediately after creation to ensure availability before UPDATE/DELETE operations in v006
  • Flyway v016 migration: Updated to only handle 12 tables created after v001 (in migrations v002-v015)
  • Native migration (1.11.1/postgres/schemaChanges.sql): Updates REPLICA IDENTITY to use namehash/fqnhash constraints after schema changes

Implementation details:

  • Migration order fix: REPLICA IDENTITY now set in v001 (before v006's UPDATE statements)
  • v001 uses original constraint names (e.g., bot_entity_name_key, tag_fullyqualifiedname_key)
  • v016 handles tables created in v002-v015 with appropriate constraint names
  • Native 1.11.1 updates to namehash/fqnhash constraints after schema migrations
  • Uses ALTER TABLE IF EXISTS for backward compatibility

Migration execution order:

  1. Flyway v001: Creates tables + sets REPLICA IDENTITY ✓
  2. Flyway v006: UPDATE statements work (no longer fails) ✓
  3. Flyway v016: Sets REPLICA IDENTITY for post-v001 tables
  4. Native migrations: Schema changes (name→namehash, FQN→fqnhash)
  5. Native 1.11.1: Updates REPLICA IDENTITY for new constraints

Tested with PostgreSQL 14: verified INDEX/FULL replica identity types, UPDATE/DELETE operations, and correct migration execution order.

Type of change:

  • Bug fix

Checklist:

  • I have read the CONTRIBUTING document.
  • My PR title is Fixes <issue-number>: <short explanation>
  • I have commented on my code, particularly in hard-to-understand areas.
  • For JSON Schema changes: I updated the migration scripts or explained why it is not needed.
  • I have added a test that covers the exact scenario we are fixing. For complex issues, comment the issue number in the test for future reference.
Original prompt

This section details on the original issue you should resolve

<issue_title>Flyway Migrations Do Not Work With PostgreSQL Replication</issue_title>
<issue_description>Affected module
It impacts installation of OpenMetadata in K8s.

Describe the bug
A clear and concise description of what the bug is.

When installing OpenMetadata using a PostgreSQL DB that has replication enabled the flyway scripts will fail with many errors like this.

Message : ERROR: cannot update table "tag" because it does not have a replica identity and publishes updates
--
Tue, Aug 15 2023 12:32:27 pm | Hint: To enable updating the table, set REPLICA IDENTITY using ALTER TABLE.
Tue, Aug 15 2023 12:32:27 pm | Location : /opt/openmetadata/bootstrap/sql/org.postgresql.Driver/v006__create_db_connection_info.sql (/opt/openmetadata/bootstrap/sql/org.postgresql.Driver/v006__create_db_connection_info.sql)
Tue, Aug 15 2023 12:32:27 pm | Line : 55
Tue, Aug 15 2023 12:32:27 pm | Statement : UPDATE tag
Tue, Aug 15 2023 12:32:27 pm | SET json = JSONB_SET(json::jsonb, '{provider}', '"system"', true)
Tue, Aug 15 2023 12:32:27 pm | WHERE fullyQualifiedName in ('PersonalData.Personal', 'PersonalData.SpecialCategory',
Tue, Aug 15 2023 12:32:27 pm | 'PII.None', 'PII.NonSensitive', 'PII.Sensitive',
Tue, Aug 15 2023 12:32:27 pm | 'Tier.Tier1', 'Tier.Tier2', 'Tier.Tier3', 'Tier.Tier4', 'Tier.Tier5')

This can be fixed as follows.

ALTER TABLE tag REPLICA IDENTITY USING INDEX tag_fullyqualifiedname_key;

To Reproduce

Simply try to do a fresh install on a PostgreSQL instance that has replication enabled. In this case the DB was an AWS RDS instance with a read replica.

Screenshots or steps to reproduce

Expected behavior
A clear and concise description of what you expected to happen.

Version:

  • OS: [e.g. iOS]
  • Python version:
  • OpenMetadata version: 1.1.1
  • OpenMetadata Ingestion package version: 1.1.1.1

Additional context
Add any other context about the problem here.
</issue_description>

<agent_instructions>make sure to look boostrap/migrations for both flyway and native and make necessary changes to support postgres replication</agent_instructions>

Comments on the Issue (you are @copilot in this section)

@harshach @AMcManigal which version of Postgres are you running?

💬 We'd love your input! Share your thoughts on Copilot coding agent in our 2 minute survey.

Copilot AI and others added 4 commits November 21, 2025 23:48
Co-authored-by: harshach <38649+harshach@users.noreply.github.com>
Co-authored-by: harshach <38649+harshach@users.noreply.github.com>
Co-authored-by: harshach <38649+harshach@users.noreply.github.com>
Co-authored-by: harshach <38649+harshach@users.noreply.github.com>
Copilot AI changed the title [WIP] Fix Flyway migrations not working with PostgreSQL replication Fix PostgreSQL logical replication support by adding REPLICA IDENTITY Nov 22, 2025
Copilot AI requested a review from harshach November 22, 2025 00:00
@harshach
Copy link
Collaborator

@copilot I believe this changes nees to be inline with all the migration files. If we add it at the end we execute thsi file at the end meanwhile earlier migrations will throw an error because of lack of syntax with replica identity

- Add REPLICA IDENTITY statements at end of v001 for all tables created in v001
- Update v016 to only handle tables created after v001 (v002-v015)
- This fixes the issue where v006 UPDATE statements fail because v016 runs too late
- Native migration 1.11.1 remains unchanged (updates to namehash/fqnhash constraints)

Co-authored-by: harshach <38649+harshach@users.noreply.github.com>
Copy link
Contributor Author

Copilot AI commented Nov 23, 2025

@copilot I believe this changes nees to be inline with all the migration files. If we add it at the end we execute thsi file at the end meanwhile earlier migrations will throw an error because of lack...

Fixed in commit 861844c. Moved REPLICA IDENTITY setup from v016 to v001, right after table creation. This ensures REPLICA IDENTITY is set before v006's UPDATE statements run. v016 now only handles tables created in v002-v015.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Flyway Migrations Do Not Work With PostgreSQL Replication

2 participants