Skip to content

2.25.1.0-b261

2.25.1.0-b261: [#24077] xCluster: preserve enum label OIDs when replicating DDLs
@mdbridge mdbridge tagged this 30 Jan 00:30 
Summary:
The diff adds the machinery necessary to preserve enum label IDs across universes when DDLs are replicated in automatic mode.

In particular,
  * when a CREATE TYPE AS ENUM is performed on the source universe, the DDL replication extension captures the resulting enum OID, enum label-> label OID mapping via an event trigger and includes this in the JSON put in the DDLs to replicate table.
  * on the target universe, the replicating DDL handler takes that JSON and passes it down to the Postgres backend via the following new function:
    * `yb_xcluster_set_next_oid_assignments`
    * that function is used to pass the assignment before the replicated DDL is executed
    * afterwards, the assignment is cleared
  * at the Postgres backend level, that function parses the JSON and saves away the assignment
  * when the CREATE TYPE AS ENUM is actually run, the code uses the assignment instead of just allocating fresh OIDs.

Similar functionality exists for ALTER TYPE as applied to enums.  Here are the complete final label assignment is passed but only one of the labels will actually be created.

As a consequence of this, as long as the enums on both sides start with identical label OIDs (the automatic DDL bootstrap process ensures this), any new enums or alterations to enums will preserve the fact that both sides have identical label OIDs.

Note that most of the code refers to labels via a enum OID, label name pair instead of just the label name.  This will be needed soon when a single DDL is allowed to refer to multiple enums (think CREATE EXTENSION), but in the meantime we cheat slightly:
```
	/*
	 * Currently we do not ensure that enums have the same *pg_type*
	 * OIDs.  We will fix that later, but in the meantime we take
	 * advantage of the fact that we currently never have a
	 * replicating DDL that refers to two different enums to ignore
	 * the actual enum OID field when comparing labels.
	 */
```

Miscellaneous:
  * the copyright notices violate the Postgres max number of columns convention; I left that assuming the copyrights might be automatically checked at some point
  * I have added defensive code in case the JSON gets corrupted in transit; this code is unlikely to ever be triggered
  * we no longer consider CREATE TYPE or ALTER TYPE as being simply passed through; this means that CREATE EXTENSION can no longer be replicated if it contains one of those statements.

FYI, for the nonregular Postgres developer, HTAB is the Postgres hash table data structure.

https://wiki.postgresql.org/wiki/HashTable gives example usage.   I am using HASH_STRINGS instead of that example's HASH_BLOB; this changes the hashing and key comparison algorithms to ignore data in the key after the first zero byte.

Fixes #24077
Jira: DB-12970

Test Plan:
Un-disabled XClusterPgRegressDDLReplicationTest.PgRegressCreateDropEnum, which carefully tests that enum label OIDs are preserved as we replicate DDLs.

That test was introduced and explained in https://phorge.dev.yugabyte.com/D41267.

Added a few test cases to that test based on implementation experience.

Added a new end-to-end test, XClusterDDLReplicationTest.CreateTableWithEnum, which verifies that we can indeed replicate rows containing enum values.

Verify the new test by turning off preserving OIDs and saw that it fails.

Modified tests to reflect the fact that CREATE EXTENSION no longer works for extensions that use CREATE TYPE inside.

To give you a sense of what the JSON looks like, the following is the beginning of the DDL queue table for the XClusterPgRegressDDLReplicationTest.PgRegressCreateDropEnum test:
```
I0122 14:04:14.004925 10259 xcluster_ddl_replication_test_base.cc:217] DDL Queue Table:
1737583447108063	408313532	{"user": "yugabyte", "query": "CREATE TYPE rainbow AS ENUM ('red', 'orange', 'yellow', 'green', 'blue', 'purple');", "schema": "public", "version": 1, "command_tag": "CREATE TYPE", "enum_label_info": [{"label": "red", "enum_oid": 16405, "label_oid": 16406}, {"label": "orange", "enum_oid": 16405, "label_oid": 16408}, {"label": "yellow", "enum_oid": 16405, "label_oid": 16410}, {"label": "green", "enum_oid": 16405, "label_oid": 16412}, {"label": "blue", "enum_oid": 16405, "label_oid": 16414}, {"lab
1737583447357244	393499029	{"user": "yugabyte", "query": "CREATE TYPE planets AS ENUM ( 'venus', 'earth', 'mars' );", "schema": "public", "version": 1, "command_tag": "CREATE TYPE", "enum_label_info": [{"label": "venus", "enum_oid": 16418, "label_oid": 16420}, {"label": "earth", "enum_oid": 16418, "label_oid": 16422}, {"label": "mars", "enum_oid": 16418, "label_oid": 16424}]}
1737583447401956	986195536	{"user": "yugabyte", "query": "ALTER TYPE planets ADD VALUE 'uranus';", "schema": "public", "version": 1, "command_tag": "ALTER TYPE", "enum_label_info": [{"label": "venus", "enum_oid": 16418, "label_oid": 16420}, {"label": "earth", "enum_oid": 16418, "label_oid": 16422}, {"label": "mars", "enum_oid": 16418, "label_oid": 16424}, {"label": "uranus", "enum_oid": 16418, "label_oid": 16426}]}
1737583447435555	92333102	{"user": "yugabyte", "query": "ALTER TYPE planets ADD VALUE 'mercury' BEFORE 'venus';", "schema": "public", "version": 1, "command_tag": "ALTER TYPE", "enum_label_info": [{"label": "venus", "enum_oid": 16418, "label_oid": 16420}, {"label": "earth", "enum_oid": 16418, "label_oid": 16422}, {"label": "mars", "enum_oid": 16418, "label_oid": 16424}, {"label": "uranus", "enum_oid": 16418, "label_oid": 16426}, {"label": "mercury", "enum_oid": 16418, "label_oid": 16427}]}
```

```
ybd  --java-test 'org.yb.pgsql.TestYsqlUpgrade#migratingIsEquivalentToReinitdb'
```

Reviewers: jhe, xCluster, hsunder

Reviewed By: jhe

Subscribers: myang, ybase, yql

Differential Revision: https://phorge.dev.yugabyte.com/D41413
Assets 2
Loading