[YSQL] Support Create Publication #18930
Assignees
Labels
area/cdcsdk
CDC SDK
area/ysql
Yugabyte SQL (YSQL)
kind/enhancement
This is an enhancement of an existing feature
priority/high
High Priority
Comments
dr0pdb
added
kind/enhancement
dr0pdb
changed the title
yugabyte-ci
added
priority/high
dr0pdb
added a commit
that referenced
this issue
Sep 27, 2023
Summary:
This diff enables the CRUD syntax for Publications in YSQL. This change is part of the project to add a YSQL API for CDC via the PG logical replication mechanism.
In PG, a Publication determines which tables should be streamed via logical replication. A user can choose to create a Publication on:
1. All tables of a database
2. All tables of a schema
3. List of tables explicitly provided in the command
In YB, a Publication will be equivalent to the configuration of a CDC stream i.e. it will specify which tables the user is interested in streaming via CDC.
The information about Publication will be kept in PG system catalog tables and the schema will be **same** as PG:
1. pg_publication: Metadata of a Publication
2. pg_publication_rel: (puboid, reloid) pair for each Publication-Table pair
3. pg_publication_tables: View over pg_publication_rel which expands the list of tables if the Publication was created for all the tables (future) of a database
The only difference between the YSQL and PG semantics are due to a few CDC limitations.
**Limitations**
- CDC does not support choosing a subset of records (insert, update, delete, truncate). So we return an error with `ERRCODE_FEATURE_NOT_SUPPORTED` error code if the user tries to specify that either during `Create` or `Alter`
- CDC does not support tables without user defined primary key. This is handled differently depending on whether the Publication is being created on all tables (`FOR ALL TABLES`) or an explicit list (`FOR TABLE t1, t2`) is provided
- `FOR ALL TABLES`: These tables are skipped. We also filter them out in the `pg_publication_tables` view
- `FOR TABLE t1, t2, ...`: We return an `ERRCODE_FEATURE_NOT_SUPPORTED` error
**Upgrade\Rollback safety:**
These changes cannot be rolled back. As a result, all the commands are disabled during upgrade using an autoflag `yb_enable_replication_commands` (LocalPersisted) and will only be enabled once the user has committed to the new version.
The autoflag is `LocalPersisted` since the YSQL API changes would modify the yb-master sys-catalog information for CDC stream in the future for Replication Slot. So for consistency, we will disable all CDC YSQL API commands during upgrade.
Jira: DB-7781, DB-7784, DB-7782
Test Plan:
New regress tests
`./yb_build.sh --java-test 'org.yb.pgsql.TestPgRegressPublication'`
Reviewers: skumar, aagrawal, asrinivasan, dsrinivasan
Reviewed By: dsrinivasan
Subscribers: hsunder, jason, yql
Differential Revision: https://phorge.dev.yugabyte.com/D28721
dr0pdb
added a commit
that referenced
this issue
Jan 7, 2024
Summary: `yb_enable_replication_commands` is a TEST flag that controls the replication commands (Publication and Replication Slot) in YSQL. It does not work in the YSQL layer correctly since we only push down flags with tag `kPg` and prefix `ysql_` to PG. As a result of this bug, the publication commands were always enabled regardless of the flag value. The Replication Slot syntax wasn't affected since we also do sanity checks in yb-master. Publication commands do not make an RPC to yb-master, hence they were affected by the bug. This issue is only on the master branch since we haven't backported publication commands to any stable branches. This revision makes `yb_enable_replication_commands` a runtime PG preview flag and adds tests to validate that it works correctly. Jira: DB-7781 Test Plan: ./yb_build.sh --cxx-test pg_wrapper-test --gtest_filter PgWrapperFlagsTest.VerifyGFlagRuntimeTag ./yb_build.sh --cxx-test cdcsdk_ysql-test --gtest_filter CDCSDKYsqlTest.TestPgPublicationDisabled ./yb_build.sh --cxx-test pgwrapper_pg_catalog_version-test --gtest_filter PgCatalogVersionTest.IncrementAllDBCatalogVersions ./yb_build.sh --java-test 'org.yb.pgsql.TestPgRegressPublication' Other CDCSDK tests Reviewers: hsunder, skumar, xCluster, asrinivasan Reviewed By: hsunder Subscribers: ybase, ycdcxcluster, yql, bogdan Differential Revision: https://phorge.dev.yugabyte.com/D31245
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Jira Link: DB-7781
Description
Support
CREATE PUBLICATIONfor managing the configuration of a CDCSDK stream.Warning: Please confirm that this issue does not contain any sensitive information
The text was updated successfully, but these errors were encountered: