Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

cdc: separate schema and topic prefix for changefeed namespace #56026

Closed
amruss opened this issue Oct 27, 2020 · 2 comments · Fixed by #59710
Closed

cdc: separate schema and topic prefix for changefeed namespace #56026

amruss opened this issue Oct 27, 2020 · 2 comments · Fixed by #59710
Assignees
Labels
A-cdc Change Data Capture C-bug Code not up to spec/doc, specs & docs deemed correct. Solution expected to change code/behavior. T-cdc T-disaster-recovery

Comments

@amruss
Copy link
Contributor

amruss commented Oct 27, 2020

Is your feature request related to a problem? Please describe.
When one kafka cluster is used for many changefeeds, with the same table name, the client cannot differentiate between changefeeds.

Describe the solution you'd like
Change the namespace of the changefeed for the schema and topic. Add a flag (WITH option) where we’ll use fully qualified name (topic+schema) for given table rather than just table name. Default behavior is just table name.

Describe alternatives you've considered
Should add schema prefix OR topic prefix as schema name -- fine with either
Either:

  • Prefix the schema namespace with what we use for topic namespace, or
  • User can specify prefix used in both

Additional context
Could also solve #28670

@amruss amruss added this to Triage in [DEPRECATED] CDC via automation Oct 27, 2020
@blathers-crl
Copy link

blathers-crl bot commented Oct 27, 2020

Hi @amruss, I've guessed the C-ategory of your issue and suitably labeled it. Please re-label if inaccurate.

While you're here, please consider adding an A- label to help keep our repository tidy.

🦉 Hoot! I am a Blathers, a bot for CockroachDB. My owner is otan.

@blathers-crl blathers-crl bot added the C-enhancement Solution expected to add code/behavior + preserve backward-compat (pg compat issues are exception) label Oct 27, 2020
@amruss amruss moved this from Triage to UX improvements in [DEPRECATED] CDC Oct 27, 2020
@amruss amruss added C-bug Code not up to spec/doc, specs & docs deemed correct. Solution expected to change code/behavior. A-cdc Change Data Capture and removed C-enhancement Solution expected to add code/behavior + preserve backward-compat (pg compat issues are exception) labels Oct 27, 2020
@mwang1026 mwang1026 added this to Triage in Disaster Recovery Backlog via automation Oct 27, 2020
@mwang1026 mwang1026 moved this from Triage to CDC in Disaster Recovery Backlog Nov 2, 2020
@amruss
Copy link
Contributor Author

amruss commented Nov 2, 2020

Options for command UX

Option 1
CREATE CHANGEFEED FOR INTO ‘kafka://endpoint?topic_prefix=X&schema_prefix=Y’ WITH fully_qualified_name

Schema namespace:
<schema_prefix>..<database_schema>.<table_name>

Topic namespace:
<topic_prefix>..<database_schema>.<table_name>

Option 2
CREATE CHANGEFEED FOR INTO ‘kafka://endpoint?topic_prefix=X’ WITH fully_qualified_name

Schema namespace:
<topic_prefix>..<database_schema>.<table_name>

Topic namespace:
<topic_prefix>..<database_schema>.<table_name>

Option 3
CREATE CHANGEFEED FOR INTO ‘kafka://endpoint?topic_prefix=X&schema_prefix=topic_prefix’ WITH fully_qualified_name

Schema namespace:
<schema_prefix>..<database_schema>.<table_name>

Topic namespace:
<topic_prefix>..<database_schema>.<table_name>

Option 4
CREATE CHANGEFEED FOR INTO ‘kafka://endpoint?topic_and_schema_prefix=X’ WITH fully_qualified_name

Schema namespace:
<topic_and_schema_prefix>..<database_schema>.<table_name>

Topic namespace:
<topic_and_schema_prefix>..<database_schema>.<table_name>

@mwang1026 mwang1026 changed the title cdc: topic based namespace for changefeeds cdc: separate schema and topic prefix for changefeed namespace Dec 1, 2020
@mwang1026 mwang1026 moved this from In Progress to January in Disaster Recovery Sprint Status Jan 5, 2021
@craig craig bot closed this as completed in c7d69b1 Feb 2, 2021
Disaster Recovery Sprint Status automation moved this from January to Acceptance Test Queue Feb 2, 2021
[DEPRECATED] CDC automation moved this from UX improvements to Closed Feb 2, 2021
Disaster Recovery Backlog automation moved this from CDC to Done Feb 2, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
A-cdc Change Data Capture C-bug Code not up to spec/doc, specs & docs deemed correct. Solution expected to change code/behavior. T-cdc T-disaster-recovery
Projects
[DEPRECATED] CDC
  
Closed
Disaster Recovery Sprint Status
  
Acceptance Test Queue
Development

Successfully merging a pull request may close this issue.

3 participants