Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add ability to start MoveTables in a Stopped state or to stop after the copy phase #7452

Closed
rohit-nayak-ps opened this issue Feb 5, 2021 · 0 comments · Fixed by #7449
Closed

Comments

@rohit-nayak-ps
Copy link
Contributor

Feature Description

  1. Add a flag -start_stopped to VReplication Workflows that enables the user to create a workflow in a Stopped state. Right now the workflows start running immediately after creation. So only the vreplication streams will be created but no data will be copied.

  2. There is an existing feature where a workflow can be started such that it only runs the copy phase. If stop_after_copy is set in the source column a workflow will run the copy phase and then stop. It will start the replication phase until that flag has been explicitly removed first before restarting the workflow. To enable this on the cli, a second flag -stop_after_copy has been added, which set this attribute in the workflow configuration.

Use Case(s)

A couple of use-cases for these flags are provided below.

Creating a workflow in the Stopped state
  1. As a Materialize short-cut
    For creating custom materialize workflows you can first create the _vt.vreplication rows for them using MoveTables using this flag, then update the required attributes (usually add special Filters for PII purposes or denormalizing some columns) or delete some unwanted tables, before running the workflow. This can often be simpler than generating the required parameters for the Materialize command and/or creating the entries manually using VReplicationExec.

  2. MoveTables flag changes (DDL)
    The default behavior, if DDLs are encountered during MoveTables, is to ignore them. If you want them to be applied on the target then you need to use set the on_ddl attribute (https://vitess.io/docs/reference/features/vreplication/#on_ddl)

Stop After Copy phase is completed:
  1. Snapshot for staging/dev environments
    If you want to take a "recent" copy of your current data for some reason: say performance testing, reporting etc you can set this attribute and the target keyspace will contain tables that are consistent as of the specified position reached by the stream.

Notes

  • This will be implemented in both v1 and v2 workflows
  • While there may not be a use-case at this point we have also implemented the flags for Reshard since it was easy to do
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
1 participant