slack-15.0: VStreams fix backports/patches, pt. 1
#373
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Signed-off-by: Tim Vaillancourt <tim@timvaillancourt.com>
…vitessio#11103) * VSCopy: Demonstrate to fail a test case on which the vstream API is supposed to resume the copy phase consistently Signed-off-by: yoheimuta <yoheimuta@gmail.com> * VSCopy: Resume the copy phase consistently from given GTID and lastpk Signed-off-by: yoheimuta <yoheimuta@gmail.com> * Build out the unit test some more Signed-off-by: Matt Lord <mattalord@gmail.com> * Update tests for new behavior Signed-off-by: Matt Lord <mattalord@gmail.com> * Improve comments Signed-off-by: Matt Lord <mattalord@gmail.com> * Limit uvstreamer changes and update test Signed-off-by: Matt Lord <mattalord@gmail.com> * Revert uvstreamer test changes Signed-off-by: Matt Lord <mattalord@gmail.com> * Revert all uvstream changes Signed-off-by: Matt Lord <mattalord@gmail.com> * VCopy: Revert the last three commits Signed-off-by: yoheimuta <yoheimuta@gmail.com> * VCopy: Add a new vstream type that allows picking up where we left off Signed-off-by: yoheimuta <yoheimuta@gmail.com> * VCopy: Revert the unit test change Signed-off-by: yoheimuta <yoheimuta@gmail.com> * VCopy: Fix the end-to-end CI test Signed-off-by: yoheimuta <yoheimuta@gmail.com> * Update logic for setting up uvstreamer based on input vgtid/tablepks. Add more catchup events to test Signed-off-by: Rohit Nayak <rohit@planetscale.com> * Refactor logic to decide if event is to be sent. Enhance unit and e2e tests. Signed-off-by: Rohit Nayak <rohit@planetscale.com> * Don't send events for tables which we can identify as ones we haven't started copy for Signed-off-by: Rohit Nayak <rohit@planetscale.com> * Minor changes after self-review Signed-off-by: Rohit Nayak <rohit@planetscale.com> * Add vstream copy resume to release notes Signed-off-by: Matt Lord <mattalord@gmail.com> * Address review comments Signed-off-by: Matt Lord <mattalord@gmail.com> Signed-off-by: yoheimuta <yoheimuta@gmail.com> Signed-off-by: Matt Lord <mattalord@gmail.com> Signed-off-by: Rohit Nayak <rohit@planetscale.com> Co-authored-by: Matt Lord <mattalord@gmail.com> Co-authored-by: Rohit Nayak <rohit@planetscale.com>
|
Thanks for the contribution! Before we can merge this, we need @yoheimuta to sign the Salesforce Inc. Contributor License Agreement. |
…itessio#11740) * VSCopy: Demonstrate to fail a test case on which the vstream API sends new events showing copy completed Signed-off-by: yoheimuta <yoheimuta@gmail.com> * VSCopy: Send new events when the copy operation is done Signed-off-by: yoheimuta <yoheimuta@gmail.com> * VSCopy: Fix typo Signed-off-by: yoheimuta <yoheimuta@gmail.com> * Initialize new map for the 'vstream * from' vtgate sql interface. Make vtadmin web protos Signed-off-by: Rohit Nayak <rohit@planetscale.com> * VSCopy: Make TestVStreamCopyBasic fail fast to avoid the end2end timeout out Signed-off-by: yoheimuta <yoheimuta@gmail.com> * VSCopy: stop sharing the 't1' table among multiple test cases running concurrently Signed-off-by: yoheimuta <yoheimuta@gmail.com> * VSCopy: refactor the function signature to be clearer Signed-off-by: yoheimuta <yoheimuta@gmail.com> * VSCopy: refactor the VEvents sorter to be simpler Signed-off-by: yoheimuta <yoheimuta@gmail.com> * VSCopy: refactor to stop the sorter from including a fully copied event Signed-off-by: yoheimuta <yoheimuta@gmail.com> Signed-off-by: yoheimuta <yoheimuta@gmail.com> Signed-off-by: Rohit Nayak <rohit@planetscale.com> Co-authored-by: Rohit Nayak <rohit@planetscale.com>
… or all keyspaces (vitessio#11909) * VSCopy: Demonstrate to fail a test case on which the vstream API request doesn't include keyspace and shard Signed-off-by: yoheimuta <yoheimuta@gmail.com> * VSCopy: Copy from all shards in all keyspaces by specifying only an empty gtid Signed-off-by: yoheimuta <yoheimuta@gmail.com> * tests: Make TestRowCount stable regardless of the number of keyspaces Signed-off-by: yoheimuta <yoheimuta@gmail.com> * tests: Cleanup TestCreateAndDropDatabase correctly to stop TestVStreamCopyWithoutKeyspaceShard from failing when running tests together Signed-off-by: yoheimuta <yoheimuta@gmail.com> * tests: Tweak to fix a comment Signed-off-by: yoheimuta <yoheimuta@gmail.com> * VSCopy: fix the unit tests when the input vgtid with an empty gtid lacks either keyspace or shard Signed-off-by: yoheimuta <yoheimuta@gmail.com> * VSCopy: Keyspace wildcard selection lines up with the table wildcard selection Signed-off-by: yoheimuta <yoheimuta@gmail.com> * VSCopy: Tests the VCopy with multiple keyspaces and resharding Signed-off-by: yoheimuta <yoheimuta@gmail.com> * VSCopy: Make TestVStreamCopyMultiKeyspaceReshard clearer to check if the streaming two keyspaces works even after reshard Signed-off-by: yoheimuta <yoheimuta@gmail.com> * VSCopy: Return an invalid argument error if shards are unspecified and gtid is neither 'current' nor empty Signed-off-by: yoheimuta <yoheimuta@gmail.com> * VSCopy: Add a test description about its purpose and target Signed-off-by: yoheimuta <yoheimuta@gmail.com> * VSCopy: Remove duplicate literals in the test Signed-off-by: yoheimuta <yoheimuta@gmail.com> * VSCopy: Retain defaultReplicas variable in the test Signed-off-by: yoheimuta <yoheimuta@gmail.com> * VSCopy: Explain why we are setting Match to 'customer.*' in the test Signed-off-by: yoheimuta <yoheimuta@gmail.com> * VSCopy: Remove an unused VStreamFlag for the test Signed-off-by: yoheimuta <yoheimuta@gmail.com> * VSCopy: Use sentence capitalization in the test Signed-off-by: yoheimuta <yoheimuta@gmail.com> * VSCopy: Verify that we didn't lose any events or get duplicates of the keyspace being reshareded in the test Signed-off-by: yoheimuta <yoheimuta@gmail.com> * VSCopy: Return a value instead of a pointer because there is no need to modify the value Signed-off-by: yoheimuta <yoheimuta@gmail.com> * VSCopy: Add a comment describing what TestVStreamCopyFromAllKeyspacesAndAllShards is doing and why Signed-off-by: yoheimuta <yoheimuta@gmail.com> * VSCopy: Add a comment describing why we expect these specific numbers of events from VStream API Signed-off-by: yoheimuta <yoheimuta@gmail.com> * VSCopy: Tweak the test case name Signed-off-by: yoheimuta <yoheimuta@gmail.com> * VSCopy: Make a utility function to sort COPY_COMPLETED events in the test Signed-off-by: yoheimuta <yoheimuta@gmail.com> * VSCopy: Replace the matcher with a simpler one in the test Signed-off-by: yoheimuta <yoheimuta@gmail.com> * VSCopy: Move the print debug call to the FailNow section in the test Signed-off-by: yoheimuta <yoheimuta@gmail.com> * VSCopy: Use require.NoError in new tests Signed-off-by: yoheimuta <yoheimuta@gmail.com> * VSCopy: Use require instead of t.Fatalf in the test Signed-off-by: yoheimuta <yoheimuta@gmail.com> * VSCopy: Apply the reviewer's suggestion to make the error message easier to read Signed-off-by: yoheimuta <yoheimuta@gmail.com> * VSCopy: Add a comment noting what we're actually testing Signed-off-by: yoheimuta <yoheimuta@gmail.com> * VSCopy: Correct the test comment and elaborate the special-case Signed-off-by: yoheimuta <yoheimuta@gmail.com> * VSCopy: Tweak an error message and a comment Signed-off-by: yoheimuta <yoheimuta@gmail.com> * VSCopy: Adjust to a change in the signature of a test function that was introduced in the main repository Signed-off-by: yoheimuta <yoheimuta@gmail.com> --------- Signed-off-by: yoheimuta <yoheimuta@gmail.com>
timvaillancourt
changed the title
slack-15.0: VStream fix backports/patches, pt. 1slack-15.0: VStreams fix backports/patches, pt. 1
Signed-off-by: Tim Vaillancourt <tim@timvaillancourt.com>
roderickyao
previously approved these changes
May 28, 2024
Signed-off-by: Tim Vaillancourt <tim@timvaillancourt.com>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description
Slack patches:
Upstream backports:
Related Issue(s)
Checklist
Deployment Notes