VSCopy: Send COPY_COMPLETED events when the copy operation is done #11740
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…s new events showing copy completed Signed-off-by: yoheimuta <yoheimuta@gmail.com>
Signed-off-by: yoheimuta <yoheimuta@gmail.com>
Signed-off-by: yoheimuta <yoheimuta@gmail.com>
yoheimuta
requested review from
deepthi,
harshit-gangal,
systay,
shlomi-noach,
rohit-nayak-ps,
frouioui,
GuptaManan100 and
mattlord
as code owners
Review ChecklistHello reviewers! 👋 Please follow this checklist when reviewing this Pull Request. General
If a new flag is being introduced:
If a workflow is added or modified:
Bug fixes
Non-trivial changes
New/Existing features
Backward compatibility
|
yoheimuta
commented
Nov 16, 2022
| `type:COPY_COMPLETED keyspace:"ks" shard:"80-"`, | ||
| `type:COPY_COMPLETED keyspace:"ks" shard:"-80"`, | ||
| `type:COPY_COMPLETED`, | ||
| } |
There was a problem hiding this comment.
With this PR change, the above three events are added to the expected events.
yoheimuta
commented
Nov 16, 2022
go/vt/vtgate/vstream_manager.go
Outdated
| case binlogdatapb.VEventType_COPY_COMPLETED: | ||
| sendevents = append(sendevents, event) | ||
| if fullyCopied := vs.generateFullyCopyCompleted(ctx, sgtid, event); fullyCopied != nil { | ||
| sendevents = append(sendevents, fullyCopied) |
There was a problem hiding this comment.
Only this addition is different from the case of, for example, OTHER.
In the case of COPY_COMPLETED, it's possible to generate and send an extra event.
yoheimuta
commented
Nov 16, 2022
| // This new event represents the completion of all the copy operations. | ||
| func (vs *vstream) generateFullyCopyCompleted(ctx context.Context, sgtid *binlogdatapb.ShardGtid, event *binlogdatapb.VEvent) *binlogdatapb.VEvent { | ||
| vs.mu.Lock() | ||
| defer vs.mu.Unlock() |
There was a problem hiding this comment.
The function generates an additional event while holding a lock for vs.vgtid.ShardGtids.
The same lock also protects the concurrent access to vs.copyCompletedShard.
yoheimuta
commented
Nov 16, 2022
| } | ||
| } | ||
| return &binlogdatapb.VEvent{ | ||
| Type: binlogdatapb.VEventType_COPY_COMPLETED, |
There was a problem hiding this comment.
This event has neither keyspace nor shard.
It represents the completion of data copy wherever the data is.
|
@mattlord @rohit-nayak-ps I made the changes I referred to at the slack thread. |
rohit-nayak-ps
added
Type: Enhancement
Thanks, @yoheimuta ! ❤️ These failures do seem legit, so can you look into those in the meantime? I will review this ASAP but no reason to wait on that front. We can discuss on Slack too if you need any help. |
…e vtadmin web protos Signed-off-by: Rohit Nayak <rohit@planetscale.com>
rohit-nayak-ps
force-pushed
the
send-copy-done-event
branch
from
ade96a5
to
8912112
Compare
I fixed the failing We also need to run If there is not already a related issue for this, can you create one and link the PR to it (in Related Issues). Maybe you can just move the description of the problem statement that you have here into the issue ... To help discovery for open source users, we are trying to enforce that all PRs require an issue and the problem description and use case is in the issue and the resolution in the PR. Thx! |
I had a quick look at your changes and they look pretty good! |
|
Thank you for your advices and fixes!
It makes sense. |
…out out Signed-off-by: yoheimuta <yoheimuta@gmail.com>
|
I'm fixing the failure of |
… concurrently Signed-off-by: yoheimuta <yoheimuta@gmail.com>
yoheimuta
commented
Nov 17, 2022
| t.Logf("TestVStreamCopyBasic was successful") | ||
| return | ||
| } else if numExpectedEvents < len(evs) { | ||
| t.Fatalf("len(events)=%v are not expected\n", len(evs)) |
There was a problem hiding this comment.
Previously, once len(evs) gets greater than numExpectedEvents, End-to-End Test job failed due to the timeout while showing no specific errors. With this error message, it's hard to narrow down the cause.
ok vitess.io/vitess/go/vt/vtctl/grpcvtctldserver/endtoend 0.130s
Error: The action has timed out.
5825b27 makes the same test fail faster and shows more straightforward error messages.
--- FAIL: TestVStreamCopyBasic (0.02s)
vstream_test.go:244: len(events)=24 are not expected
...
FAIL vitess.io/vitess/go/vt/vtgate/endtoend 77.838s
yoheimuta
commented
Nov 17, 2022
| @@ -169,7 +169,7 @@ func TestVStreamCopyBasic(t *testing.T) { | |||
| gconn, conn, mconn, closeConnections := initialize(ctx, t) | |||
| defer closeConnections() | |||
|
|
|||
| _, err := conn.ExecuteFetch("insert into t1(id1,id2) values(1,1), (2,2), (3,3), (4,4), (5,5), (6,6), (7,7), (8,8)", 1, false) | |||
| _, err := conn.ExecuteFetch("insert into t1_copy_basic(id1,id2) values(1,1), (2,2), (3,3), (4,4), (5,5), (6,6), (7,7), (8,8)", 1, false) | |||
There was a problem hiding this comment.
To pass the End-to-End Test, I prepared the exclusive table named t1_copy_basic. 9e4b5d8
The problem sharing t1 table with the other test cases (e.g. TestConsistentLookupMultiInsertIgnore) running concurrently is not introduced by this PR. Although the number of the received events matches by chance, content of the events vary depending on the concurrent tests in the first place.
(base) ❯ git branch | grep main
* main
# The events are expected.
(base) ❯ go test -v -count=1 -run "^TestVStreamCopyBasic$" vitess.io/vitess/go/vt/vtgate/endtoend --alsologtostderr
...
===START===
Event 0; type:BEGIN keyspace:"ks" shard:"-80"
Event 1; type:FIELD field_event:{table_name:"ks.t1" fields:{name:"id1" type:INT64 table:"t1" org_table:"t1" database:"vt_ks_-80" org_name:"id1" column_length:20 charset:63 flags:53251} fields:{name:"id2" type:INT64 table:"t1" org_table:"t1" database:"vt_ks_-80" org_name:"id2" column_length:20 charset:63 flags:32768} keyspace:"ks" shard:"-80"} keyspace:"ks" shard:"-80"
Event 2; type:VGTID vgtid:{shard_gtids:{keyspace:"ks" shard:"-80" gtid:"MySQL56/82178ac0-6666-11ed-b225-e84ab9c3a7f8:1-75" table_p_ks:{table_name:"t1" lastpk:{fields:{name:"id1" type:INT32} rows:{lengths:1 values:"4"}}}} shard_gtids:{keyspace:"ks" shard:"80-" table_p_ks:{table_name:"t1" lastpk:{fields:{name:"id1" type:INT32} rows:{lengths:1 values:"4"}}}}} keyspace:"ks" shard:"-80"
Event 3; type:ROW row_event:{table_name:"ks.t1" row_changes:{after:{lengths:1 lengths:1 values:"55"}} keyspace:"ks" shard:"-80"} keyspace:"ks" shard:"-80"
Event 4; type:VGTID vgtid:{shard_gtids:{keyspace:"ks" shard:"-80" gtid:"MySQL56/82178ac0-6666-11ed-b225-e84ab9c3a7f8:1-75" table_p_ks:{table_name:"t1" lastpk:{rows:{lengths:1 values:"5"}}}} shard_gtids:{keyspace:"ks" shard:"80-" table_p_ks:{table_name:"t1" lastpk:{fields:{name:"id1" type:INT32} rows:{lengths:1 values:"4"}}}}} keyspace:"ks" shard:"-80"
Event 5; type:COMMIT keyspace:"ks" shard:"-80"
Event 6; type:BEGIN keyspace:"ks" shard:"80-"
Event 7; type:FIELD field_event:{table_name:"ks.t1" fields:{name:"id1" type:INT64 table:"t1" org_table:"t1" database:"vt_ks_80-" org_name:"id1" column_length:20 charset:63 flags:53251} fields:{name:"id2" type:INT64 table:"t1" org_table:"t1" database:"vt_ks_80-" org_name:"id2" column_length:20 charset:63 flags:32768} keyspace:"ks" shard:"80-"} keyspace:"ks" shard:"80-"
Event 8; type:VGTID vgtid:{shard_gtids:{keyspace:"ks" shard:"-80" gtid:"MySQL56/82178ac0-6666-11ed-b225-e84ab9c3a7f8:1-75" table_p_ks:{table_name:"t1" lastpk:{rows:{lengths:1 values:"5"}}}} shard_gtids:{keyspace:"ks" shard:"80-" gtid:"MySQL56/82178ac0-6666-11ed-b225-e84ab9c3a7f8:1-75" table_p_ks:{table_name:"t1" lastpk:{fields:{name:"id1" type:INT32} rows:{lengths:1 values:"4"}}}}} keyspace:"ks" shard:"80-"
Event 9; type:ROW row_event:{table_name:"ks.t1" row_changes:{after:{lengths:1 lengths:1 values:"66"}} keyspace:"ks" shard:"80-"} keyspace:"ks" shard:"80-"
Event 10; type:ROW row_event:{table_name:"ks.t1" row_changes:{after:{lengths:1 lengths:1 values:"77"}} keyspace:"ks" shard:"80-"} keyspace:"ks" shard:"80-"
Event 11; type:ROW row_event:{table_name:"ks.t1" row_changes:{after:{lengths:1 lengths:1 values:"88"}} keyspace:"ks" shard:"80-"} keyspace:"ks" shard:"80-"
Event 12; type:VGTID vgtid:{shard_gtids:{keyspace:"ks" shard:"-80" gtid:"MySQL56/82178ac0-6666-11ed-b225-e84ab9c3a7f8:1-75" table_p_ks:{table_name:"t1" lastpk:{rows:{lengths:1 values:"5"}}}} shard_gtids:{keyspace:"ks" shard:"80-" gtid:"MySQL56/82178ac0-6666-11ed-b225-e84ab9c3a7f8:1-75" table_p_ks:{table_name:"t1" lastpk:{rows:{lengths:1 values:"8"}}}}} keyspace:"ks" shard:"80-"
Event 13; type:COMMIT keyspace:"ks" shard:"80-"
Event 14; type:BEGIN keyspace:"ks" shard:"80-"
Event 15; type:VGTID vgtid:{shard_gtids:{keyspace:"ks" shard:"-80" gtid:"MySQL56/82178ac0-6666-11ed-b225-e84ab9c3a7f8:1-75" table_p_ks:{table_name:"t1" lastpk:{rows:{lengths:1 values:"5"}}}} shard_gtids:{keyspace:"ks" shard:"80-" gtid:"MySQL56/82178ac0-6666-11ed-b225-e84ab9c3a7f8:1-75"}} keyspace:"ks" shard:"80-"
Event 16; type:COMMIT keyspace:"ks" shard:"80-"
===END===
vstream_test.go:222: TestVStreamCopyBasic was successful
--- PASS: TestVStreamCopyBasic (0.01s)
PASS
# Apparently, Event 6, Event 13, and Event 14 are not expected.
## I guess that these rows are inserted at https://github.com/vitessio/vitess/blob/4c33b2b5a90dc200e6cc649681cbace2520725bd/go/vt/vtgate/endtoend/lookup_test.go#L292.
$ go test -v -count=1 vitess.io/vitess/go/vt/vtgate/endtoend --alsologtostderr
...
===START===
Event 0; type:BEGIN keyspace:"ks" shard:"80-"
Event 1; type:FIELD field_event:{table_name:"ks.t1" fields:{name:"id1" type:INT64 table:"t1" org_table:"t1" database:"vt_ks_80-" org_name:"id1" column_length:20 charset:63 flags:53251} fields:{name:"id2" type:INT64 table:"t1" org_table:"t1" database:"vt_ks_80-" org_name:"id2" column_length:20 charset:63 flags:32768} keyspace:"ks" shard:"80-"} keyspace:"ks" shard:"80-"
Event 2; type:VGTID vgtid:{shard_gtids:{keyspace:"ks" shard:"-80" table_p_ks:{table_name:"t1" lastpk:{fields:{name:"id1" type:INT32} rows:{lengths:1 values:"4"}}}} shard_gtids:{keyspace:"ks" shard:"80-" gtid:"MySQL56/2ac38034-6667-11ed-a179-dd231a1931a8:1-148" table_p_ks:{table_name:"t1" lastpk:{fields:{name:"id1" type:INT32} rows:{lengths:1 values:"4"}}}}} keyspace:"ks" shard:"80-"
Event 3; type:ROW row_event:{table_name:"ks.t1" row_changes:{after:{lengths:1 lengths:1 values:"66"}} keyspace:"ks" shard:"80-"} keyspace:"ks" shard:"80-"
Event 4; type:ROW row_event:{table_name:"ks.t1" row_changes:{after:{lengths:1 lengths:1 values:"77"}} keyspace:"ks" shard:"80-"} keyspace:"ks" shard:"80-"
Event 5; type:ROW row_event:{table_name:"ks.t1" row_changes:{after:{lengths:1 lengths:1 values:"88"}} keyspace:"ks" shard:"80-"} keyspace:"ks" shard:"80-"
Event 6; type:ROW row_event:{table_name:"ks.t1" row_changes:{after:{lengths:2 lengths:2 values:"5060"}} keyspace:"ks" shard:"80-"} keyspace:"ks" shard:"80-"
Event 7; type:VGTID vgtid:{shard_gtids:{keyspace:"ks" shard:"-80" table_p_ks:{table_name:"t1" lastpk:{fields:{name:"id1" type:INT32} rows:{lengths:1 values:"4"}}}} shard_gtids:{keyspace:"ks" shard:"80-" gtid:"MySQL56/2ac38034-6667-11ed-a179-dd231a1931a8:1-148" table_p_ks:{table_name:"t1" lastpk:{rows:{lengths:2 values:"50"}}}}} keyspace:"ks" shard:"80-"
Event 8; type:COMMIT keyspace:"ks" shard:"80-"
Event 9; type:BEGIN keyspace:"ks" shard:"-80"
Event 10; type:FIELD field_event:{table_name:"ks.t1" fields:{name:"id1" type:INT64 table:"t1" org_table:"t1" database:"vt_ks_-80" org_name:"id1" column_length:20 charset:63 flags:53251} fields:{name:"id2" type:INT64 table:"t1" org_table:"t1" database:"vt_ks_-80" org_name:"id2" column_length:20 charset:63 flags:32768} keyspace:"ks" shard:"-80"} keyspace:"ks" shard:"-80"
Event 11; type:VGTID vgtid:{shard_gtids:{keyspace:"ks" shard:"-80" gtid:"MySQL56/2ac38034-6667-11ed-a179-dd231a1931a8:1-148" table_p_ks:{table_name:"t1" lastpk:{fields:{name:"id1" type:INT32} rows:{lengths:1 values:"4"}}}} shard_gtids:{keyspace:"ks" shard:"80-" gtid:"MySQL56/2ac38034-6667-11ed-a179-dd231a1931a8:1-148" table_p_ks:{table_name:"t1" lastpk:{rows:{lengths:2 values:"50"}}}}} keyspace:"ks" shard:"-80"
Event 12; type:ROW row_event:{table_name:"ks.t1" row_changes:{after:{lengths:1 lengths:1 values:"55"}} keyspace:"ks" shard:"-80"} keyspace:"ks" shard:"-80"
Event 13; type:ROW row_event:{table_name:"ks.t1" row_changes:{after:{lengths:2 lengths:2 values:"1020"}} keyspace:"ks" shard:"-80"} keyspace:"ks" shard:"-80"
Event 14; type:ROW row_event:{table_name:"ks.t1" row_changes:{after:{lengths:2 lengths:2 values:"3040"}} keyspace:"ks" shard:"-80"} keyspace:"ks" shard:"-80"
Event 15; type:VGTID vgtid:{shard_gtids:{keyspace:"ks" shard:"-80" gtid:"MySQL56/2ac38034-6667-11ed-a179-dd231a1931a8:1-148" table_p_ks:{table_name:"t1" lastpk:{rows:{lengths:2 values:"30"}}}} shard_gtids:{keyspace:"ks" shard:"80-" gtid:"MySQL56/2ac38034-6667-11ed-a179-dd231a1931a8:1-148" table_p_ks:{table_name:"t1" lastpk:{rows:{lengths:2 values:"50"}}}}} keyspace:"ks" shard:"-80"
Event 16; type:COMMIT keyspace:"ks" shard:"-80"
===END===
vstream_test.go:222: TestVStreamCopyBasic was successful
--- PASS: TestVStreamCopyBasic (0.01s)
Since this discrepancy caused the test failure in my PR, I fixed it.
|
All checks have passed. Thank you! |
rohit-nayak-ps
approved these changes
Nov 29, 2022
mattlord
reviewed
Nov 29, 2022
There was a problem hiding this comment.
Nice work on this! I had some minor questions/comments/suggestions. I'll let you make the final call on the subjective preferences noted. We can quickly address these and then get this merged! Feel free to respond here or reach out on slack, whichever you prefer. ❤️
|
A note about docs... I'm on the fence regarding whether or not this should be in the release notes. @rohit-nayak-ps, what do you think? I also think that we should probably add info here about these new events and what they signify: vitessio/website#1216 I'll link the two PRs. |
Signed-off-by: yoheimuta <yoheimuta@gmail.com>
Signed-off-by: yoheimuta <yoheimuta@gmail.com>
Signed-off-by: yoheimuta <yoheimuta@gmail.com>
|
@mattlord I applied all your suggestions and answered your question. I think it looks possible to pass the failed test by rerunning https://github.com/vitessio/vitess/actions/runs/3580407547/jobs/6022506484. |
Yes, it succeeded on a retry! |
mattlord
approved these changes
Nov 30, 2022
There was a problem hiding this comment.
This is really good. Thanks again for fixing that long flaky test too! 🙂
Regarding the docs, I think we're good as:
- We have the existing release notes section which covers the new feature in general and links to the in progress new docs page: https://github.com/vitessio/vitess/blob/main/doc/releasenotes/16_0_0_summary.md#vstream-copy-resume
- We have that same open docs PR that we can add to for these new events: Document VStream website#1216
frouioui
added a commit
to planetscale/vitess
that referenced
this pull request
Nov 21, 2023
…io#1625) * Trigger OnlineDDL workflow when test data changes (#11827) * trigger OnlineDDL workflow when test data changes Signed-off-by: Florent Poinsard <florent.poinsard@outlook.fr> * trigger schemadiff workflows too Signed-off-by: Florent Poinsard <florent.poinsard@outlook.fr> Signed-off-by: Florent Poinsard <florent.poinsard@outlook.fr> * BugFix: Escaping Percentage and Underscore require special handling (#11823) * test: add failing end to end test for escaped % and _ Signed-off-by: Manan Gupta <manan@planetscale.com> * feat: preserve escaping of \ and % in the parser Signed-off-by: Manan Gupta <manan@planetscale.com> * feat: add end to end test and planner test to verify that % and _ handling is correct in Vitess on vtgate level Signed-off-by: Manan Gupta <manan@planetscale.com> * feat: add test to evaluation engine to verify that backslash and underscore and handled correctly even when escaped Signed-off-by: Manan Gupta <manan@planetscale.com> * feat: remove unrequired changes Signed-off-by: Manan Gupta <manan@planetscale.com> Signed-off-by: Manan Gupta <manan@planetscale.com> * Incremental logical backup and point in time recovery (#11097) * Unexplode Backup() function, pass BackupRequest as argument Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com> * use tabletmanagerdata.BackupRequest Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com> * make proto Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com> * removed duplicate tabletmanagerdata imports Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com> * tabletmanagerdatapb Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com> * vschemapb Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com> * require.NoError Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com> * proto: BackupRequest.incremental_from_pos Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com> * pass IncrementalFromPos Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com> * populate incremental_from_pos Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com> * Storing ServerUUID and TabletAlias as part of backup MANIFESTO Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com> * use IncrementalFromPos in BackupParams, populate Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com> * executeIncrementalBackup Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com> * add unit tests for GTID 'Contains' Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com> * Add binlog related functions in MySQLDaemon Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com> * More functionality in incremental backup Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com> * add backupBinlogDir as a valid backup directory Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com> * include FromPosition in the backup manifest. Find binlog files to backup Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com> * complete incremental backup Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com> * clarify the difference between user's requested position, and the FromPosition in the manifest. Add 'Incremental' (bool) to the manifest Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com> * make vtadmin_web_proto_types Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com> * Add Keyspace, Shard to backup manifest Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com> * for good order, keyspace comes first Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com> * take into account purged GTIDs. Fix value of 'incrementalBackupToGTID' Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com> * endtoend tests for incremental backup. No restore validation as yet. Tests do not have a GitHub workflow yet. Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com> * Adding CI shard: 'backup_pitr' Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com> * cleanup Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com> * backup_pitr tested via mysql80 Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com> * insert data with hint Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com> * refactor Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com> * FindPITRPath: find a shortest path to recover a GTID position, base on one full backup and zero or more inremental backups Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com> * more validation Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com> * more test cases Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com> * RestoreFromBackupRequest: RestoreToPos Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com> * vtctl Restore: '--restore_to_pos' Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com> * make vtadmin_web_proto_types Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com> * RestoreFromBackupRequest: RestoreToPos Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com> * Unexplode: RestoreFromBackup() receives req *tabletmanagerdatapb.RestoreFromBackupRequest Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com> * Unexplode: RestoreFromBackup() receives req *tabletmanagerdatapb.RestoreFromBackupRequest Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com> * make vtadmin_web_proto_types Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com> * populate restoreParams.RestoreToPos Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com> * simplifying the logic of finding relevant backup Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com> * fix switch/break logic Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com> * towards a restore path Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com> * golang version Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com> * fix workflows ubuntu version Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com> * skip nil manifests Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com> * FindBackupToRestore() returns a RestorePath, which is an ordered sequence of backup manifests/handles Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com> * linter suggestion Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com> * fix backup time comparison logic Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com> * vtctl Restore supports --dry_run flag Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com> * flag --incremental-from-pos accepts the value 'auto', which takes the next incremental backup from last good backup Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com> * make vtadmin_web_proto_types Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com> * endtoend: validate --incremental_from_pos=auto Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com> * towards applying binary logs: extracting onto temporary directory Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com> * apply binary log file Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com> * do not restore replication at end of PITR Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com> * take dryrun into consideration Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com> * testing restore to pos Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com> * testing restore to pos: wait for replication, avoid bogus writes Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com> * validating PITR path when binary logs are missing history Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com> * full backup manifest now includes 'PurgedPosition', which is necessary to build a restore path. Now evaluated in IsValidIncrementalBakcup Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com> * more recovery paths tests Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com> * restrucutre tests Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com> * log restore path Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com> * generate CI workflows Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com> * code comments Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com> * code comments Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com> * code comments Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com> * CI 57 and 80 Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com> * flags test Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com> * copyright year Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com> * go version Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com> * removed legacy mysql80 test Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com> * PITR: stop search for possible resotre path with the first valid path, even if it's not the optimal Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com> * support incrementally union-izing of previous-GTIDs when iterating binary logs Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com> * removed local metadata info Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com> * merged main, regenerated workflows Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com> * go mod tidy Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com> * rename conflicting variable Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com> * refactor: const value Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com> * dry run restore now returns with 0 exit code, no error Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com> * release notes Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com> Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com> * Migrate to GitHub OIDC based auth for Launchable (#11808) This change updates the CI workflows to use the new GitHub OpenID connect based authentication flow. GitHub started to provide a public-key signed token that contain pull-request data. This is commonly used as a short-lived token in the authentication flow (Open ID Connect). Launchable recently started supporting this. Migrate to this new method. The permission clause added to the jobs is more restrictive than the default access except for the "id-token: write" permission (https://docs.github.com/en/actions/security-guides/automatic-token-authentication#permissions-for-the-github_token). This should give the e2e tests enough permissions to run. The id-token permission is necessary to get the OIDC ID tokens as instructed by the GitHub article https://docs.github.com/en/actions/deployment/security-hardening-your-deployments/about-security-hardening-with-openid-connect#adding-permissions-settings. See https://docs.launchableinc.com/sending-data-to-launchable/migration-to-github-oidc-auth for the overview and the process. Signed-off-by: Masaya Suzuki <draftcode@gmail.com> Signed-off-by: Masaya Suzuki <draftcode@gmail.com> * feat: add instructions on how to fix a self-hosted runner running out of disk space (#11839) Signed-off-by: Manan Gupta <manan@planetscale.com> Signed-off-by: Manan Gupta <manan@planetscale.com> * Fix closing the body for HTTP requests (#11842) One of the easy things to miss that can cause resource leaks is not closing a response body for an HTTP request. There's a linter available to make it easier to catch this, so this enables that linter and fixes all the cases reported. This is almost all test cases, except for one production code path case in the throttler. Signed-off-by: Dirkjan Bussink <d.bussink@gmail.com> Signed-off-by: Dirkjan Bussink <d.bussink@gmail.com> * Remove vtctld2 UI and vtctld server components that serve the app UI Signed-off-by: notfelineit <notfelineit@gmail.com> * Add back http port Signed-off-by: notfelineit <notfelineit@gmail.com> * Release notes for 15.0.1 (#11850) Signed-off-by: Rameez Sajwani <rameezwazirali@hotmail.com> Signed-off-by: Rameez Sajwani <rameezwazirali@hotmail.com> * Release notes summary of `14.0.4` (#11849) * Release notes for 14.0.4 Signed-off-by: Florent Poinsard <florent.poinsard@outlook.fr> * release summary for v14.0.4 Signed-off-by: Florent Poinsard <florent.poinsard@outlook.fr> Signed-off-by: Florent Poinsard <florent.poinsard@outlook.fr> * Fix deprecated usage of set-output (#11844) The `set-output` command is deprecated and will break in the future. This moves to the new style for all our GitHub Actions. See also https://github.blog/changelog/2022-10-11-github-actions-deprecating-save-state-and-set-output-commands/ Signed-off-by: Dirkjan Bussink <d.bussink@gmail.com> Signed-off-by: Dirkjan Bussink <d.bussink@gmail.com> * VReplication: Remove Deprecated V1 Client Commands (#11705) * Remove SwitchReads/SwitchWrites client commands Signed-off-by: Matt Lord <mattalord@gmail.com> * Remove Reshard/MoveTables v1 client commands Signed-off-by: Matt Lord <mattalord@gmail.com> * Misc text clarifications Signed-off-by: Matt Lord <mattalord@gmail.com> * Fix TestVreplicationCopyThrottling Signed-off-by: Matt Lord <mattalord@gmail.com> * Remove DropSources Signed-off-by: Matt Lord <mattalord@gmail.com> * Add removal info to release notes Signed-off-by: Matt Lord <mattalord@gmail.com> * Reduce flakiness of copy_state optimization check Signed-off-by: Matt Lord <mattalord@gmail.com> * Minor tweaks after self review Signed-off-by: Matt Lord <mattalord@gmail.com> * De-flake sharded pitr test Signed-off-by: Matt Lord <mattalord@gmail.com> Signed-off-by: Matt Lord <mattalord@gmail.com> * updating summary and release notes for v15.0.1 (#11852) * updating summary and release notes for v15.0.1 Signed-off-by: Rameez Sajwani <rameezwazirali@hotmail.com> * fixing typos Signed-off-by: Rameez Sajwani <rameezwazirali@hotmail.com> Signed-off-by: Rameez Sajwani <rameezwazirali@hotmail.com> * Simplify recursive data structure in CFC Vindex (#11843) * fix: remove recursize data dag in CFC Vindex Signed-off-by: Harshit Gangal <harshit@planetscale.com> * test: update test expectation Signed-off-by: Harshit Gangal <harshit@planetscale.com> * addressed review comments Signed-off-by: Harshit Gangal <harshit@planetscale.com> Signed-off-by: Harshit Gangal <harshit@planetscale.com> * Update throttler-topo workflow file (#11784) Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com> Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com> * [cleanup] Explicitly include DDLStrategySetting in the sizegen target (#11857) Closes #11856. Signed-off-by: Andrew Mason <andrew@planetscale.com> Signed-off-by: Andrew Mason <andrew@planetscale.com> * VSCopy: Send COPY_COMPLETED events when the copy operation is done (#11740) * VSCopy: Demonstrate to fail a test case on which the vstream API sends new events showing copy completed Signed-off-by: yoheimuta <yoheimuta@gmail.com> * VSCopy: Send new events when the copy operation is done Signed-off-by: yoheimuta <yoheimuta@gmail.com> * VSCopy: Fix typo Signed-off-by: yoheimuta <yoheimuta@gmail.com> * Initialize new map for the 'vstream * from' vtgate sql interface. Make vtadmin web protos Signed-off-by: Rohit Nayak <rohit@planetscale.com> * VSCopy: Make TestVStreamCopyBasic fail fast to avoid the end2end timeout out Signed-off-by: yoheimuta <yoheimuta@gmail.com> * VSCopy: stop sharing the 't1' table among multiple test cases running concurrently Signed-off-by: yoheimuta <yoheimuta@gmail.com> * VSCopy: refactor the function signature to be clearer Signed-off-by: yoheimuta <yoheimuta@gmail.com> * VSCopy: refactor the VEvents sorter to be simpler Signed-off-by: yoheimuta <yoheimuta@gmail.com> * VSCopy: refactor to stop the sorter from including a fully copied event Signed-off-by: yoheimuta <yoheimuta@gmail.com> Signed-off-by: yoheimuta <yoheimuta@gmail.com> Signed-off-by: Rohit Nayak <rohit@planetscale.com> Co-authored-by: Rohit Nayak <rohit@planetscale.com> * add frances to maintainers (#11865) Signed-off-by: deepthi <deepthi@planetscale.com> Signed-off-by: deepthi <deepthi@planetscale.com> * VReplication: Prevent Orphaned VDiff2 Jobs (#11768) * Prevent orphaned VDiffs in two ways... 1. When opening the engine, restart any vdiffs that are in the started state as this indicates it did not complete and was unable to save the final state and must be restarted. 2. When a vdiff run fails, retry saving the error state with an exponential backoff until the engine shuts down. This way the normal retry mechanism will kick in OR #1 will kick in when the engine is next opened on the primary tablet. Signed-off-by: Matt Lord <mattalord@gmail.com> * Handle failures before vdiff_table records are created Signed-off-by: Matt Lord <mattalord@gmail.com> * Add more ephemeral client errors Signed-off-by: Matt Lord <mattalord@gmail.com> * Show vdiff state of error even if no vdiff_table records Signed-off-by: Matt Lord <mattalord@gmail.com> * Minor cleanup Signed-off-by: Matt Lord <mattalord@gmail.com> * Add vdiff2 unit tests Signed-off-by: Matt Lord <mattalord@gmail.com> * Add unit test for retry Signed-off-by: Matt Lord <mattalord@gmail.com> * Small cleanup Signed-off-by: Matt Lord <mattalord@gmail.com> * Addressing review comments and other improvements Signed-off-by: Matt Lord <mattalord@gmail.com> * Use warning log for ... warnings :-) Signed-off-by: Matt Lord <mattalord@gmail.com> * Minor touch ups Signed-off-by: Matt Lord <mattalord@gmail.com> Signed-off-by: Matt Lord <mattalord@gmail.com> * update golangci-lint to 1.50.1 (#11873) Signed-off-by: Andres Taylor <andres@planetscale.com> Signed-off-by: Andres Taylor <andres@planetscale.com> * [planner] Better AST equality (#11867) * make it possible to do deep comparisons with custom logic Signed-off-by: Andres Taylor <andres@planetscale.com> * use the new EqualS function to do deep semantic equality Signed-off-by: Andres Taylor <andres@planetscale.com> * re gen the asthelpergen integration test data Signed-off-by: Florent Poinsard <florent.poinsard@outlook.fr> * update equal gen comment Signed-off-by: Florent Poinsard <florent.poinsard@outlook.fr> * test: add unit tests showing the new EqualsExpr functionality Signed-off-by: Andres Taylor <andres@planetscale.com> * [planner] use the semantic equality in more places Signed-off-by: Andres Taylor <andres@planetscale.com> * comments - update to reflect new code Signed-off-by: Andres Taylor <andres@planetscale.com> * small fixes around tests Signed-off-by: Andres Taylor <andres@planetscale.com> * update integration test build Signed-off-by: Andres Taylor <andres@planetscale.com> * use semantic equality for Anding together expressions Signed-off-by: Andres Taylor <andres@planetscale.com> * test: simplify test Signed-off-by: Andres Taylor <andres@planetscale.com> Signed-off-by: Andres Taylor <andres@planetscale.com> Signed-off-by: Florent Poinsard <florent.poinsard@outlook.fr> Co-authored-by: Florent Poinsard <florent.poinsard@outlook.fr> * Parse binlog variable length encoded columns (#11871) * Test for large table map binlog events Signed-off-by: Dirkjan Bussink <d.bussink@gmail.com> * Use variable length encoded integers for tablemap The table map binlog event is defined as using variable integer lengths for the number of columns and the metadata length. This was hardcoded to using a single byte, so any lengths longer than 250 would be interpreted wrong. This was already annotated as a FIXME. Signed-off-by: Dirkjan Bussink <d.bussink@gmail.com> * Add test for large row event Signed-off-by: Dirkjan Bussink <d.bussink@gmail.com> * Use variable length encoding for binlog row events Signed-off-by: Dirkjan Bussink <d.bussink@gmail.com> * Adding endtoend vreplicatoin test that validates many columns in a table Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com> * test that adds a column on top of many columns Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com> Signed-off-by: Dirkjan Bussink <d.bussink@gmail.com> Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com> Co-authored-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com> * Online DDL: fix 'vtctlclient OnlineDDL' template queries (#11889) Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com> Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com> * Fix MySQL56 GTID parsing when SID/UUID repeats (#11888) * Fix MySQL56 GTID parsing when SID/UUID repeats Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com> * beautify tests Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com> Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com> * feat: fix checkMySQL and add tests and documentation (#11895) Signed-off-by: Manan Gupta <manan@planetscale.com> Signed-off-by: Manan Gupta <manan@planetscale.com> * Add back tablet proxy for api, remove workflows, modify vtctld scripts to remove workflow flags Signed-off-by: notfelineit <notfelineit@gmail.com> * Add vtctld removal to release notes for 16 Signed-off-by: notfelineit <notfelineit@gmail.com> * Remove traces of deleted flags Signed-off-by: notfelineit <notfelineit@gmail.com> * Make vtctld_show_topology_crud flag deprecated instead of removed Signed-off-by: notfelineit <notfelineit@gmail.com> * more unit tests for QueryMatchesTemplates() (#11894) Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com> Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com> * sqlparser: new Equality API (#11906) * sqlparser: use a new Equals API Signed-off-by: Vicent Marti <vmg@strn.cat> * tools: refactor how asthelpergen is invoked Signed-off-by: Vicent Marti <vmg@strn.cat> * goimports, flags and linter Signed-off-by: Andres Taylor <andres@planetscale.com> Signed-off-by: Vicent Marti <vmg@strn.cat> Signed-off-by: Andres Taylor <andres@planetscale.com> Co-authored-by: Andres Taylor <andres@planetscale.com> * bugfix: allow predicates without dependencies with derived tables to be handled correctly (#11911) * bugfix: allow predicates on top of derived tables to be handled without dependencies Signed-off-by: Andres Taylor <andres@planetscale.com> * stop normalizer from changing literals into bindvars Signed-off-by: Andres Taylor <andres@planetscale.com> Signed-off-by: Andres Taylor <andres@planetscale.com> * Simplify `getPlan` and `gen4CompareV3` (#11903) * simplify getPlan and gen4CompareV3 Signed-off-by: Florent Poinsard <florent.poinsard@outlook.fr> * fix error return in getPlan Signed-off-by: Florent Poinsard <florent.poinsard@outlook.fr> Signed-off-by: Florent Poinsard <florent.poinsard@outlook.fr> * optimize joins, redirect dml for reference tables (#11875) * go/vt/vtgate, proto: add "source" field to vschema * Add "source" field to vschema reference tables. * Model "source" field in go/vt/vtgate/vindexes/vschema.go. * Treat multiple tables in different keyspaces as unambiguous when all of those tables are references to or by each other. Signed-off-by: Max Englander <max@planetscale.com> * go/vt/vtgate/planbuilder: plan reference joins Given: * Two unsharded keyspace `k1` and sharded keyspace `k2`. * Source table `k1.r` and reference table `k2.r`. * Sharded table `k2.x`. The previous commit adds support to globally route `r` to `k1.r`. This means that `SELECT x JOIN r` will join `k1.r` to `x`, but what we want is to join `k2.r`. This commit propagates `vindexes.Table.Source` info from the `vindexes.VSchema` into `operators.Route`, and uses that information when planning joins. Signed-off-by: Max Englander <max@planetscale.com> * go/vt/vtgate/vindexes: test reference source constraints Signed-off-by: Max Englander <max@planetscale.com> * go/vt/vtgate/vindexes: FindTable unit tests for reference sources Signed-off-by: Max Englander <max@planetscale.com> * go/vt/vtgate/vindexes: build keyspace allow unresolved references Signed-off-by: Max Englander <max@planetscale.com> * go/vt/vtgate: executor unit test reference tables Signed-off-by: Max Englander <max@planetscale.com> * go/vt/vtgate: executor unit test reference dml Signed-off-by: Max Englander <max@planetscale.com> * go/vt/vtgate/vindexes: avoid bi-directional references (causes infinite loop) Signed-off-by: Max Englander <max@planetscale.com> * go/test/endtoend/vtgate: test reference tables Signed-off-by: Max Englander <max@planetscale.com> * tidy up Signed-off-by: Max Englander <max@planetscale.com> * address pr comments round 1 Signed-off-by: Max Englander <max@planetscale.com> * address pr feedback round 2: dont use replicas in tests when not needed Signed-off-by: Max Englander <max@planetscale.com> * address pr feedback round 2: use vterrors in vschema.go Signed-off-by: Max Englander <max@planetscale.com> * fix tests Signed-off-by: Max Englander <max@planetscale.com> * address pr feedback round 3: fix e2e test Signed-off-by: Max Englander <max@planetscale.com> * address pr feedback round 3: optimally route dml Signed-off-by: Max Englander <max@planetscale.com> * address pr feedback round 3: fix e2e & plan tests (primitive.GetTablesUsed) Signed-off-by: Max Englander <max@planetscale.com> * forgot to commit these :facepalm: Signed-off-by: Max Englander <max@planetscale.com> * move TablesUsed from engine => planbuilder Signed-off-by: Max Englander <max@planetscale.com> * rm engine.(any).GetTablesUsed Signed-off-by: Max Englander <max@planetscale.com> * rm engine.(any).MergedWith; revert engine.(*Route).TableNames => engine.(*Route).TableName Signed-off-by: Max Englander <max@planetscale.com> * tidy up Signed-off-by: Max Englander <max@planetscale.com> * address pr feedback round 3: redir gen4 insert to source Signed-off-by: Max Englander <max@planetscale.com> * address pr feedback round 4: remove unused (vindexFunc).TableName Signed-off-by: Max Englander <max@planetscale.com> Signed-off-by: Max Englander <max@planetscale.com> * Upgrade to `go1.19.4` (#11905) * upgrade to go1.19.4 Signed-off-by: Florent Poinsard <florent.poinsard@outlook.fr> * use bootstrap version 14 Signed-off-by: Florent Poinsard <florent.poinsard@outlook.fr> Signed-off-by: Florent Poinsard <florent.poinsard@outlook.fr> * Add support for views in vtgate (#11195) * feat: add view support to vtgate Signed-off-by: Andres Taylor <andres@planetscale.com> * feat: added views to the vschema Signed-off-by: Andres Taylor <andres@planetscale.com> * refactor: change the VSchemaViews to return SelectStatement Signed-off-by: Andres Taylor <andres@planetscale.com> * feat: hook up the new views in the vschema with the ast rewriter Signed-off-by: Andres Taylor <andres@planetscale.com> * feat: make it possible to find uniquely named views Signed-off-by: Andres Taylor <andres@planetscale.com> * fix: make sure to create new ColNames of the view AST Signed-off-by: Andres Taylor <andres@planetscale.com> * test: added views unit test Signed-off-by: Harshit Gangal <harshit@planetscale.com> * test: added e2e tesT Signed-off-by: Harshit Gangal <harshit@planetscale.com> * feat: added create view vschema ddl Signed-off-by: Harshit Gangal <harshit@planetscale.com> * feat: implement create view vschema ddl action update Signed-off-by: Harshit Gangal <harshit@planetscale.com> * sizegen rerun Signed-off-by: Harshit Gangal <harshit@planetscale.com> * test: added unit test for create view vschema ddl Signed-off-by: Harshit Gangal <harshit@planetscale.com> * feat: added replace view support in vschema ddl Signed-off-by: Harshit Gangal <harshit@planetscale.com> * doc: view support summary notes Signed-off-by: Harshit Gangal <harshit@planetscale.com> * feat: vschema drop view ddl support Signed-off-by: Harshit Gangal <harshit@planetscale.com> * Revert "feat: vschema drop view ddl support" This reverts commit 744f2e9e763d49654355340548dc8746f6c80d6a. Signed-off-by: Harshit Gangal <harshit@planetscale.com> * Revert "feat: added replace view support in vschema ddl" This reverts commit 5394e9b8d64f28e708591bfbb9137ddb961497e2. Signed-off-by: Harshit Gangal <harshit@planetscale.com> * Revert "test: added unit test for create view vschema ddl" This reverts commit 54ffe11ba96d127fe043eb9a7d379085caecce59. Signed-off-by: Harshit Gangal <harshit@planetscale.com> * Revert "sizegen rerun" This reverts commit 9f7ff6b31dfc32ea483f66d93a7f7d7f0239a3d6. Signed-off-by: Harshit Gangal <harshit@planetscale.com> * Revert "feat: implement create view vschema ddl action update" This reverts commit c0123447bf2c8b6533bc7629bc7acd176ae8c764. Signed-off-by: Harshit Gangal <harshit@planetscale.com> * Revert "feat: added create view vschema ddl" This reverts commit be623f935689082c3a20e8507efc3d1c5c1a245b. Signed-off-by: Harshit Gangal <harshit@planetscale.com> * fix: e2e test Signed-off-by: Harshit Gangal <harshit@planetscale.com> * removed views from vschema proto Signed-off-by: Harshit Gangal <harshit@planetscale.com> * go back to using Table and not *string for uniqueTables Signed-off-by: Andres Taylor <andres@planetscale.com> * salvage bad merge Signed-off-by: Andres Taylor <andres@planetscale.com> Signed-off-by: Andres Taylor <andres@planetscale.com> Signed-off-by: Harshit Gangal <harshit@planetscale.com> Co-authored-by: Harshit Gangal <harshit@planetscale.com> * don't run test on older versions, since they fail (#11923) Signed-off-by: Andres Taylor <andres@planetscale.com> Signed-off-by: Andres Taylor <andres@planetscale.com> * Better clone of the VCursor (#11926) * Better clone of the VCursor Signed-off-by: Florent Poinsard <florent.poinsard@outlook.fr> * Move new implementations to the proper files Signed-off-by: Florent Poinsard <florent.poinsard@outlook.fr> * apply review suggestions Signed-off-by: Florent Poinsard <florent.poinsard@outlook.fr> Signed-off-by: Florent Poinsard <florent.poinsard@outlook.fr> * feat: add test and fix the error of not sending a ServerLost error on server side error (#11920) Signed-off-by: Manan Gupta <manan@planetscale.com> Signed-off-by: Manan Gupta <manan@planetscale.com> * Revert "Better clone of the VCursor (#11926)" (#11931) This reverts commit 035d6b7124e4ff4a99aa87aa549cc82acc05044a. Signed-off-by: Florent Poinsard <florent.poinsard@outlook.fr> Signed-off-by: Florent Poinsard <florent.poinsard@outlook.fr> * Correct minor inaccuracies in governing docs (#11933) Signed-off-by: deepthi <deepthi@planetscale.com> Signed-off-by: deepthi <deepthi@planetscale.com> * log: also log error in DiscoverInstance when force discovery is specified (#11936) Signed-off-by: Manan Gupta <manan@planetscale.com> Signed-off-by: Manan Gupta <manan@planetscale.com> * Onlineddl: formalize "immediate operations", respect `--postpone-completion` strategy flag (#11910) * Added endtoend test to validate postponed REVERT of an ALTER VIEW statement: test indicates a bug Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com> * ALTER VIEW now respects --postpone-completion flag Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com> * Online DDL: formalize 'immediate operations', respect --postpone-completion Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com> * typo Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com> * only test instant ddl in capable versions (ie 8.0) Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com> * more specific instant ddl capability Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com> * Update go/vt/vttablet/onlineddl/executor.go Co-authored-by: Deepthi Sigireddi <deepthi.sigireddi@gmail.com> Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com> * Update go/vt/vttablet/onlineddl/executor.go Co-authored-by: Deepthi Sigireddi <deepthi.sigireddi@gmail.com> Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com> * Update go/vt/vttablet/onlineddl/executor.go Co-authored-by: Deepthi Sigireddi <deepthi.sigireddi@gmail.com> Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com> Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com> Co-authored-by: Deepthi Sigireddi <deepthi.sigireddi@gmail.com> * add Arthur to the maintainers file (#11937) * add Arthur to the mainternes file Signed-off-by: Andres Taylor <andres@planetscale.com> * add Arthur as code owner for the vtgate planbuilder Signed-off-by: Andres Taylor <andres@planetscale.com> * Update MAINTAINERS.md Co-authored-by: Arthur Schreiber <arthurschreiber@github.com> Signed-off-by: Andres Taylor <andres@taylor.se> Signed-off-by: Andres Taylor <andres@planetscale.com> Signed-off-by: Andres Taylor <andres@taylor.se> Co-authored-by: Arthur Schreiber <arthurschreiber@github.com> * Online DDL: ensure message is valid `utf8` in `updateMigrationMessage()` (#11914) * OnlineDDL: handle binary/invalid data in updateMigrationMessage() Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com> * Ensure error message is valid utf8 Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com> * cap length; special replacement character Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com> * swap order, so that utf8 compliance of string is ensured Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com> Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com> * [planner] Schema information on the information_schema views (#11941) * add info_schema information Signed-off-by: Andres Taylor <andres@planetscale.com> * add SchemaInformation handling for info_schema tables Signed-off-by: Andres Taylor <andres@planetscale.com> * fix bad test query Signed-off-by: Andres Taylor <andres@planetscale.com> * add support for information_schema on mysql 5.7 Signed-off-by: Andres Taylor <andres@planetscale.com> * columns sorted just like mysql, and tests for 5.7 Signed-off-by: Andres Taylor <andres@planetscale.com> * test: skip test that should not run Signed-off-by: Andres Taylor <andres@planetscale.com> Signed-off-by: Andres Taylor <andres@planetscale.com> * CODEOWNERS: Add vrepl team members for vtgate vstream and tablet picker (#11950) * Add vrepl team members for vtgate vstream and tablet picker Both vtgate components/services are related to VReplication. Signed-off-by: Matt Lord <mattalord@gmail.com> * Add vstream endtoend tests too Signed-off-by: Matt Lord <mattalord@gmail.com> Signed-off-by: Matt Lord <mattalord@gmail.com> * Release notes for 15.0.2 (#11963) Signed-off-by: Florent Poinsard <florent.poinsard@outlook.fr> Signed-off-by: Florent Poinsard <florent.poinsard@outlook.fr> * update the release 15.0.2 summary doc (#11956) Signed-off-by: Florent Poinsard <florent.poinsard@outlook.fr> Signed-off-by: Florent Poinsard <florent.poinsard@outlook.fr> Co-authored-by: Florent Poinsard <florent.poinsard@outlook.fr> * moved missed flags to pflags in vtgate (#11966) * used pflags over flags on pending usages Signed-off-by: Harshit Gangal <harshit@planetscale.com> * removed flags from go/vt/vtgate/endtoend package Signed-off-by: Harshit Gangal <harshit@planetscale.com> * test: update test helper output Signed-off-by: Harshit Gangal <harshit@planetscale.com> * addressed review comments Signed-off-by: Harshit Gangal <harshit@planetscale.com> Signed-off-by: Harshit Gangal <harshit@planetscale.com> * Document error code in `vtgate/planbuilder` (#10738) * Addition of fundation for new error codes Signed-off-by: Florent Poinsard <florent.poinsard@outlook.fr> * Generate README on the stdout for error code Signed-off-by: Florent Poinsard <florent.poinsard@outlook.fr> * Added support for more error code Signed-off-by: Florent Poinsard <florent.poinsard@outlook.fr> * test new error code Signed-off-by: Florent Poinsard <florent.poinsard@outlook.fr> * test remove error code Signed-off-by: Florent Poinsard <florent.poinsard@outlook.fr> * test new error code Signed-off-by: Florent Poinsard <florent.poinsard@outlook.fr> * addition of the error code for the planbuilder package Signed-off-by: Florent Poinsard <florent.poinsard@outlook.fr> * Handle more errors of the vtgate package in the centralized error code Signed-off-by: Florent Poinsard <florent.poinsard@outlook.fr> * Use error code in more places after merge Signed-off-by: Florent Poinsard <florent.poinsard@outlook.fr> * Update code long usage Signed-off-by: Florent Poinsard <florent.poinsard@outlook.fr> * Add cause to OurError Signed-off-by: Florent Poinsard <florent.poinsard@outlook.fr> * update the plantest output Signed-off-by: Florent Poinsard <florent.poinsard@outlook.fr> * Update test expectation and run make codegen Signed-off-by: Florent Poinsard <florent.poinsard@outlook.fr> * Update test with VT05003 error Signed-off-by: Florent Poinsard <florent.poinsard@outlook.fr> * Fix vtgate tests Signed-off-by: Florent Poinsard <florent.poinsard@outlook.fr> * Add missing error code to doc list Signed-off-by: Florent Poinsard <florent.poinsard@outlook.fr> * Fix test TestExecutorAddDropVindexDDL Signed-off-by: Florent Poinsard <florent.poinsard@outlook.fr> * Fix english typo in VT03023 Signed-off-by: Florent Poinsard <florent.poinsard@outlook.fr> * Fix plan tests Signed-off-by: Florent Poinsard <florent.poinsard@outlook.fr> * revert unwanted change Signed-off-by: Florent Poinsard <florent.poinsard@outlook.fr> * make proto Signed-off-by: Andres Taylor <andres@planetscale.com> * Addition of comment to the error doc generator Signed-off-by: Florent Poinsard <florent.poinsard@outlook.fr> * Rename OurError to VitessError Signed-off-by: Florent Poinsard <florent.poinsard@outlook.fr> * consistent lower-case short error description Signed-off-by: Florent Poinsard <florent.poinsard@outlook.fr> * fix VT03001 long description Signed-off-by: Florent Poinsard <florent.poinsard@outlook.fr> * fix VT03005 short and long descriptions Signed-off-by: Florent Poinsard <florent.poinsard@outlook.fr> * fix VT03007, VT03012 and VT03021 long descriptions Signed-off-by: Florent Poinsard <florent.poinsard@outlook.fr> * fix VT05***, VT06001 and VT09*** descriptions Signed-off-by: Florent Poinsard <florent.poinsard@outlook.fr> * simplify VT09008 Signed-off-by: Florent Poinsard <florent.poinsard@outlook.fr> * fix VT12*** and VT13*** long descriptions Signed-off-by: Florent Poinsard <florent.poinsard@outlook.fr> * rename vterrors generation package to vterrorsgen instead of main Signed-off-by: Florent Poinsard <florent.poinsard@outlook.fr> * fix error syntax for distinct checkCol bug Signed-off-by: Florent Poinsard <florent.poinsard@outlook.fr> * fix syntax in error call Signed-off-by: Florent Poinsard <florent.poinsard@outlook.fr> * fix syntax in error call Signed-off-by: Florent Poinsard <florent.poinsard@outlook.fr> * fix syntax issues in error messages Signed-off-by: Florent Poinsard <florent.poinsard@outlook.fr> * fix syntax issues in error messages Signed-off-by: Florent Poinsard <florent.poinsard@outlook.fr> * fix syntax issues in error messages Signed-off-by: Florent Poinsard <florent.poinsard@outlook.fr> * fix syntax based on review suggestions Signed-off-by: Florent Poinsard <florent.poinsard@outlook.fr> * [gen4 tests] update test expectations Signed-off-by: Andres Taylor <andres@planetscale.com> * tests: updated more tests to use new error strings Signed-off-by: Andres Taylor <andres@planetscale.com> * added snippet to release notes Signed-off-by: Andres Taylor <andres@planetscale.com> * prepare errors so we can have more implementations Signed-off-by: Andres Taylor <andres@planetscale.com> * start cleaning up semantic errors Signed-off-by: Andres Taylor <andres@planetscale.com> * prefix unsupported errors Signed-off-by: Andres Taylor <andres@planetscale.com> * move more errors to structured errors Signed-off-by: Andres Taylor <andres@planetscale.com> * commit Signed-off-by: Andres Taylor <andres@planetscale.com> * moved binder to user structured errors Signed-off-by: Andres Taylor <andres@planetscale.com> * combine template must and parse Signed-off-by: Florent Poinsard <florent.poinsard@outlook.fr> * humanize error in hasCodeForRow for probeTable Signed-off-by: Florent Poinsard <florent.poinsard@outlook.fr> * humanize engine primitive errors Signed-off-by: Florent Poinsard <florent.poinsard@outlook.fr> * go through all the unsupported errors once Signed-off-by: Florent Poinsard <florent.poinsard@outlook.fr> * apply review suggestions Signed-off-by: Florent Poinsard <florent.poinsard@outlook.fr> * go through error call Signed-off-by: Florent Poinsard <florent.poinsard@outlook.fr> * add missing error in planbuilder Signed-off-by: Florent Poinsard <florent.poinsard@outlook.fr> * fix plan tests Signed-off-by: Florent Poinsard <florent.poinsard@outlook.fr> * fix insert E2E tests Signed-off-by: Florent Poinsard <florent.poinsard@outlook.fr> * clean release summary file Signed-off-by: Florent Poinsard <florent.poinsard@outlook.fr> * clean up long description for VT09006 and VT09007 Signed-off-by: Florent Poinsard <florent.poinsard@outlook.fr> * self-review PR once Signed-off-by: Florent Poinsard <florent.poinsard@outlook.fr> * update plan test output Signed-off-by: Florent Poinsard <florent.poinsard@outlook.fr> * update sementics package test output Signed-off-by: Florent Poinsard <florent.poinsard@outlook.fr> * database -> keyspace in error strings Signed-off-by: Florent Poinsard <florent.poinsard@outlook.fr> * fix plan_test output Signed-off-by: Florent Poinsard <florent.poinsard@outlook.fr> * use same long description as the one previously written in the docs Signed-off-by: Florent Poinsard <florent.poinsard@outlook.fr> * re generate the plan tests output Signed-off-by: Florent Poinsard <florent.poinsard@outlook.fr> * reformat error VT09003 Signed-off-by: Florent Poinsard <florent.poinsard@outlook.fr> * remove heading line from the error code generation template Signed-off-by: Florent Poinsard <florent.poinsard@outlook.fr> * update test expectation Signed-off-by: Andres Taylor <andres@planetscale.com> Signed-off-by: Florent Poinsard <florent.poinsard@outlook.fr> Signed-off-by: Andres Taylor <andres@planetscale.com> Co-authored-by: Andres Taylor <andres@planetscale.com> * Migrate missed vtctld flags to pflag and immediately deprecate them (#11974) Signed-off-by: Andrew Mason <andrew@planetscale.com> Signed-off-by: Andrew Mason <andrew@planetscale.com> * Timeout Fixes and VTOrc Improvement (#11881) * refactor: move tests out of newfeaturestest so that they run on upgrade-downgrade tests too Signed-off-by: Manan Gupta <manan@planetscale.com> * feat: add failing ers test for handling multiple vttablet failures with default values of flags Signed-off-by: Manan Gupta <manan@planetscale.com> * feat: add a new lock-timeout flag and use that instead of remote-operation-timeout Signed-off-by: Manan Gupta <manan@planetscale.com> * feat: augment DownPrimary test to reproduce the issue of VTOrc not handling multiple failures Signed-off-by: Manan Gupta <manan@planetscale.com> * feat: remove LockShardTimeout configuration from VTOrc and add parallelism to refresh of tablets Signed-off-by: Manan Gupta <manan@planetscale.com> * log: add more logging lines around ers in vtorc Signed-off-by: Manan Gupta <manan@planetscale.com> * test: get the test to work Signed-off-by: Manan Gupta <manan@planetscale.com> * feat: fix usage of wait for replicas timeout Signed-off-by: Manan Gupta <manan@planetscale.com> * test: fix flags expected output Signed-off-by: Manan Gupta <manan@planetscale.com> * test: fix race in test now that the function is called in parallel multiple times Signed-off-by: Manan Gupta <manan@planetscale.com> * feat: fix default of onCloseTimeout to 1 second Signed-off-by: Manan Gupta <manan@planetscale.com> * test: add failing unit test to refreshTabletsInKeyspaceShard Signed-off-by: Manan Gupta <manan@planetscale.com> * feat: fix vtorc to not forget a tablet which has been deleted Signed-off-by: Manan Gupta <manan@planetscale.com> * feat: fix backward compatibility, add tests and release notes docs Signed-off-by: Manan Gupta <manan@planetscale.com> * test: fix flags output Signed-off-by: Manan Gupta <manan@planetscale.com> * test: use disable-replication-manager instead of disable-active-reparents to allow vttablets to setup replication when restarted Signed-off-by: Manan Gupta <manan@planetscale.com> * test: fix flaky test by not checking for an error Signed-off-by: Manan Gupta <manan@planetscale.com> * feat: handle the case of empty hostname in tablet initialization Signed-off-by: Manan Gupta <manan@planetscale.com> * feat: update onclose timeout to 10 seconds Signed-off-by: Manan Gupta <manan@planetscale.com> * test: fix unit test Signed-off-by: Manan Gupta <manan@planetscale.com> * feat: address review comments Signed-off-by: Manan Gupta <manan@planetscale.com> * docs: add comments explaining the test functions Signed-off-by: Manan Gupta <manan@planetscale.com> * feat: add summary docs for 'lock-shard-timeout' deprecation Signed-off-by: Manan Gupta <manan@planetscale.com> Signed-off-by: Manan Gupta <manan@planetscale.com> * Add manan and florent to Docker files CODEOWNERS (#11981) Signed-off-by: Florent Poinsard <florent.poinsard@outlook.fr> Signed-off-by: Florent Poinsard <florent.poinsard@outlook.fr> * vttestserver: make tablet_refresh_interval configurable and reduce default value (#11918) * setting up refresh_interval and check_leader_interval Signed-off-by: Rameez Sajwani <rameezwazirali@hotmail.com> * setting default value, if not provided Signed-off-by: Rameez Sajwani <rameezwazirali@hotmail.com> * fixing vttablet flags Signed-off-by: Rameez Sajwani <rameezwazirali@hotmail.com> * fixing vtttestserver flags Signed-off-by: Rameez Sajwani <rameezwazirali@hotmail.com> * adding comments and removing unwanted script Signed-off-by: Rameez Sajwani <rameezwazirali@hotmail.com> * code review Signed-off-by: Rameez Sajwani <rameezwazirali@hotmail.com> * removing leader Check interval Signed-off-by: Rameez Sajwani <rameezwazirali@hotmail.com> * change to flags out put file Signed-off-by: Rameez Sajwani <rameezwazirali@hotmail.com> * adding summary Signed-off-by: Rameez Sajwani <rameezwazirali@hotmail.com> * code review feedback Signed-off-by: Rameez Sajwani <rameezwazirali@hotmail.com> Signed-off-by: Rameez Sajwani <rameezwazirali@hotmail.com> * Add support for Views DDL (#11896) * create views table and start supporting it in vtgate Signed-off-by: Florent Poinsard <florent.poinsard@outlook.fr> * start supporting views in vttablet Signed-off-by: Florent Poinsard <florent.poinsard@outlook.fr> * feat: add create view ddl support by inserting into _vt.views table Signed-off-by: Harshit Gangal <harshit@planetscale.com> * feat: add support for create or replace view Signed-off-by: Harshit Gangal <harshit@planetscale.com> * feat: on view insert failure, return view already exists error Signed-off-by: Harshit Gangal <harshit@planetscale.com> * feat: add support for alter view ddl in vttablet Signed-off-by: Harshit Gangal <harshit@planetscale.com> * feat: add support for drop view ddl in vttablet Signed-off-by: Harshit Gangal <harshit@planetscale.com> * feat: add support for view ddl in vtgate Signed-off-by: Harshit Gangal <harshit@planetscale.com> * changed error to use new vitess error code Signed-off-by: Harshit Gangal <harshit@planetscale.com> * fix test expectation Signed-off-by: Harshit Gangal <harshit@planetscale.com> * change flag name to queryserver-enable-views to not conflict with vtgate for vtcombo binary Signed-off-by: Harshit Gangal <harshit@planetscale.com> * test: updated test help output Signed-off-by: Harshit Gangal <harshit@planetscale.com> * test: add create view ddl to fakedb query expectation Signed-off-by: Harshit Gangal <harshit@planetscale.com> * addressed review comments Signed-off-by: Harshit Gangal <harshit@planetscale.com> Signed-off-by: Florent Poinsard <florent.poinsard@outlook.fr> Signed-off-by: Harshit Gangal <harshit@planetscale.com> Co-authored-by: Harshit Gangal <harshit@planetscale.com> * Add `VStreamerCount` stat to `vttablet` (#11978) * Add `VStreamersActive` stat to `vttablet` Signed-off-by: Tim Vaillancourt <tim@timvaillancourt.com> * Improve desc Signed-off-by: Tim Vaillancourt <tim@timvaillancourt.com> * Add PR suggestions Signed-off-by: Tim Vaillancourt <tim@timvaillancourt.com> * Move to *stats.Gauge Signed-off-by: Tim Vaillancourt <tim@timvaillancourt.com> * Single defer Signed-off-by: Tim Vaillancourt <tim@timvaillancourt.com> Signed-off-by: Tim Vaillancourt <tim@timvaillancourt.com> * VExplain statement (#11892) * feat: ast changes for vtexplain and adding the primitive to the interfaces Signed-off-by: Manan Gupta <manan@planetscale.com> * feat: add planning for json output from vtexplain Signed-off-by: Manan Gupta <manan@planetscale.com> * test: add tests for vtexplain statements supported Signed-off-by: Manan Gupta <manan@planetscale.com> * feat: handle todo for converting some explain statements to vtexplain Signed-off-by: Manan Gupta <manan@planetscale.com> * feat: update parser after comments on rfc Signed-off-by: Manan Gupta <manan@planetscale.com> * feat: fix planner to align with the new input format Signed-off-by: Manan Gupta <manan@planetscale.com> * feat: add basic implementation for vexplain all Signed-off-by: Manan Gupta <manan@planetscale.com> * merge mysql and vitess primitive JSON and add E2E tests Signed-off-by: Florent Poinsard <florent.poinsard@outlook.fr> * remove old unit test and generate new plan tests Signed-off-by: Florent Poinsard <florent.poinsard@outlook.fr> * feat: fix todos and augment tests Signed-off-by: Manan Gupta <manan@planetscale.com> * refactor: reintroduce removed comments Signed-off-by: Manan Gupta <manan@planetscale.com> * refactor: reintroduce more removed comments Signed-off-by: Manan Gupta <manan@planetscale.com> * feat: add summary docs Signed-off-by: Manan Gupta <manan@planetscale.com> * feat: remvoe exhaustive keyword which isn't required anymore Signed-off-by: Manan Gupta <manan@planetscale.com> * feat: fix config.json file to point to the right folder Signed-off-by: Manan Gupta <manan@planetscale.com> * test: fix unit tests Signed-off-by: Manan Gupta <manan@planetscale.com> Signed-off-by: Manan Gupta <manan@planetscale.com> Signed-off-by: Florent Poinsard <florent.poinsard@outlook.fr> Co-authored-by: Florent Poinsard <florent.poinsard@outlook.fr> * only expand when we have full information (#11998) Signed-off-by: Andres Taylor <andres@planetscale.com> Signed-off-by: Andres Taylor <andres@planetscale.com> * remove e2e test from partial_keyspace config (#12005) Signed-off-by: Andres Taylor <andres@planetscale.com> Signed-off-by: Andres Taylor <andres@planetscale.com> * remove log line (#12000) Signed-off-by: Andres Taylor <andres@planetscale.com> Signed-off-by: Andres Taylor <andres@planetscale.com> * Allow override of build git env in docker/base builds (#11968) * Pass BUILD_GIT_BRANCH and BUILD_GIT_REV env into docker build Signed-off-by: Tim Vaillancourt <tim@timvaillancourt.com> * Fix bootstrap version Signed-off-by: Tim Vaillancourt <tim@timvaillancourt.com> * Fix bootstrap version, again Signed-off-by: Tim Vaillancourt <tim@timvaillancourt.com> * Fix ws Signed-off-by: Tim Vaillancourt <tim@timvaillancourt.com> Signed-off-by: Tim Vaillancourt <tim@timvaillancourt.com> Signed-off-by: Florent Poinsard <florent.poinsard@outlook.fr> Co-authored-by: Florent Poinsard <florent.poinsard@outlook.fr> * feat: fix vttestserver run script defaults (#12004) Signed-off-by: Manan Gupta <manan@planetscale.com> Signed-off-by: Manan Gupta <manan@planetscale.com> * Fix parsing and normalization of floating point types (#12009) * Fix parsing of float type with single length attribute MySQL can also parse this so ensure that we're compatible here. This is deprecated but it can still be used to create tables but in MySQL 8 this length is ignored on the final table. Signed-off-by: Dirkjan Bussink <d.bussink@gmail.com> * Add float4 and float8 aliases for float and double Signed-off-by: Dirkjan Bussink <d.bussink@gmail.com> * Normalize floating point values according to MySQL rules Signed-off-by: Dirkjan Bussink <d.bussink@gmail.com> * Simplify parser with reuse Signed-off-by: Dirkjan Bussink <d.bussink@gmail.com> Signed-off-by: Dirkjan Bussink <d.bussink@gmail.com> * feat: add vtorc port to vitess local docker run (#12001) Signed-off-by: Manan Gupta <manan@planetscale.com> Signed-off-by: Manan Gupta <manan@planetscale.com> * VDiff2: Migrate VDiff1 Unit Tests (#11916) * Initial workflow differ tests Signed-off-by: Matt Lord <mattalord@gmail.com> * More tests Signed-off-by: Matt Lord <mattalord@gmail.com> * Playing with vdiffEnv Signed-off-by: Matt Lord <mattalord@gmail.com> * more vdiffEnv work Signed-off-by: Matt Lord <mattalord@gmail.com> * Use a shard testenv as it creates a vttestserver and mysqld instanc3 Signed-off-by: Matt Lord <mattalord@gmail.com> * Add vtctl client results/report tests Signed-off-by: Matt Lord <mattalord@gmail.com> * Add TestVDiff to test full set of queries Signed-off-by: Matt Lord <mattalord@gmail.com> * Finish up vtctl report building unit tests Signed-off-by: Matt Lord <mattalord@gmail.com> * Deflake the unit tests in the CI Signed-off-by: Matt Lord <mattalord@gmail.com> * Add TestVDiff2Sharded Signed-off-by: Matt Lord <mattalord@gmail.com> * Minor changes after self review Signed-off-by: Matt Lord <mattalord@gmail.com> * Reduce resource usage for race condition unit tests We were creating a somewhat crazy number of resources. Signed-off-by: Matt Lord <mattalord@gmail.com> * Fix data race in TestVDiff Signed-off-by: Matt Lord <mattalord@gmail.com> * Correct Wrangler semaphore usage Signed-off-by: Matt Lord <mattalord@gmail.com> * Minor change after another self review Signed-off-by: Matt Lord <mattalord@gmail.com> Signed-off-by: Matt Lord <mattalord@gmail.com> * mysqlctl is a command-line client so remove server flags (#12022) * mysqlctl is a client command-line tool So remove the [gRPC] server related flags. [mysqlctld] is the gRPC server that can be used instead of mysqlctl. Signed-off-by: Matt Lord <mattalord@gmail.com> * Add release notes coverage Signed-off-by: Matt Lord <mattalord@gmail.com> * Add section header for release notes item Signed-off-by: Matt Lord <mattalord@gmail.com> * Remove errant flag type in list Signed-off-by: Matt Lord <mattalord@gmail.com> Signed-off-by: Matt Lord <mattalord@gmail.com> * OnlineDDL: scheduleNextMigration() to only read reviewed migrations (#12014) * OnlineDDL: scheduleNextMigration() to only read reviewed migrations Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com> * reviewImmediateOperations(): do not attempt to classify a REVERT of an ALTER Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com> Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com> * sqlparser: QueryMatchesTemplates uses canonical string (#11990) Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com> Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com> * OnlineDDL: support --unsafe-allow-foreign-keys strategy flag (#11976) * OnlineDDL: support --unsafe-allow-foreign-keys strategy flag Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com> * double dashes Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com> Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com> * BugFix: Errant GTID detection for a single replica (#12024) * test: add failing tests Signed-off-by: Manan Gupta <manan@planetscale.com> * feat: fix errant gtid detection Signed-off-by: Manan Gupta <manan@planetscale.com> * comment: add few comments to better explain the test and the function Signed-off-by: Manan Gupta <manan@planetscale.com> * feat: fix ers to not fail when there is only one replica which fails to set replication source Signed-off-by: Manan Gupta <manan@planetscale.com> * feat: address review comments Signed-off-by: Manan Gupta <manan@planetscale.com> Signed-off-by: Manan Gupta <manan@planetscale.com> * Fix rbac config in the vtop example (#12034) * feat: fix rbac config in the operator example Signed-off-by: Manan Gupta <manan@planetscale.com> * feat: fix clusters value too Signed-off-by: Manan Gupta <manan@planetscale.com> Signed-off-by: Manan Gupta <manan@planetscale.com> * VReplication: Test Migrations From MariaDB to MySQL (#12036) * Add MariaDB Migration e2e Test Signed-off-by: Matt Lord <mattalord@gmail.com> * Consolidate MariaDB Version Configs Signed-off-by: Matt Lord <mattalord@gmail.com> * Minor changes after self review Signed-off-by: Matt Lord <mattalord@gmail.com> * Additional MariaDB cleanup from unit test generation Signed-off-by: Dirkjan Bussink <d.bussink@gmail.com> * Small additional cleanup Signed-off-by: Matt Lord <mattalord@gmail.com> * Use latest MariaDB LTS release as default guess from ENV https://mariadb.com/kb/en/changes-improvements-in-mariadb-106/ Signed-off-by: Matt Lord <mattalord@gmail.com> Signed-off-by: Matt Lord <mattalord@gmail.com> Signed-off-by: Dirkjan Bussink <d.bussink@gmail.com> Co-authored-by: Dirkjan Bussink <d.bussink@gmail.com> * [vtctldclient|docs] apply doc feedback based on website PR feedback (#12030) * apply doc feedback based on website PR feedback Signed-off-by: Andrew Mason <andrew@planetscale.com> * always link to latest Signed-off-by: Andrew Mason <andrew@planetscale.com> * update flags test data Signed-off-by: Andrew Mason <andrew@planetscale.com> Signed-off-by: Andrew Mason <andrew@planetscale.com> * VTOrc Code Cleanup - generate_base, replace cluster_name with keyspace and shard. (#12012) * feat: refactor generate commands of VTOrc to be in a single file Signed-off-by: Manan Gupta <manan@planetscale.com> * refactor: cleanup create table formatting Signed-off-by: Manan Gupta <manan@planetscale.com> * feat: cleanup the usage of IsSQLite and IsMySQL Signed-off-by: Manan Gupta <manan@planetscale.com> * feat: remove unused minimal instance Signed-off-by: Manan Gupta <manan@planetscale.com> * feat: remove unused table cluster_domain_name Signed-off-by: Manan Gupta <manan@planetscale.com> * feat: fix vtorc database to store keyspace and shard instead of cluster Signed-off-by: Manan Gupta <manan@planetscale.com> * feat: remove unused attributes Signed-off-by: Manan Gupta <manan@planetscale.com> * feat: remove unused cluster domain Signed-off-by: Manan Gupta <manan@planetscale.com> * feat: change GetClusterName to GetKeyspaceAndShardName Signed-off-by: Manan Gupta <manan@planetscale.com> * feat: fix insertion into database_instance Signed-off-by: Manan Gupta <manan@planetscale.com> * feat: fix SnapshotTopologies Signed-off-by: Manan Gupta <manan@planetscale.com> * feat: remove inject unseen primary and inject seed Signed-off-by: Manan Gupta <manan@planetscale.com> * feat: remove ClusterName from Instance Signed-off-by: Manan Gupta <manan@planetscale.com> * feat: fix Audit operations Signed-off-by: Manan Gupta <manan@planetscale.com> * feat: add Keyspace and Shard to cluster information to replace ClusterName Signed-off-by: Manan Gupta <manan@planetscale.com> * feat: fix attempt failure detection registeration Signed-off-by: Manan Gupta <manan@planetscale.com> * feat: fix blocked topology recoveries Signed-off-by: Manan Gupta <manan@planetscale.com> * feat: fix topology recovery Signed-off-by: Manan Gupta <manan@planetscale.com> * feat: reading recovery instances Signed-off-by: Manan Gupta <manan@planetscale.com> * feat: fix get replication and analysis Signed-off-by: Manan Gupta <manan@planetscale.com> * feat: fix bug in query Signed-off-by: Manan Gupta <manan@planetscale.com> * test: add tests to check that filtering by keyspace works for APIs Signed-off-by: Manan Gupta <manan@planetscale.com> * feat: remove remaining usages of ClusterName Signed-off-by: Manan Gupta <manan@planetscale.com> * refactor: fix comment explaining sleep in the test Signed-off-by: Manan Gupta <manan@planetscale.com> * feat: add code to prevent filtering just by shard and add tests for it Signed-off-by: Manan Gupta <manan@planetscale.com> Signed-off-by: Manan Gupta <manan@planetscale.com> * Add missing backslash to run.sh script (#12033) * Add missing backslash to run.sh script Backslash was missing, which was causing, that --schema_dir was not applied and therefore tables in DB were not created. This will fix it Signed-off-by: kbslvsk <76602353+kbslvsk@users.noreply.github.com> * Add missing backslash to run.sh script Backslash was missing, which was causing, that --schema_dir was not applied and therefore tables in DB were not created. This will fix it Signed-off-by: kbslvsk <76602353+kbslvsk@users.noreply.github.com> Signed-off-by: kbslvsk <76602353+kbslvsk@users.noreply.github.com> * VDiff2: Properly App…
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description
The
COPY_COMPLETEDevent is sent every time either condition is met:In most cases (e.g., my use case), only the event without values is enough for error handling.
In some cases, the event corresponding to a single shard is helpful to handle it in a customized way.
Related Issue(s)
#11764
Checklist
Deployment Notes