Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

sharding: Use InlineVerifier by default #129

Merged
merged 3 commits into from
Sep 6, 2019

Conversation

shuhaowu
Copy link
Contributor

@shuhaowu shuhaowu commented Sep 6, 2019

Note that CompressionVerifierTest is removed from sharding/test. The CompressionVerifier is superceded by the InlineVerifier which has the features to verify compressed columns without a separate struct. The compression verifier tests rely on a hard coded IterativeVerifier on sharding.Ferry. It would require major changes to deprecated functionality if we want to keep these tests, which is deprecated itself.

Hopefully over the next while I can remove most of IterativeVerifier once we further validate the InlineVerifier.

This functionality is superceded by the InlineVerifier. The tests fail
without a IterativeVerifier hard coded into sharding.Ferry. It is not
easy to retain these tests without major changes to deprecated features.
@shuhaowu
Copy link
Contributor Author

shuhaowu commented Sep 6, 2019

Oops I ran the wrong test suites locally and CI just failed on a real bug. Looks like the InlineVerifier is not playing nicely with the delta copier. I'll fix it and send another comment once I'm done.

If all binlog events are filtered out by ApplicableEvents within the
BinlogStreamer, an empty event list will be sent to the event listeners.
This may be unexpected to the event listeners. To save the trouble, if
the BinlogStreamer detects an empty binlog event list, it'll just not
invoke the event listeners.
@shuhaowu
Copy link
Contributor Author

shuhaowu commented Sep 6, 2019

The issue was that the join table tests only inserts into joined tables, which are filtered out by the ApplicableEvents filter within the BinlogStreamer. This means the list of events the BinlogStreamer emits to the event listeners is empty. The InlineVerifier binlog event handler does not expect this and thus panics. The fix is made within the BinlogStreamer such that the event listeners are not invoked if all events are filtered.

I'm ready for another review now.

@shuhaowu shuhaowu merged commit a257570 into Shopify:master Sep 6, 2019
@shuhaowu shuhaowu deleted the sharding-inline-verifier branch September 6, 2019 17:20
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

2 participants