New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
checksum: do not check inverted index files #47607
Conversation
@@ -46,6 +46,10 @@ void MergeTreeDataPartChecksum::checkEqual(const MergeTreeDataPartChecksum & rhs | |||
|
|||
void MergeTreeDataPartChecksum::checkSize(const IDataPartStorage & storage, const String & name) const | |||
{ | |||
/// Do not check inverted index files |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Could you please move l. 50+51 down to l. 55? (the exception for inverted indexes is the least important one of the three if
's)
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Kindly add a reason to the comment in l. 49, e.g. /// Skip inverted index files, these have a default MergeTreeDataPartChecksum with file_size == 0
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
If you don't mind, I would like to understand how the database can end up with a checksum with file_size == 0 for inverted indexes (e.g. which methods are called in which order). Asking because I never encountered this error.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
What would be great if you could add a simple end-to-end stateless test (tests/queries/0_stateless/) which makes sure the situation doesn't occur again.
@rschu1ze I just built clickhouse on latest master branch(commit id: 8e0a2a3), then:
The data can not be loaded, and there are some error in logs as |
@rschu1ze I should put the
There are some errors in logs as |
Alright, that makes sense. Thanks a ton! I can confirm after local testing that the test indeed turns green with your fix. |
inverted index did introduce some special treatmemts. |
potentially another: |
MergeTreeDataPartChecksums::checkSize
will check file size while loading parts, but inverted index files only have defaultMergeTreeDataPartChecksum
(file_size = 0), these parts will fail to load.Changelog category (leave one):