A better alternative to #46344

[rschu1ze]

The (experimental) inverted index writes/reads files different from the standard files written by the other skip indexes. The original problem was that with database engine "ordinary", DROP TABLE of a table with inverted index finds unknown files in persistence and complains. The same will happen with engine "atomic" but deferred. As a hotfix, the error was silenced in #46344 by explicitly adding the four files created in a specific test to the deletion code.

This PR tries a cleaner solution where all needed files are provided via the normal checksum mechanism. One drawback remains which is that the affected files were written earlier and we don't have their checksums available. Therefore, the inverted index is currently excluded from CHECK TABLE (but it was like that also before).

Minimal repro:

  SET allow_experimental_inverted_index = 1;
  DROP TABLE IF EXISTS tab;
  CREATE TABLE tab(s String, INDEX af(s) TYPE inverted(2)) ENGINE = MergeTree() ORDER BY s;
  INSERT INTO tab VALUES ('Alick a01');
  CHECK TABLE tab;
  DROP TABLE IF EXISTS tab;

Then run ./clickhouse-test with --db-engine Ordinary

Changelog category (leave one):

• Not for changelog

[UnamedRus]

Therefore, the inverted index is currently excluded from CHECK TABLE (but it was like that also before).

AFAIK, ClickHouse recalculate checksums if some files does not exist in checksum.txt.

So it's completely OK?

[rschu1ze]

AFAIK, ClickHouse recalculate checksums if some files does not exist in checksum.txt.

True.

So it's completely OK?

Well, "Storages/MergeTree/MergeTreeDataPartChecksum.cpp" was changed to suppress that. My local testing was green. The whole situation isn't great right now but better than before and hopefully only an interim solution.

[tavplubix]

It would be great to add a test with a mutation that hardlinks all files (to check that it keeps/fills the checksums)

[rschu1ze]

Can do that for sure. But I don't really understand how I could make a mutation hardlink the files. Is there a setting for that?

[tavplubix]

No, it depends on what files have to be modified by a mutation. For example, when you run ALTER ... UPDATE col1 = ..., it rewrites col1 and hardlinks other columns (and indexes that do not depend on col1). And ALTER ... DROP COLUMN col1 hardlinks everything except col1.