MergeTree mutations reuse source part index granularity

[kitaisreal]

Changelog category (leave one):

• Bug Fix (user-visible misbehavior in an official stable release)

Changelog entry (a user-readable short description of the changes that goes to CHANGELOG.md):

Fix possible broken skipping indexes after materialization in MergeTree compact parts.

[robot-ch-test-poll4]

This is an automated comment for commit 6578fe5 with description of existing statuses. It's updated for the latest CI running

❌ Click here to open a full report in a separate page

Successful checks

Check name	Description	Status
AST fuzzer	Runs randomly generated queries to catch program errors. The build type is optionally given in parenthesis. If it fails, ask a maintainer for help	✅ success
CI running	A meta-check that indicates the running CI. Normally, it's in success or pending state. The failed status indicates some problems with the PR	✅ success
ClickHouse build check	Builds ClickHouse in various configurations for use in further steps. You have to fix the builds that fail. Build logs often has enough information to fix the error, but you might have to reproduce the failure locally. The cmake options can be found in the build log, grepping for cmake. Use these options and follow the general build process	✅ success
Compatibility check	Checks that clickhouse binary runs on distributions with old libc versions. If it fails, ask a maintainer for help	✅ success
Docker image for servers	The check to build and optionally push the mentioned image to docker hub	✅ success
Fast test	Normally this is the first check that is ran for a PR. It builds ClickHouse and runs most of stateless functional tests, omitting some. If it fails, further checks are not started until it is fixed. Look at the report to see which tests fail, then reproduce the failure locally as described here	✅ success
Flaky tests	Checks if new added or modified tests are flaky by running them repeatedly, in parallel, with more randomization. Functional tests are run 100 times with address sanitizer, and additional randomization of thread scheduling. Integrational tests are run up to 10 times. If at least once a new test has failed, or was too long, this check will be red. We don't allow flaky tests, read the doc	✅ success
Install packages	Checks that the built packages are installable in a clear environment	✅ success
Integration tests	The integration tests report. In parenthesis the package type is given, and in square brackets are the optional part/total tests	✅ success
Mergeable Check	Checks if all other necessary checks are successful	✅ success
Performance Comparison	Measure changes in query performance. The performance test report is described in detail here. In square brackets are the optional part/total tests	✅ success
Push to Dockerhub	The check for building and pushing the CI related docker images to docker hub	✅ success
SQLTest	There's no description for the check yet, please add it to tests/ci/ci_config.py:CHECK_DESCRIPTIONS	✅ success
SQLancer	Fuzzing tests that detect logical bugs with SQLancer tool	✅ success
Sqllogic	Run clickhouse on the sqllogic test set against sqlite and checks that all statements are passed	✅ success
Stateful tests	Runs stateful functional tests for ClickHouse binaries built in various configurations -- release, debug, with sanitizers, etc	✅ success
Stateless tests	Runs stateless functional tests for ClickHouse binaries built in various configurations -- release, debug, with sanitizers, etc	✅ success
Style Check	Runs a set of checks to keep the code style clean. If some of tests failed, see the related log from the report	✅ success
Unit tests	Runs the unit tests for different release types	✅ success
Upgrade check	Runs stress tests on server version from last release and then tries to upgrade it to the version from the PR. It checks if the new server can successfully startup without any errors, crashes or sanitizer asserts	✅ success

Check name	Description	Status
Bugfix validate check	Checks that either a new test (functional or integration) or there some changed tests that fail with the binary built on master branch	❌ error
Stress test	Runs stateless functional tests concurrently from several clients to detect concurrency-related errors	❌ failure

[kitaisreal]

I cannot reproduce issue without my data:
Create table:

DROP TABLE IF EXISTS mkita_index_test_file;
CREATE TABLE mkita_index_test_file (`_` LowCardinality(String), `_` String, `_` String, `_` LowCardinality(String),
`_` Int64, `_` LowCardinality(String), `_` String, `_` LowCardinality(String),
`_` String, `_` String, `_` String, `_` Int64, `_` LowCardinality(String),
`_` String, `_` Float64, `_` String, `_` LowCardinality(String), `properties` Map(String, String),
`_` Date, `_` Int64) ENGINE = MergeTree() ORDER BY (_) SETTINGS index_granularity = 32768;

Insert data from broken part:

INSERT INTO mkita_index_test_file FROM INFILE 'mkita_index_test_table.tsv.gz'
112618 rows in set. Elapsed: 0.693 sec. Processed 112.62 thousand rows, 8.31 MB (162.45 thousand rows/s., 11.99 MB/s.)
Peak memory usage: 305.98 MiB.

Add and materialize index:

ALTER TABLE mkita_index_test_file ADD INDEX index_properties_bf mapValues(properties) TYPE bloom_filter(0.001) GRANULARITY 1

ALTER TABLE mkita_index_test_file MATERIALIZE INDEX index_properties_bf

In parts index is materialized:

SELECT path, name, marks, formatReadableSize(bytes_on_disk) FROM system.parts WHERE table = 'mkita_index_test_file' AND active = 1

┌─path──────────────────────────────────────────────────────────────────────────────────────────────┬─name─────────────┬─marks─┬─formatReadableSize(bytes_on_disk)─┐
│ /var/clickhouse/disk_s3_metadata/store/763/76312eb4-b3a5-4363-a6c4-bfad512ff54f/20231122_1_1_0_2/ │ 20231122_1_1_0_2 │    11 │ 5.38 MiB                          │
└───────────────────────────────────────────────────────────────────────────────────────────────────┴──────────────────┴───────┴───────────────────────────────────┘

Materialize index once again (amount of granules in part changes):

ALTER TABLE mkita_index_test_file MATERIALIZE INDEX index_properties_bf

In system.parts marks size changed in this part:

SELECT path, name, marks, formatReadableSize(bytes_on_disk) FROM system.parts WHERE table = 'mkita_index_test_file' AND active = 1

┌─path──────────────────────────────────────────────────────────────────────────────────────────────┬─name─────────────┬─marks─┬─formatReadableSize(bytes_on_disk)─┐
│ /var/clickhouse/disk_s3_metadata/store/763/76312eb4-b3a5-4363-a6c4-bfad512ff54f/20231122_1_1_0_3/ │ 20231122_1_1_0_3 │     7 │ 5.38 MiB                          │
└───────────────────────────────────────────────────────────────────────────────────────────────────┴──────────────────┴───────┴───────────────────────────────────┘

Check system.data_skipping_indices table, index looks broken:

SELECT * FROM system.data_skipping_indices

┌─database─┬─table─────────────────┬─name────────────────┬─type─────────┬─type_full───────────┬─expr──────────────────┬─granularity─┬─data_compressed_bytes─┬─data_uncompressed_bytes─┬─marks─┐
│ default  │ mkita_index_test_file │ index_properties_bf │ bloom_filter │ bloom_filter(0.001) │ mapValues(properties) │           1 │                207219 │                       0 │    94 │
└──────────┴───────────────────────┴─────────────────────┴──────────────┴─────────────────────┴───────────────────────┴─────────────┴───────────────────────┴─────────────────────────┴───────┘

After than each next index materialization will change marks size size:

SELECT path, name, marks, formatReadableSize(bytes_on_disk) FROM system.parts WHERE table = 'mkita_index_test_file' AND active = 1

SELECT
    path,
    name,
    marks,
    formatReadableSize(bytes_on_disk)
FROM system.parts
WHERE (table = 'mkita_index_test_file') AND (active = 1)

Query id: ba7946c2-de6a-4754-ad65-f19cd4c9290a

┌─path──────────────────────────────────────────────────────────────────────────────────────────────┬─name─────────────┬─marks─┬─formatReadableSize(bytes_on_disk)─┐
│ /var/clickhouse/disk_s3_metadata/store/763/76312eb4-b3a5-4363-a6c4-bfad512ff54f/20231122_1_1_0_4/ │ 20231122_1_1_0_4 │     8 │ 5.42 MiB                          │
└───────────────────────────────────────────────────────────────────────────────────────────────────┴──────────────────┴───────┴───────────────────────────────────┘

SELECT path, name, marks, formatReadableSize(bytes_on_disk) FROM system.parts WHERE table = 'mkita_index_test_file' AND active = 1

┌─path──────────────────────────────────────────────────────────────────────────────────────────────┬─name─────────────┬─marks─┬─formatReadableSize(bytes_on_disk)─┐
│ /var/clickhouse/disk_s3_metadata/store/763/76312eb4-b3a5-4363-a6c4-bfad512ff54f/20231122_1_1_0_5/ │ 20231122_1_1_0_5 │     7 │ 5.38 MiB                          │
└───────────────────────────────────────────────────────────────────────────────────────────────────┴──────────────────┴───────┴───────────────────────────────────┘

But index is stil broken:

SELECT * FROM system.data_skipping_indices

┌─database─┬─table─────────────────┬─name────────────────┬─type─────────┬─type_full───────────┬─expr──────────────────┬─granularity─┬─data_compressed_bytes─┬─data_uncompressed_bytes─┬─marks─┐
│ default  │ mkita_index_test_file │ index_properties_bf │ bloom_filter │ bloom_filter(0.001) │ mapValues(properties) │           1 │                207219 │                       0 │    94 │
└──────────┴───────────────────────┴─────────────────────┴──────────────┴─────────────────────┴───────────────────────┴─────────────┴───────────────────────┴─────────────────────────┴───────┘

CompactPart writer logs during this process:

Create table and insert data from file. We make single write with 112618 block size:

MergedBlockOutputStream::MergedBlockOutputStream name all_1_1_0
MergeTreeDataPartWriterCompact::MergeTreeDataPartWriterCompact 0x7f9068cc7c00 part name: all_1_1_0
MergeTreeDataPartWriterCompact::write 0x7f9068cc7c00 block size: 112618
MergeTreeDataPartWriterCompact::finish 0x7f9068cc7c00 marks: 8 total rows: 112618 written rows: 112618
index granularity marks rows partial sums: 16088, 32176, 48264, 64352, 80440, 96528, 112618, 112618

Add index and materialize it:

MutateTask::prepare MutateAllPartColumnsTask
MergedBlockOutputStream::MergedBlockOutputStream name all_1_1_0_2
MergeTreeDataPartWriterCompact::MergeTreeDataPartWriterCompact 0x7f90564fc000 part name: all_1_1_0_2
MergeTreeDataPartWriterCompact::write 0x7f90564fc000 block size: 16088
MergeTreeDataPartWriterCompact::write 0x7f90564fc000 block size: 16088
MergeTreeDataPartWriterCompact::write 0x7f90564fc000 block size: 16088
MergeTreeDataPartWriterCompact::write 0x7f90564fc000 block size: 16088
MergeTreeDataPartWriterCompact::write 0x7f90564fc000 block size: 16088
MergeTreeDataPartWriterCompact::write 0x7f90564fc000 block size: 16088
MergeTreeDataPartWriterCompact::write 0x7f90564fc000 block size: 16090
MergeTreeDataPartWriterCompact::finish 0x7f90564fc000 marks: 11 total rows: 112618 written rows: 112618
index granularity marks rows partial sums: 17303, 32176, 64944, 80440, 88484, 96528, 100550, 104572, 108594, 112618, 112618

Materialize index once again:

MutateTask::prepare MutateAllPartColumnsTask
MergedBlockOutputStream::MergedBlockOutputStream name all_1_1_0_3
MergeTreeDataPartWriterCompact::MergeTreeDataPartWriterCompact 0x7f91535ad800 part name: all_1_1_0_3
MergeTreeDataPartWriterCompact::write 0x7f91535ad800 block size: 17303
MergeTreeDataPartWriterCompact::write 0x7f91535ad800 block size: 14873
MergeTreeDataPartWriterCompact::write 0x7f91535ad800 block size: 32768
MergeTreeDataPartWriterCompact::write 0x7f91535ad800 block size: 15496
MergeTreeDataPartWriterCompact::write 0x7f91535ad800 block size: 8044
MergeTreeDataPartWriterCompact::write 0x7f91535ad800 block size: 8044
MergeTreeDataPartWriterCompact::write 0x7f91535ad800 block size: 4022
MergeTreeDataPartWriterCompact::write 0x7f91535ad800 block size: 4022
MergeTreeDataPartWriterCompact::write 0x7f91535ad800 block size: 4022
MergeTreeDataPartWriterCompact::write 0x7f91535ad800 block size: 4024
MergeTreeDataPartWriterCompact::finish 0x7f91535ad800 marks: 7 total rows: 112618 written rows: 112618
index granularity marks rows partial sums: 32176, 64944, 97712, 100550, 104572, 112618, 112618

Materialize index once again:

MergedBlockOutputStream::MergedBlockOutputStream name all_1_1_0_4
MergeTreeDataPartWriterCompact::MergeTreeDataPartWriterCompact 0x7f90438ec800 part name: all_1_1_0_4
MergeTreeDataPartWriterCompact::write 0x7f90438ec800 block size: 32176
MergeTreeDataPartWriterCompact::write 0x7f90438ec800 block size: 32768
MergeTreeDataPartWriterCompact::write 0x7f90438ec800 block size: 32768
MergeTreeDataPartWriterCompact::write 0x7f90438ec800 block size: 2838
MergeTreeDataPartWriterCompact::write 0x7f90438ec800 block size: 4022
MergeTreeDataPartWriterCompact::write 0x7f90438ec800 block size: 8046
MergeTreeDataPartWriterCompact::finish 0x7f90438ec800 marks: 8 total rows: 112618 written rows: 112618
index granularity marks rows partial sums: 32176, 64944, 81328, 97712, 100819, 104572, 112618, 112618

Materialize index once again:

MutateTask::prepare MutateAllPartColumnsTask
MergedBlockOutputStream::MergedBlockOutputStream name all_1_1_0_5
MergeTreeDataPartWriterCompact::MergeTreeDataPartWriterCompact 0x7f903c75d800 part name: all_1_1_0_5
MergeTreeDataPartWriterCompact::write 0x7f903c75d800 block size: 32176
MergeTreeDataPartWriterCompact::write 0x7f903c75d800 block size: 32768
MergeTreeDataPartWriterCompact::write 0x7f903c75d800 block size: 16384
MergeTreeDataPartWriterCompact::write 0x7f903c75d800 block size: 16384
MergeTreeDataPartWriterCompact::write 0x7f903c75d800 block size: 3107
MergeTreeDataPartWriterCompact::write 0x7f903c75d800 block size: 3753
MergeTreeDataPartWriterCompact::write 0x7f903c75d800 block size: 8046
MergeTreeDataPartWriterCompact::finish 0x7f903c75d800 marks: 7 total rows: 112618 written rows: 112618
index granularity marks rows partial sums: 32176, 64944, 97712, 100881, 104572, 112618, 112618

[kitaisreal]

It can be related #16925.
In my example if I will try to use broken index I have:

EXPLAIN header = 1, indexes = 1 SELECT * FROM mkita_index_test_file WHERE properties['test'] = 'value'

EXPLAIN header = 1, indexes = 1
SELECT *
FROM mkita_index_test_file
WHERE (properties['test']) = 'value'

Query id: de611cc9-ac66-4cf7-93f2-d98032fedf72


Elapsed: 0.037 sec. 

Received exception from server (version 23.11.1):
Code: 33. DB::Exception: Received from localhost:9000. DB::Exception: Too many marks in file ./store/48b/48be9692-76f2-4eb2-a441-b7f664e54fc2/all_1_1_0_4/skp_idx_index_properties_bf.cmrk3, marks expected 7 (bytes size 168). (CANNOT_READ_ALL_DATA)

[kitaisreal]

After fix:

Create table and insert data from file. We make single write with 112618 block size:

MergedBlockOutputStream::MergedBlockOutputStream name all_1_1_0
MergeTreeDataPartWriterCompact::MergeTreeDataPartWriterCompact 0x7f2670db5c00 part name: all_1_1_0
MergeTreeDataPartWriterCompact::write 0x7f2670db5c00 block size: 112618
MergeTreeDataPartWriterCompact::finish 0x7f2670db5c00 marks: 8 total rows: 112618 index granularity marks rows partial sums: 16088, 32176, 48264, 64352, 80440, 96528, 112618, 112618

Materialize index once again:

MutateTask::prepare MutateAllPartColumnsTask
MergedBlockOutputStream::MergedBlockOutputStream name all_1_1_0_2
MergeTreeDataPartWriterCompact::MergeTreeDataPartWriterCompact 0x7f2753dfe800 part name: all_1_1_0_2
MergeTreeDataPartWriterCompact::write 0x7f2753dfe800 block size: 16088
MergeTreeDataPartWriterCompact::write 0x7f2753dfe800 block size: 16088
MergeTreeDataPartWriterCompact::write 0x7f2753dfe800 block size: 16088
MergeTreeDataPartWriterCompact::write 0x7f2753dfe800 block size: 16088
MergeTreeDataPartWriterCompact::write 0x7f2753dfe800 block size: 16088
MergeTreeDataPartWriterCompact::write 0x7f2753dfe800 block size: 16088
MergeTreeDataPartWriterCompact::write 0x7f2753dfe800 block size: 16090
MergeTreeDataPartWriterCompact::finish 0x7f2753dfe800 marks: 8 total rows: 112618 index granularity marks rows partial sums: 16088, 32176, 48264, 64352, 80440, 96528, 112618, 112618

Materialize index once again:

MutateTask::prepare MutateAllPartColumnsTask
MergedBlockOutputStream::MergedBlockOutputStream name all_1_1_0_3
MergeTreeDataPartWriterCompact::MergeTreeDataPartWriterCompact 0x7f263aa49800 part name: all_1_1_0_3
MergeTreeDataPartWriterCompact::write 0x7f263aa49800 block size: 16088
MergeTreeDataPartWriterCompact::write 0x7f263aa49800 block size: 16088
MergeTreeDataPartWriterCompact::write 0x7f263aa49800 block size: 16088
MergeTreeDataPartWriterCompact::write 0x7f263aa49800 block size: 16088
MergeTreeDataPartWriterCompact::write 0x7f263aa49800 block size: 16088
MergeTreeDataPartWriterCompact::write 0x7f263aa49800 block size: 16088
MergeTreeDataPartWriterCompact::write 0x7f263aa49800 block size: 16090
MergeTreeDataPartWriterCompact::finish 0x7f263aa49800 marks: 8 total rows: 112618 index granularity marks rows partial sums: 16088, 32176, 48264, 64352, 80440, 96528, 112618, 112618

[alexey-milovidov]

@CurtizJ, you have merged it without a test:

Bugfix validate check — Changed tests don't reproduce the bug

This is not allowed.