-
Notifications
You must be signed in to change notification settings - Fork 6.6k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Added information about paths to system.merges
.
#8043
Merged
alexey-milovidov
merged 13 commits into
ClickHouse:master
from
excitoon-favorites:systemmergespaths
Dec 8, 2019
Merged
Changes from all commits
Commits
Show all changes
13 commits
Select commit
Hold shift + click to select a range
41c965e
Attempt to add `is_move`, `source_part_paths`, `result_part_path` col…
excitoon b23d411
Filling `result_part_path` column of `system.merges` during the merge.
excitoon ef5c4fb
Remake of `result_part_path`.
excitoon 8dbfcdb
Fixed failing tests.
excitoon 7c75629
Removed `is_move` from `system.merges`.
excitoon 05a05b5
Fixed `result_part_part` of a mutation.
excitoon b64b2cf
Added simple non-replicated tests for `system.merges`.
excitoon 1fbfb50
Fixed updating `result_part_path` while merging and mutating.
excitoon 1513e5a
Added replicated tests for `system.merges`.
excitoon ed2ec2d
Fixed `test_system_merges` to correctly check replicated node set.
excitoon c60493f
Fixed `test_system_merges/test.py::test_merge_simple[replicated]` test.
excitoon bb0de25
Fixed `test_system_merges/test.py::test_mutation_simple[replicated]` …
excitoon 984e818
Fixed Clang build.
excitoon File filter
Filter by extension
Conversations
Failed to load comments.
Jump to
Jump to file
Failed to load files.
Diff view
Diff view
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Empty file.
16 changes: 16 additions & 0 deletions
16
dbms/tests/integration/test_system_merges/configs/config.d/cluster.xml
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,16 @@ | ||
<yandex> | ||
<remote_servers> | ||
<test_cluster> | ||
<shard> | ||
<replica> | ||
<host>node1</host> | ||
<port>9000</port> | ||
</replica> | ||
<replica> | ||
<host>node2</host> | ||
<port>9000</port> | ||
</replica> | ||
</shard> | ||
</test_cluster> | ||
</remote_servers> | ||
</yandex> |
17 changes: 17 additions & 0 deletions
17
dbms/tests/integration/test_system_merges/configs/logs_config.xml
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,17 @@ | ||
<yandex> | ||
<shutdown_wait_unfinished>3</shutdown_wait_unfinished> | ||
<logger> | ||
<level>trace</level> | ||
<log>/var/log/clickhouse-server/log.log</log> | ||
<errorlog>/var/log/clickhouse-server/log.err.log</errorlog> | ||
<size>1000M</size> | ||
<count>10</count> | ||
<stderr>/var/log/clickhouse-server/stderr.log</stderr> | ||
<stdout>/var/log/clickhouse-server/stdout.log</stdout> | ||
</logger> | ||
<part_log> | ||
<database>system</database> | ||
<table>part_log</table> | ||
<flush_interval_milliseconds>500</flush_interval_milliseconds> | ||
</part_log> | ||
</yandex> |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,160 @@ | ||
import pytest | ||
import threading | ||
import time | ||
from helpers.cluster import ClickHouseCluster | ||
|
||
cluster = ClickHouseCluster(__file__) | ||
|
||
node1 = cluster.add_instance('node1', | ||
config_dir='configs', | ||
main_configs=['configs/logs_config.xml'], | ||
with_zookeeper=True, | ||
macros={"shard": 0, "replica": 1} ) | ||
|
||
node2 = cluster.add_instance('node2', | ||
config_dir='configs', | ||
main_configs=['configs/logs_config.xml'], | ||
with_zookeeper=True, | ||
macros={"shard": 0, "replica": 2} ) | ||
|
||
|
||
@pytest.fixture(scope="module") | ||
def started_cluster(): | ||
try: | ||
cluster.start() | ||
yield cluster | ||
|
||
finally: | ||
cluster.shutdown() | ||
|
||
|
||
def split_tsv(data): | ||
return [ x.split("\t") for x in data.splitlines() ] | ||
|
||
|
||
@pytest.mark.parametrize("replicated", [ | ||
"", | ||
"replicated" | ||
]) | ||
def test_merge_simple(started_cluster, replicated): | ||
try: | ||
clickhouse_path = "/var/lib/clickhouse" | ||
name = "test_merge_simple" | ||
nodes = [node1, node2] if replicated else [node1] | ||
engine = "ReplicatedMergeTree('/clickhouse/test_merge_simple', '{replica}')" if replicated else "MergeTree()" | ||
node_check = nodes[-1] | ||
starting_block = 0 if replicated else 1 | ||
|
||
for node in nodes: | ||
node.query(""" | ||
CREATE TABLE {name} | ||
( | ||
`a` Int64 | ||
) | ||
ENGINE = {engine} | ||
ORDER BY sleep(2) | ||
""".format(engine=engine, name=name)) | ||
|
||
node1.query("INSERT INTO {name} VALUES (1)".format(name=name)) | ||
node1.query("INSERT INTO {name} VALUES (2)".format(name=name)) | ||
node1.query("INSERT INTO {name} VALUES (3)".format(name=name)) | ||
|
||
parts = ["all_{}_{}_0".format(x, x) for x in range(starting_block, starting_block+3)] | ||
result_part = "all_{}_{}_1".format(starting_block, starting_block+2) | ||
|
||
def optimize(): | ||
node1.query("OPTIMIZE TABLE {name}".format(name=name)) | ||
|
||
wait = threading.Thread(target=time.sleep, args=(5,)) | ||
wait.start() | ||
t = threading.Thread(target=optimize) | ||
t.start() | ||
|
||
time.sleep(1) | ||
assert split_tsv(node_check.query(""" | ||
SELECT database, table, num_parts, source_part_names, source_part_paths, result_part_name, result_part_path, partition_id, is_mutation | ||
FROM system.merges | ||
WHERE table = '{name}' | ||
""".format(name=name))) == [ | ||
[ | ||
"default", | ||
name, | ||
"3", | ||
"['{}','{}','{}']".format(*parts), | ||
"['{clickhouse}/data/default/{name}/{}/','{clickhouse}/data/default/{name}/{}/','{clickhouse}/data/default/{name}/{}/']".format(*parts, clickhouse=clickhouse_path, name=name), | ||
result_part, | ||
"{clickhouse}/data/default/{name}/{}/".format(result_part, clickhouse=clickhouse_path, name=name), | ||
"all", | ||
"0" | ||
] | ||
] | ||
t.join() | ||
wait.join() | ||
|
||
assert node_check.query("SELECT * FROM system.merges WHERE table = '{name}'".format(name=name)) == "" | ||
|
||
finally: | ||
for node in nodes: | ||
node.query("DROP TABLE {name}".format(name=name)) | ||
|
||
|
||
@pytest.mark.parametrize("replicated", [ | ||
"", | ||
"replicated" | ||
]) | ||
def test_mutation_simple(started_cluster, replicated): | ||
try: | ||
clickhouse_path = "/var/lib/clickhouse" | ||
name = "test_mutation_simple" | ||
nodes = [node1, node2] if replicated else [node1] | ||
engine = "ReplicatedMergeTree('/clickhouse/test_mutation_simple', '{replica}')" if replicated else "MergeTree()" | ||
node_check = nodes[-1] | ||
starting_block = 0 if replicated else 1 | ||
|
||
for node in nodes: | ||
node.query(""" | ||
CREATE TABLE {name} | ||
( | ||
`a` Int64 | ||
) | ||
ENGINE = {engine} | ||
ORDER BY tuple() | ||
""".format(engine=engine, name=name)) | ||
|
||
node1.query("INSERT INTO {name} VALUES (1)".format(name=name)) | ||
part = "all_{}_{}_0".format(starting_block, starting_block) | ||
result_part = "all_{}_{}_0_{}".format(starting_block, starting_block, starting_block+1) | ||
|
||
def alter(): | ||
node1.query("ALTER TABLE {name} UPDATE a = 42 WHERE sleep(2) OR 1".format(name=name)) | ||
|
||
t = threading.Thread(target=alter) | ||
t.start() | ||
|
||
time.sleep(1) | ||
assert split_tsv(node_check.query(""" | ||
SELECT database, table, num_parts, source_part_names, source_part_paths, result_part_name, result_part_path, partition_id, is_mutation | ||
FROM system.merges | ||
WHERE table = '{name}' | ||
""".format(name=name))) == [ | ||
[ | ||
"default", | ||
name, | ||
"1", | ||
"['{}']".format(part), | ||
"['{clickhouse}/data/default/{name}/{}/']".format(part, clickhouse=clickhouse_path, name=name), | ||
result_part, | ||
"{clickhouse}/data/default/{name}/{}/".format(result_part, clickhouse=clickhouse_path, name=name), | ||
"all", | ||
"1" | ||
], | ||
] | ||
t.join() | ||
|
||
time.sleep(1.5) | ||
|
||
assert node_check.query("SELECT * FROM system.merges WHERE table = '{name}'".format(name=name)) == "" | ||
|
||
finally: | ||
for node in nodes: | ||
node.query("DROP TABLE {name}".format(name=name)) |
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This will make the test flacky.