[Green CI] test 03164_s3_settings_for_queries_and_merges is flaky

[divanik]

Changelog category (leave one):

• Not for changelog (changelog entry is not required)

Closes https://github.com/ClickHouse/clickhouse-private/issues/13733

CI Settings (Only check the boxes if you know what you are doing):

• Allow: All Required Checks
• Allow: Stateless tests
• Allow: Stateful tests
• Allow: Integration Tests
• Allow: Performance tests
• Allow: All Builds
• Allow: batch 1, 2 for multi-batch jobs
• Allow: batch 3, 4, 5, 6 for multi-batch jobs

---

• Exclude: Style check
• Exclude: Fast test
• Exclude: All with ASAN
• Exclude: All with TSAN, MSAN, UBSAN, Coverage
• Exclude: All with aarch64, release, debug

---

• Do not test
• Woolen Wolfdog
• Upload binaries for special builds
• Disable merge-commit
• Disable CI cache

[divanik]

We investigated problems in one of reports and concluded that the problem is in retries:

2024.07.24 13:05:56.414547 [ 18830 ] {c1649c1a-3ad4-4da0-9cc2-d603243b5025} <Debug> executeQuery: (from [::1]:36902) (comment: 03164_s3_settings_for_queries_and_merges.sql) INSERT INTO t_compact_bytes_s3 SELECT number, number, number, number, number FROM numbers(512 * 32 * 40); (stage: Complete)
2024.07.24 13:05:56.414878 [ 18830 ] {c1649c1a-3ad4-4da0-9cc2-d603243b5025} <Trace> Planner: Query to stage Complete
2024.07.24 13:05:56.414972 [ 18830 ] {c1649c1a-3ad4-4da0-9cc2-d603243b5025} <Trace> Planner: Query from stage FetchColumns to stage Complete
2024.07.24 13:05:56.433626 [ 122528 ] {c1649c1a-3ad4-4da0-9cc2-d603243b5025} <Trace> test_j3hqh0ct.t_compact_bytes_s3 (fd0093c9-38bb-44b3-9a01-35d2db810477): Trying to reserve 12.50 MiB using storage policy from min volume index 0
2024.07.24 13:05:56.465617 [ 122528 ] {c1649c1a-3ad4-4da0-9cc2-d603243b5025} <Trace> MergedBlockOutputStream: filled checksums all_1_1_0 (state Temporary)
2024.07.24 13:05:57.730947 [ 122528 ] {c1649c1a-3ad4-4da0-9cc2-d603243b5025} <Information> AWSClient: Failed to make request to: http://localhost:11111/test/00170_test/bvh/srcwnvzbwqftmkmyybxqcnbftkzzy: Poco::Exception. Code: 1000, e.code() = 0, Timeout, Stack trace (when copying this message, always include the lines below):
2024.07.24 13:05:57.771755 [ 122528 ] {c1649c1a-3ad4-4da0-9cc2-d603243b5025} <Information> AWSClient: AWSXmlClient: HTTP response code: -1
2024.07.24 13:05:57.771943 [ 122528 ] {c1649c1a-3ad4-4da0-9cc2-d603243b5025} <Information> AWSClient: If the signature check failed. This could be because of a time skew. Attempting to adjust the signer.
2024.07.24 13:05:57.771962 [ 122528 ] {c1649c1a-3ad4-4da0-9cc2-d603243b5025} <Information> AWSClient: Request failed, now waiting 0 ms before attempting again.
2024.07.24 13:05:57.784883 [ 122528 ] {c1649c1a-3ad4-4da0-9cc2-d603243b5025} <Trace> test_j3hqh0ct.t_compact_bytes_s3 (fd0093c9-38bb-44b3-9a01-35d2db810477): Renaming temporary part tmp_insert_all_1_1_0 to all_1_1_0 with tid (1, 1, 00000000-0000-0000-0000-000000000000).
2024.07.24 13:05:57.785194 [ 18830 ] {c1649c1a-3ad4-4da0-9cc2-d603243b5025} <Debug> executeQuery: Read 655360 rows, 5.00 MiB in 1.370733 sec., 478109.15765506483 rows/sec., 3.65 MiB/sec.
2024.07.24 13:05:57.785413 [ 18830 ] {c1649c1a-3ad4-4da0-9cc2-d603243b5025} <Debug> MemoryTracker: Peak memory usage (for query): 52.44 MiB.
2024.07.24 13:05:57.785423 [ 18830 ] {c1649c1a-3ad4-4da0-9cc2-d603243b5025} <Debug> TCPHandler: Processed in 1.371203953 sec.

2024.07.28 19:04:21.858229 [ 97391 ] {3d7d377a-45c5-4375-bf7c-f4301e2bf060::all_1_1_1} <Debug> MergeTask::PrepareStage: Merging 1 parts: from all_1_1_0 to all_1_1_0 into Compact with storage Full
2024.07.28 19:04:21.858368 [ 97391 ] {3d7d377a-45c5-4375-bf7c-f4301e2bf060::all_1_1_1} <Debug> MergeTask::PrepareStage: Selected MergeAlgorithm: Horizontal
2024.07.28 19:04:21.858391 [ 97391 ] {3d7d377a-45c5-4375-bf7c-f4301e2bf060::all_1_1_1} <Debug> MergeTreeSequentialSource: Reading 1281 marks from part all_1_1_0, total 655360 rows starting from the beginning of the part
2024.07.28 19:04:22.369054 [ 4628 ] {3d7d377a-45c5-4375-bf7c-f4301e2bf060::all_1_1_1} <Information> AWSClient: Failed to make request to: http://localhost:11111/test/00170_test/imr/vgjvxospfujolgxbpeppawbysumhz: Poco::Exception. Code: 1000, e.code() = 0, Timeout, Stack trace (when copying this message, always include the lines below):
2024.07.28 19:04:22.578243 [ 4628 ] {3d7d377a-45c5-4375-bf7c-f4301e2bf060::all_1_1_1} <Information> AWSClient: AWSXmlClient: HTTP response code: -1
2024.07.28 19:04:22.578266 [ 4628 ] {3d7d377a-45c5-4375-bf7c-f4301e2bf060::all_1_1_1} <Information> AWSClient: If the signature check failed. This could be because of a time skew. Attempting to adjust the signer.
2024.07.28 19:04:22.578279 [ 4628 ] {3d7d377a-45c5-4375-bf7c-f4301e2bf060::all_1_1_1} <Information> AWSClient: Request failed, now waiting 0 ms before attempting again.
2024.07.28 19:04:22.667304 [ 97391 ] {3d7d377a-45c5-4375-bf7c-f4301e2bf060::all_1_1_1} <Debug> MergeTask::MergeProjectionsStage: Merge sorted 655360 rows, containing 5 columns (5 merged, 0 gathered) in 0.809168617 sec., 809917.7182992504 rows/sec., 15.45 MiB/sec.
2024.07.28 19:04:23.464256 [ 97391 ] {3d7d377a-45c5-4375-bf7c-f4301e2bf060::all_1_1_1} <Trace> MergedBlockOutputStream: filled checksums all_1_1_1 (state Temporary)
2024.07.28 19:04:24.069720 [ 97391 ] {3d7d377a-45c5-4375-bf7c-f4301e2bf060::all_1_1_1} <Trace> test_pzo8ggrf.t_compact_bytes_s3 (3d7d377a-45c5-4375-bf7c-f4301e2bf060): Renaming temporary part tmp_merge_all_1_1_1 to all_1_1_1 with tid (1, 1, 00000000-0000-0000-0000-000000000000).
2024.07.28 19:04:24.069816 [ 97391 ] {3d7d377a-45c5-4375-bf7c-f4301e2bf060::all_1_1_1} <Trace> test_pzo8ggrf.t_compact_bytes_s3 (3d7d377a-45c5-4375-bf7c-f4301e2bf060) (MergerMutator): Merged 1 parts: [all_1_1_0, all_1_1_0] -> all_1_1_1

That's why subtraction of errors count should help.

[robot-clickhouse]

This is an automated comment for commit aefed7c with description of existing statuses. It's updated for the latest CI running

✅ Click here to open a full report in a separate page

Successful checks

Check name	Description	Status
Builds	There's no description for the check yet, please add it to tests/ci/ci_config.py:CHECK_DESCRIPTIONS	✅ success
Fast test	Normally this is the first check that is ran for a PR. It builds ClickHouse and runs most of stateless functional tests, omitting some. If it fails, further checks are not started until it is fixed. Look at the report to see which tests fail, then reproduce the failure locally as described here	✅ success
Flaky tests	Checks if new added or modified tests are flaky by running them repeatedly, in parallel, with more randomization. Functional tests are run 100 times with address sanitizer, and additional randomization of thread scheduling. Integration tests are run up to 10 times. If at least once a new test has failed, or was too long, this check will be red. We don't allow flaky tests, read the doc	✅ success
Stateless tests	Runs stateless functional tests for ClickHouse binaries built in various configurations -- release, debug, with sanitizers, etc	✅ success
Stress test	Runs stateless functional tests concurrently from several clients to detect concurrency-related errors	✅ success
Style check	Runs a set of checks to keep the code style clean. If some of tests failed, see the related log from the report	✅ success
Unit tests	Runs the unit tests for different release types	✅ success
Upgrade check	Runs stress tests on server version from last release and then tries to upgrade it to the version from the PR. It checks if the new server can successfully startup without any errors, crashes or sanitizer asserts	✅ success