run 01171 test in parallel

[CheSema]

I added MAX_ITERATION and MIN_TIME, MAX_TIME.

Now the logic is folowed:

• run each thread not less that MIN_TIME
• not much that MAX_ITERATION and MAX_TIME
• after that it is checked that MIN_ITERATION has been achieved.

That allows to run fast on release build and up to MAX_TIME on slow builds.

Changelog category (leave one):

• Not for changelog (changelog entry is not required)

Changelog entry (a user-readable short description of the changes that goes to CHANGELOG.md):

...

Documentation entry for user-facing changes

• Documentation is written (mandatory for new features)

Information about CI checks: https://clickhouse.com/docs/en/development/continuous-integration/

CI Settings (Only check the boxes if you know what you are doing):

• Allow: All Required Checks
• Allow: Stateless tests
• Allow: Stateful tests
• Allow: Integration Tests
• Allow: Performance tests
• Allow: All Builds
• Allow: batch 1, 2 for multi-batch jobs
• Allow: batch 3, 4, 5, 6 for multi-batch jobs

---

• Exclude: Style check
• Exclude: Fast test
• Exclude: All with ASAN
• Exclude: All with TSAN, MSAN, UBSAN, Coverage
• Exclude: All with aarch64, release, debug

---

• Do not test
• Woolen Wolfdog
• Upload binaries for special builds
• Disable merge-commit
• Disable CI cache

[Algunenano]

400 seconds seem too much, no matter if in parallel or sequential. Why not break out if a certain number of iterations is reached?

[CheSema]

Why not break out if a certain number of iterations is reached?

It was like that recently. It required 100 iterations.
100 iteration on release take ~100 seconds. In debug it takes more that 10 minutes.
So we need to make this iteration limited by time.

I could add MAX_ITERATION that would be what you want.

[Algunenano]

I see sub-10 second runs in master with release mode: https://s3.amazonaws.com/clickhouse-test-reports/0/8f60b9776fef6e14ca0e4b4a8b88ab329f17d3f2/stateless_tests__release_.html:

01171_mv_select_insert_isolation_long	OK	8.94

[Algunenano]

In fact we can see that it has skyrocketed:

https://play.clickhouse.com/play?user=play#U2VsZWN0IHRvU3RhcnRPZkRheShjaGVja19zdGFydF90aW1lKSBhcyB0LCBhdmcodGVzdF9kdXJhdGlvbl9tcykKZnJvbSBjaGVja3MKd2hlcmUgdGVzdF9uYW1lID0gJzAxMTcxX212X3NlbGVjdF9pbnNlcnRfaXNvbGF0aW9uX2xvbmcnIGFuZCB0ZXN0X3N0YXR1cz0nT0snIEFORCBwdWxsX3JlcXVlc3RfbnVtYmVyID0gMCBhbmQgY2hlY2tfc3RhcnRfdGltZSA+ICcyMDI0LTA3LTAxIDAwOjAwOjAwJyBhbmQgY2hlY2tfbmFtZSA9ICdTdGF0ZWxlc3MgdGVzdHMgKHJlbGVhc2UpJwpHUk9VUCBieSB0Ck9SREVSIEJZIHQgQVND

It was 12 seconds until 10 of July, then it got 50%-100% slower around 10th of June, and now it's 20x slower

[CheSema]

yes, you are right, it is fast in release build

https://play.clickhouse.com/play?user=play#c2VsZWN0IGNoZWNrX25hbWUsIGNvdW50KCkgYXMgYywgcXVhbnRpbGVzKDAuNSwgMC45LCAwLjk5LCAwLjk5OTkpKHRlc3RfZHVyYXRpb25fbXMpLCBtYXgodGVzdF9kdXJhdGlvbl9tcykgYXMgbWF4X2QgZnJvbSBjaGVja3Mgd2hlcmUgcHVsbF9yZXF1ZXN0X251bWJlciA9IDAgYW5kIGNoZWNrX3N0YXJ0X3RpbWUgPiAnMjAyNC0wMS0xMCAxODozMzoxMycgYW5kIHRlc3RfbmFtZSA9ICcwMTE3MV9tdl9zZWxlY3RfaW5zZXJ0X2lzb2xhdGlvbl9sb25nJyBhbmQgdGVzdF9zdGF0dXM9J09LJyBncm91cCBieSBjaGVja19uYW1lIGhhdmluZyBjID4gNTAgb3JkZXIgYnkgbWF4X2Q=

[robot-ch-test-poll]

This is an automated comment for commit 851d4d3 with description of existing statuses. It's updated for the latest CI running

✅ Click here to open a full report in a separate page

Successful checks

Check name	Description	Status
Builds	There's no description for the check yet, please add it to tests/ci/ci_config.py:CHECK_DESCRIPTIONS	✅ success
Fast test	Normally this is the first check that is ran for a PR. It builds ClickHouse and runs most of stateless functional tests, omitting some. If it fails, further checks are not started until it is fixed. Look at the report to see which tests fail, then reproduce the failure locally as described here	✅ success
Flaky tests	Checks if new added or modified tests are flaky by running them repeatedly, in parallel, with more randomization. Functional tests are run 100 times with address sanitizer, and additional randomization of thread scheduling. Integration tests are run up to 10 times. If at least once a new test has failed, or was too long, this check will be red. We don't allow flaky tests, read the doc	✅ success
Stateless tests	Runs stateless functional tests for ClickHouse binaries built in various configurations -- release, debug, with sanitizers, etc	✅ success
Style check	Runs a set of checks to keep the code style clean. If some of tests failed, see the related log from the report	✅ success
Unit tests	Runs the unit tests for different release types	✅ success

[CheSema]

Stateless tests flaky check (asan)
01171_mv_select_insert_isolation_long -- 1 out of 30 has failed:
19/20 : not enough iterations of select_insert_action has been made from 14 until 414
I could make setting more low, but I think it is ok. No need to run it one more time.

[Algunenano]

Kind of complex, but it looks fine. OTOH, the test is flaky so probably not fine.

[CheSema]

Kind of complex, but it looks fine. OTOH, the test is flaky so probably not fine.

Ok, one more run.

[CheSema]

Stateless tests flaky check (asan) — ok