Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

CI Fest #8 #14414

Closed
37 of 73 tasks
qoega opened this issue Sep 2, 2020 · 34 comments
Closed
37 of 73 tasks

CI Fest #8 #14414

qoega opened this issue Sep 2, 2020 · 34 comments
Labels
comp-ci Continuous integration testing Special issue with list of bugs found by CI

Comments

@qoega
Copy link
Member

qoega commented Sep 2, 2020 •
edited by alexey-milovidov

Infrastructure Failures

Test Failures
@qoega qoega added the testing Special issue with list of bugs found by CI label Sep 2, 2020
@alexey-milovidov alexey-milovidov pinned this issue Sep 2, 2020
@qoega qoega closed this as completed Sep 2, 2020
@qoega qoega reopened this Sep 2, 2020
@qoega qoega mentioned this issue Sep 2, 2020
Closed
40 tasks
@alexey-milovidov
Copy link
Member

@qoega Integration test about DNS cache is broken here: #14323

@qoega
Copy link
Member Author

qoega commented Sep 4, 2020

@qoega Integration test about DNS cache is broken here: #14323
More likely test_host_ip_change was flaky all time long and recent decrease in retries from 5 to 3 showed its flakiness.

When I tried to fix it in #14323 it seemed that there is no guarantee, that docker-compose dns removes old IP as soon as we stop one container. And added round robin strategy in resolv.conf and it started to work as expected in most cases.

@alexey-milovidov
Copy link
Member

Pipeline stuck: #14632 #14670 @KochetovNicolai
Test version update, two nodes: #14664 #14668 @alesapin

@4ertus2
Copy link
Contributor

4ertus2 commented Sep 14, 2020

Unknown crash in stress test in CacheDictionary #14697 (comment)

@alesapin
Copy link
Member

00652_replicated_mutations_zookeeper need to check test https://clickhouse-test-reports.s3.yandex.net/14609/393c213866ff56ef98ae2f66ee50afebc9459319/functional_stateless_tests_(thread).html#fail1

@alesapin alesapin mentioned this issue Sep 15, 2020
Merged
@BohuTANG
Copy link
Contributor

test_materialize_mysql_database/test.py::test_materialize_database_ddl_with_empty_transaction_8_0
https://clickhouse-test-reports.s3.yandex.net/14809/2b113f7553f7f73c9c21a15e11999de10dfe5953/integration_tests_(asan).html#fail1

@vitlibar
Copy link
Member

test_materialize_mysql_database/test.py::test_materialize_database_ddl_with_empty_transaction_5_7
https://clickhouse-test-reports.s3.yandex.net/14809/2b113f7553f7f73c9c21a15e11999de10dfe5953/integration_tests_(asan)/integration_run.3.txt.out.log

@akuzm akuzm mentioned this issue Sep 21, 2020
Merged
@azat
Copy link
Collaborator

azat commented Sep 21, 2020

test_s3_with_proxy - https://clickhouse-test-reports.s3.yandex.net/14731/6657be90a1e9621fa60321d891a30266174f8525/integration_tests_(release).html#fail1

@azat
Copy link
Collaborator

azat commented Sep 22, 2020

01370_client_autocomplete_word_break_characters - https://clickhouse-test-reports.s3.yandex.net/15063/3fd85430f2431af8b8ef11a1c80966a0820be6e5/functional_stateless_tests_(thread).html#fail1

@filimonov
Copy link
Contributor

filimonov commented Sep 22, 2020 •
edited

Stress tests ThreadSanitizer: data race /build/obj-x86_64-linux-gnu/../src/Core/SettingsFields.h:43:52 in DB::SettingFieldNumber::operator=(unsigned long) https://clickhouse-test-reports.s3.yandex.net/14265/494e59e7c12e0887a32b2564463a84aea500118b/stress_test_(thread)/stderr.log

@filimonov filimonov mentioned this issue Sep 22, 2020
Merged
This was referenced Sep 23, 2020
Merged
Merged
@filimonov
Copy link
Contributor

Stress tests DB::LazyPipeFDs::close() data race: https://clickhouse-test-reports.s3.yandex.net/14265/e2ee97202bf7b81cae3ab05458b6c4d8609c2617/stress_test_(thread)/stderr.log

@azat
Copy link
Collaborator

azat commented Sep 24, 2020

docker: invalid reference format: repository name must be lowercase

Does not report as tests failure - https://clickhouse-test-reports.s3.yandex.net/15007/1bb547422a46ee0d5d211666e99e22cb079da573/integration_tests_(asan).html

@filimonov filimonov added the comp-ci Continuous integration label Sep 24, 2020
@vitlibar vitlibar mentioned this issue Sep 24, 2020
Merged
@azat
Copy link
Collaborator

azat commented Sep 24, 2020

docker: invalid reference format: repository name must be lowercase

And this also breaks integration tests completely, I guess the problem is missing space before -e:

-e DOCKER_BASE_TAG=15007-371eafea11b0f80e8fca46013f043713aefb329b-e 

Running pytest container as: 'docker run --net=host  --rm --name clickhouse_integration_tests --privileged --volume=/place/sandbox-data/tasks/0/5/780148250/clickhouse-odbc-bridge:/clickhouse-odbc-bridge --volume=/place/sandbox-data/tasks/0/5/780148250/clickhouse:/clickhouse         --volume=/place/sandbox-data/tasks/0/5/780148250/ClickHouse/programs/server:/clickhouse-config --volume=/place/sandbox-data/tasks/0/5/780148250/ClickHouse/tests/integration:/ClickHouse/tests/integration         --volume=clickhouse_integration_tests_volume:/var/lib/docker -e DOCKER_MYSQL_GOLANG_CLIENT_TAG=latest -e DOCKER_MYSQL_JAVA_CLIENT_TAG=latest -e DOCKER_MYSQL_JS_CLIENT_TAG=latest -e DOCKER_MYSQL_PHP_CLIENT_TAG=latest -e DOCKER_POSTGRESQL_JAVA_CLIENT_TAG=latest -e DOCKER_BASE_TAG=15007-371eafea11b0f80e8fca46013f043713aefb329b-e DOCKER_KERBEROS_KDC_TAG=latest -e PYTEST_OPTS='-ss  -rfEp --color=no --durations=0 ' yandex/clickhouse-integration-tests-runner:15007-371eafea11b0f80e8fca46013f043713aefb329b '.

@tavplubix
Copy link
Member

MemorySanitizer - use-of-uninitialized-value in MergedBlockOutputStream::writeWithPermutation(...)
https://clickhouse-test-reports.s3.yandex.net/15240/dce6a436f37713224c242092446e1a9fcf8b4a4a/stress_test_(memory)/stderr.log

@alexey-milovidov
Copy link
Member

@tavplubix #15258

@qoega
Copy link
Member Author

qoega commented Sep 25, 2020 •
edited

MemorySanitizer src/Common/hex.h:57:9 in void writeHexUIntImpl
#15284

@tavplubix
Copy link
Member

MemorySanitizer - use-of-uninitialized-value when logging an exception
https://clickhouse-test-reports.s3.yandex.net/15003/fb826557b7ce3a3015fe00df78609ed42d9c5412/stress_test_(memory)/stderr.log

@tavplubix
Copy link
Member

Stress tests ThreadSanitizer: data race /build/obj-x86_64-linux-gnu/../src/Core/SettingsFields.h:43:52 in DB::SettingFieldNumber::operator=(unsigned long) https://clickhouse-test-reports.s3.yandex.net/14265/494e59e7c12e0887a32b2564463a84aea500118b/stress_test_(thread)/stderr.log

Fixed in #15243

@qoega
Copy link
Member Author

qoega commented Sep 25, 2020

01249_bad_arguments_for_bloom_filter
https://clickhouse-test-reports.s3.yandex.net/0/788b62b931c28b63ae47f1adce4ce5ca80cd3c68/functional_stateless_tests_(release,_databaseatomic).html

@qoega
Copy link
Member Author

qoega commented Sep 25, 2020

01455_shard_leaf_max_rows_bytes_to_read
https://clickhouse-test-reports.s3.yandex.net/0/651518566a527f8a0c225f1d8a130712c8b810ee/functional_stateless_tests_(address).html

@tavplubix
Copy link
Member

01249_bad_arguments_for_bloom_filter
https://clickhouse-test-reports.s3.yandex.net/0/788b62b931c28b63ae47f1adce4ce5ca80cd3c68/functional_stateless_tests_(release,_databaseatomic).html

Already fixed

@akuzm akuzm mentioned this issue Sep 29, 2020
Merged
@4ertus2
Copy link
Contributor

4ertus2 commented Sep 30, 2020

Stress (thread)
https://clickhouse-test-reports.s3.yandex.net/15242/0db8b6d647b67ffb480790626015a13b2b929035/stress_test_(thread).html#fail1

@qoega
Copy link
Member Author

qoega commented Sep 30, 2020 •
edited

#15475
Stress undefined

@KochetovNicolai
Copy link
Member

Crash in quantilesDeterministic #15683

@azat
Copy link
Collaborator

azat commented Oct 6, 2020

00933_test_fix_extra_seek_on_compressed_cache - https://clickhouse-test-reports.s3.yandex.net/15642/e00008e61dc13cedb8e9f4c8afd6501f9f7b6a6e/fast_test.html#fail1 (refs #13747)

@filimonov
Copy link
Contributor

01064_incremental_streaming_from_2_src_with_feedback

fixed in #11692

@azat
Copy link
Collaborator

azat commented Oct 13, 2020 •
edited

00966_live_view_watch_events_http - https://clickhouse-test-reports.s3.yandex.net/15879/f7bf94b5d361d0a54e85673c14e630cfdf8dbe01/functional_stateless_tests_(release,_polymorphic_parts_enabled).html#fail1 - #15945

@KochetovNicolai
Copy link
Member

heap-use-after-free, msan, MergedBlockOutputStream: https://clickhouse-test-reports.s3.yandex.net/15857/1bcacde8d53fe25d5a2d79521402305091bbe033/stress_test_(thread).html#fail1

WARNING: ThreadSanitizer: heap-use-after-free (pid=346)
  Read of size 4 at 0x7b6c055722e0 by thread T148 (mutexes: write M910707081661661768):
    #0 DB::MergedBlockOutputStream::finalizePartOnDisk(std::__1::shared_ptr<DB::IMergeTreeDataPart> const&, DB::NamesAndTypesList&, DB::MergeTreeDataPartChecksums&, bool) /build/obj-x86_64-linux-gnu/../src/Storages/MergeTree/MergedBlockOutputStream.cpp:136:9 (clickhouse+0x1187a67c)
    #1 DB::MergedBlockOutputStream::writeSuffixAndFinalizePart(std::__1::shared_ptr<DB::IMergeTreeDataPart>&, bool, DB::NamesAndTypesList const*, DB::MergeTreeDataPartChecksums*) /build/obj-x86_64-linux-gnu/../src/Storages/MergeTree/MergedBlockOutputStream.cpp:116:9 (clickhouse+0x1187a129)
    #2 DB::MergeTreeDataWriter::writeTempPart(DB::BlockWithPartition&, std::__1::shared_ptr<DB::StorageInMemoryMetadata const> const&) /build/obj-x86_64-linux-gnu/../src/Storages/MergeTree/MergeTreeDataWriter.cpp:324:9 (clickhouse+0x117d5188)
    #3 DB::MergeTreeBlockOutputStream::write(DB::Block const&) /build/obj-x86_64-linux-gnu/../src/Storages/MergeTree/MergeTreeBlockOutputStream.cpp:25:65 (clickhouse+0x116f76dc)


  Previous write of size 8 at 0x7b6c055722e0 by thread T5 (mutexes: write M35361):
    #0 operator delete(void*, unsigned long) <null> (clickhouse+0x8112f1e)
    #1 std::__1::default_delete<DB::StorageMergeTree>::operator()(DB::StorageMergeTree*) const /build/obj-x86_64-linux-gnu/../contrib/libcxx/include/memory:2363:5 (clickhouse+0x118e5718)
    #2 std::__1::__shared_ptr_pointer<DB::StorageMergeTree*, std::__1::default_delete<DB::StorageMergeTree>, std::__1::allocator<DB::StorageMergeTree> >::__on_zero_shared() /build/obj-x86_64-linux-gnu/../contrib/libcxx/include/memory:3536:5 (clickhouse+0x118e5718)
    #3 std::__1::__shared_count::__release_shared() /build/obj-x86_64-linux-gnu/../contrib/libcxx/include/memory:3440:9 (clickhouse+0x10ab62a6)
    #4 std::__1::__shared_weak_count::__release_shared() /build/obj-x86_64-linux-gnu/../contrib/libcxx/include/memory:3482:27 (clickhouse+0x10ab62a6)
    #5 std::__1::shared_ptr<DB::IStorage>::~shared_ptr() /build/obj-x86_64-linux-gnu/../contrib/libcxx/include/memory:4207:19 (clickhouse+0x10ab62a6)
    #6 DB::DatabaseCatalog::TableMarkedAsDropped::~TableMarkedAsDropped() /build/obj-x86_64-linux-gnu/../src/Interpreters/DatabaseCatalog.h:212:12 (clickhouse+0x10ab62a6)
    #7 DB::DatabaseCatalog::dropTableDataTask() /build/obj-x86_64-linux-gnu/../src/Interpreters/DatabaseCatalog.cpp:769:1 (clickhouse+0x10ab62a6)

@tavplubix @CurtizJ

@tavplubix
Copy link
Member

heap-use-after-free, msan, MergedBlockOutputStream: https://clickhouse-test-reports.s3.yandex.net/15857/1bcacde8d53fe25d5a2d79521402305091bbe033/stress_test_(thread).html#fail1

#15984

@akuzm akuzm mentioned this issue Oct 15, 2020
Merged
@filimonov
Copy link
Contributor

double free or corruption (out) and sig_abort near ExpressionTransform / ExpressionActions
#15199 (comment)

@azat
Copy link
Collaborator

azat commented Oct 30, 2020 •
edited

@azat azat mentioned this issue Oct 31, 2020
Merged
@akuzm akuzm mentioned this issue Nov 2, 2020
Merged
@alexey-milovidov
Copy link
Member

alexey-milovidov commented Nov 5, 2020 •
edited 

SELECT 
    check_name,
    test_name, 
    count() AS c, 
    min(check_start_time) AS min_time,
    max(check_start_time) AS max_time,
    round(c * 86400 / (max_time - min_time), 3) AS frequency,
    argMax(commit_sha, check_start_time) AS sha 
FROM `gh-data`.checks
WHERE 
    test_status = 'FAIL'
--    AND check_name LIKE 'Functional%'
    AND pull_request_number = 0
    AND check_start_time >= now() - INTERVAL 100 DAY 
GROUP BY check_name, test_name
HAVING max(check_start_time) > now() - INTERVAL 10 DAY 
ORDER BY c DESC
LIMIT 100
FORMAT Markdown
check_name test_name c min_time max_time frequency sha
Stress test (undefined) Server failed to start 87 2020-07-29 15:43:53 2020-11-05 13:59:06 0.879 39cd1c0
Functional stateless tests (thread) 00840_long_concurrent_select_and_drop_deadlock 45 2020-10-29 17:49:41 2020-11-05 13:53:16 6.583 bea0984
Stress test (thread) exit_signal 34 2020-09-28 22:29:53 2020-11-05 03:34:20 0.914 199c32b
Stress test (address) Server failed to start 25 2020-08-05 01:44:39 2020-10-28 07:56:07 0.297 512dddb
Stress test (memory) Server failed to start 16 2020-08-06 13:22:19 2020-10-27 21:38:33 0.194 e1b8d0d
Testflows check /clickhouse 15 2020-08-11 01:34:36 2020-11-04 10:29:13 0.176 654d127
Testflows check /clickhouse/ldap 12 2020-10-20 20:39:02 2020-11-04 10:29:13 0.823 654d127
Testflows check /clickhouse/ldap/external user directory 11 2020-10-20 20:39:02 2020-11-04 10:29:13 0.755 654d127
Stress test (undefined) exit_signal 10 2020-09-28 16:31:13 2020-11-03 13:04:44 0.279 11fc6fd
Testflows check /clickhouse/ldap/external user directory/user authentications 9 2020-10-20 20:39:02 2020-11-04 10:29:13 0.617 654d127
Functional stateless tests (debug) 01541_max_memory_usage_for_user 8 2020-10-25 02:18:54 2020-11-04 22:15:08 0.739 ef111f1
Testflows check /clickhouse/ldap/external user directory/user authentications/login after ldap external user directory is removed 7 2020-10-20 20:39:02 2020-11-04 10:29:13 0.48 654d127
Functional stateless tests (release, wide parts enabled) 01493_alter_remove_properties_zookeeper 6 2020-11-02 08:01:00 2020-11-02 15:44:59 18.621 14f6ec6
Functional stateless tests (release, wide parts enabled) 00993_system_parts_race_condition_drop_zookeeper 5 2020-10-28 08:21:47 2020-10-29 07:47:22 5.122 ae4d66a
Functional stateless tests (release) 01150_ddl_guard_rwr 4 2020-10-27 15:13:30 2020-11-04 12:04:54 0.508 c82d8dd
Functional stateless tests (release, wide parts enabled) 01150_ddl_guard_rwr 4 2020-10-28 00:50:32 2020-10-28 08:23:25 12.719 8951c9b
Stress test (undefined) stress_test_run 3 2020-09-28 16:31:13 2020-10-28 00:41:56 0.102 eee81ee
Integration tests (thread) test_reload_zookeeper/test.py::test_reload_zookeeper 2 2020-09-19 20:29:51 2020-10-30 12:44:21 0.049 be18302
Functional stateless tests (release) 00993_system_parts_race_condition_drop_zookeeper 2 2020-10-28 17:09:25 2020-10-29 03:26:35 4.666 d46cf39
Functional stateless tests (ubsan) 00993_system_parts_race_condition_drop_zookeeper 2 2020-10-27 14:35:27 2020-10-28 06:42:12 2.979 ba34145
Functional stateless tests (release, DatabaseOrdinary) 01133_max_result_rows 2 2020-10-02 18:25:00 2020-10-31 12:59:20 0.07 db146ee
Functional stateless tests (release, wide parts enabled) 01133_max_result_rows 2 2020-10-29 14:54:32 2020-11-05 09:46:36 0.295 40a9648
Fast test 00933_ttl_with_default 2 2020-08-12 08:30:34 2020-10-28 13:55:30 0.026 8e8bdeb
Functional stateless tests (debug) 01150_ddl_guard_rwr 1 2020-10-26 20:40:35 2020-10-26 20:40:35 inf cc5f15d
Functional stateless tests (memory) 01150_ddl_guard_rwr 1 2020-10-28 08:19:48 2020-10-28 08:19:48 inf 7c4b0e5
Integration tests (release) test_https_replication/test.py::test_replication_after_partition 1 2020-10-28 02:34:41 2020-10-28 02:34:41 inf f573aeb
Functional stateless tests (release, wide parts enabled) 01533_distinct_depends_on_max_threads 1 2020-10-30 06:03:02 2020-10-30 06:03:02 inf 2bc4d7b
Functional stateless tests (release) 01533_distinct_depends_on_max_threads 1 2020-10-30 09:46:17 2020-10-30 09:46:17 inf e06f65e
Integration tests (release) test_reload_zookeeper/test.py::test_reload_zookeeper 1 2020-10-28 08:14:13 2020-10-28 08:14:13 inf 0cb377d
Integration tests (thread) test_input_format_parallel_parsing_memory_tracking/test.py::test_memory_tracking_total 1 2020-10-30 05:51:42 2020-10-30 05:51:42 inf 2bc4d7b
Functional stateless tests (release) 01534_lambda_array_join 1 2020-10-30 09:46:17 2020-10-30 09:46:17 inf e06f65e
Integration tests (thread) test_concurrent_ttl_merges/test.py::test_limited_ttl_merges_two_replicas 1 2020-10-28 00:31:55 2020-10-28 00:31:55 inf 4911f33
Functional stateless tests (release, DatabaseOrdinary) 01455_opentelemetry_distributed 1 2020-10-28 17:29:51 2020-10-28 17:29:51 inf a1f1db7
Functional stateless tests (release) 01307_multiple_leaders_zookeeper 1 2020-10-31 05:16:30 2020-10-31 05:16:30 inf 4b3c995
Functional stateless tests (release) 01305_replica_create_drop_zookeeper 1 2020-11-03 08:11:00 2020-11-03 08:11:00 inf 01898be
Functional stateless tests (address) 01150_ddl_guard_rwr 1 2020-10-28 07:42:45 2020-10-28 07:42:45 inf 6667261
Functional stateless tests (release, wide parts enabled) 01534_lambda_array_join 1 2020-10-30 06:03:02 2020-10-30 06:03:02 inf 2bc4d7b

@akuzm akuzm mentioned this issue Nov 9, 2020
Merged
@nikitamikhaylov
Copy link
Member

Leak in openssl: https://gist.github.com/nikitamikhaylov/3ddede6efb865ff25f905f4d11c9d232

@alexey-milovidov
Copy link
Member

Closed because of too many changes.

@alexey-milovidov alexey-milovidov unpinned this issue Nov 29, 2020
@amosbird amosbird mentioned this issue Sep 10, 2023
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
comp-ci Continuous integration testing Special issue with list of bugs found by CI
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests