Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix flaky test_version_update_after_mutation/test.py::test_upgrade_while_mutation #48783

Merged

Conversation

kssenii
Copy link
Member

@kssenii kssenii commented Apr 14, 2023

Changelog category (leave one):

  • Not for changelog (changelog entry is not required)

Closes #48377.

the test test_version_update_after_mutation/test.py::test_upgrade_while_mutation does
node3.restart_with_latest_version(signal=9, fix_metadata=True)
then on server startup it sees that data/system is not empty
actually there are only some leftovers

data/system
└── part_log
    ├── detached
    └── format_version.txt

because right before server was shutdown, it managed to create only part_log.sql.tmp, but did not manage to rename to part_log.sql

<Debug> SystemLog (system.part_log): Creating new table system.part_log for PartLog

then on server restart it removed part_log.sql.tmp :
<Information> DatabaseOrdinary (system): Removing file /var/lib/clickhouse/metadata/system/part_log.sql.tmp
and then started to move system table from ordinary to atomic db
but did not delete data/system/part_log/detached and data/system/part_log/format_version.txt because it transferred to atomic db only data of tables which were successfully loaded.
Therefore there was an exception on server startup

Cannot drop: filesystem error: in remove: Directory not empty ["/var/lib/clickhouse/data/system/"]. 

@robot-ch-test-poll1 robot-ch-test-poll1 added the pr-not-for-changelog This PR should not be mentioned in the changelog label Apr 14, 2023
@tavplubix tavplubix self-assigned this Apr 14, 2023
Co-authored-by: alesapin <alesapin@clickhouse.com>
@kssenii
Copy link
Member Author

kssenii commented Apr 15, 2023

ClickHouse special build check — 7/8 artifact groups are OK

Apr 14 15:29:46 [10005/11759] Building CXX object src/CMakeFiles/dbms.dir/Processors/Formats/IOutputFormat.cpp.o
Apr 14 15:29:49 [10006/11759] Building CXX object src/CMakeFiles/dbms.dir/Processors/Formats/PullingOutputFormat.cpp.o
Apr 14 15:29:51 [10007/11759] Building CXX object src/CMakeFiles/dbms.dir/Processors/Executors/PipelineExecutor.cpp.o
time="2023-04-14T15:29:56Z" level=error msg="error waiting for container: unexpected EOF"
Traceback (most recent call last):
  File "/home/ubuntu/actions-runner/_work/_temp/build_check/ClickHouse/docker/packager/./packager", line 495, in <module>
    main()
  File "/home/ubuntu/actions-runner/_work/_temp/build_check/ClickHouse/docker/packager/./packager", line 483, in main
    run_docker_image_with_env(
  File "/home/ubuntu/actions-runner/_work/_temp/build_check/ClickHouse/docker/packager/./packager", line 112, in run_docker_image_with_env
    subprocess.check_call(cmd, shell=True)
  File "/usr/lib/python3.10/subprocess.py", line 369, in check_call
    raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command 'docker run --network=host --user=1000:1000 --rm --volume=/home/ubuntu/actions-runner/_work/_temp/build_check/binary_tidy:/output --volume=/home/ubuntu/actions-runner/_work/_temp/build_check/ClickHouse:/build  -e OUTPUT_DIR=/output -e DEB_ARCH=amd64 -e CC=clang-15 -e CXX=clang++-15 -e BUILD_TYPE=Debug -e SCCACHE_BUCKET=clickhouse-builds -e SCCACHE_S3_KEY_PREFIX=ccache/sccache -e CTCACHE_S3_BUCKET=clickhouse-builds -e CTCACHE_S3_FOLDER=ccache/clang-tidy-cache -e NINJA_FLAGS=-k0 -e VERSION_STRING='23.4.1.1' -e CMAKE_FLAGS="$CMAKE_FLAGS -DCMAKE_C_COMPILER=clang-15 -DCMAKE_CXX_COMPILER=clang++-15 -DCOMPILER_CACHE=sccache -DENABLE_CLANG_TIDY=1 -DENABLE_TESTS=1 -DENABLE_EXAMPLES=1 -DENABLE_UTILS=1" -e BUILD_TARGET='all'  clickhouse/binary-builder:latest' returned non-zero exit status 125.

:/

@kssenii kssenii merged commit e312c4c into ClickHouse:master Apr 15, 2023
3 of 5 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
pr-not-for-changelog This PR should not be mentioned in the changelog
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Flaky test test_version_update_after_mutation/test.py::test_upgrade_while_mutation
4 participants