gitlab-ci: it is needed correct hosts cleanup in gitlab-ci jobs #5036
Comments
avtikhon
added
bug
|
Also found the same issue on regular testing hosts Found the job that saved the directories with root permissions The issue appeared because the current job uses the script type of run in gitlab-runner to run docker itself as additional command, that caused the gitlab-runner cleanup of the workspace to be outside the docker (in opposite to the jobs when docker set as the job type run, in this mode cleanup runs inside the docker and no fails exists ever) and it fails to remove files created by root in the docker image before and which were shared to global host by the following: The current issue doesn't appear on other jobs that use script in gitlab-runner to run docker itself (like packing jobs https://gitlab.com/tarantool/tarantool/-/jobs/578370621), because it uses shared workspaces in RO mode with out-of-source build, which avoids of creating files in the shared directory with root permissions. To fix the issue need to run docker images through the gitlab-runner only in RO mode with OOS builds in it either use the docker type gitlab-runner jobs runs. Anyway script type gitlab-runner jobs with additional calls to dockers can't be control for the branches of developers, to avoid of it need to make some workaround to cleanup the working paths for each job with script mode in it. Currently the only found workaround is to add crontab to running host with changing ownership of the workspaces paths recursively either setup gitlab-runner configuration as described in: |
avtikhon
changed the title
avtikhon
added a commit
that referenced
this issue
Jun 3, 2020
Found the issue on regular testing hosts: https://gitlab.com/tarantool/tarantool/-/jobs/577884238 Fetching changes... 00:04 Reinitialized existing Git repository in /home/gitlab-runner/builds/zzyC6hh5/0/tarantool/tarantool/.git/ Checking out 8ff7f32 as ... warning: failed to remove CMakeFiles/Makefile.cmake Found the job that saved the directories with root permissions https://gitlab.com/tarantool/tarantool/-/jobs/577768553 The issue appeared because the issue caused job used the script runner to run docker in it as additional command. It caused the the gitlab-runner default workspace cleanup to be outside the docker (in opposite to the jobs when docker runner used, the cleanup runs inside the docker and no fails exists ever). Cleanup failed to remove files created by root inside the docker container before and which were shared to global host. To fix the issue need to run docker images using the gitlab-runner only in RO mode with Out-Of-Source builds in it. Either use the docker runners when docker containers are needed. Anyway script runner jobs with additional calls to dockers can't be control for the branches of developers, to avoid of it need to make local cleanup instead of default to the working paths for each job with script runners use. Decided to setup gitlab-runner configuration as described in: https://docs.gitlab.com/ce/ci/yaml/README.html#git-clean-flags Closes #5036
avtikhon
added a commit
that referenced
this issue
Jun 3, 2020
Found the issue on regular testing hosts: https://gitlab.com/tarantool/tarantool/-/jobs/577884238 Fetching changes... 00:04 Reinitialized existing Git repository in /home/gitlab-runner/builds/zzyC6hh5/0/tarantool/tarantool/.git/ Checking out 8ff7f32 as ... warning: failed to remove CMakeFiles/Makefile.cmake Found the job that saved the directories with root permissions https://gitlab.com/tarantool/tarantool/-/jobs/577768553 The issue appeared because the issue caused job used the script runner to run docker in it as additional command. It caused the the gitlab-runner default workspace cleanup to be outside the docker (in opposite to the jobs when docker runner used, the cleanup runs inside the docker and no fails exists ever). Cleanup failed to remove files created by root inside the docker container before and which were shared to global host. To fix the issue need to run docker images using the gitlab-runner only in RO mode with Out-Of-Source builds in it. Either use the docker runners when docker containers are needed. Anyway script runner jobs with additional calls to dockers can't be control for the branches of developers, to avoid of it need to make local cleanup instead of default to the working paths for each job with script runners use. Decided to setup gitlab-runner configuration as described in: https://docs.gitlab.com/ce/ci/yaml/README.html#git-clean-flags Closes #5036
avtikhon
added a commit
that referenced
this issue
Jun 3, 2020
Found the issue on regular testing hosts: https://gitlab.com/tarantool/tarantool/-/jobs/577884238 Fetching changes... 00:04 Reinitialized existing Git repository in /home/gitlab-runner/builds/zzyC6hh5/0/tarantool/tarantool/.git/ Checking out 8ff7f32 as ... warning: failed to remove CMakeFiles/Makefile.cmake Found the job that saved the directories with root permissions https://gitlab.com/tarantool/tarantool/-/jobs/577768553 The issue appeared because the issue caused job used the script runner to run docker in it as additional command. It caused the the gitlab-runner default workspace cleanup to be outside the docker (in opposite to the jobs when docker runner used, the cleanup runs inside the docker and no fails exists ever). Cleanup failed to remove files created by root inside the docker container before and which were shared to global host. To fix the issue need to run docker images using the gitlab-runner only in RO mode with Out-Of-Source builds in it. Either use the docker runners when docker containers are needed. Anyway script runner jobs with additional calls to dockers can't be control for the branches of developers, to avoid of it need to make local cleanup instead of default to the working paths for each job with script runners use. Decided to setup gitlab-runner configuration as described in: https://docs.gitlab.com/ce/ci/yaml/README.html#git-clean-flags Closes #5036
avtikhon
added a commit
that referenced
this issue
Jun 3, 2020
Found the issue on regular testing hosts: https://gitlab.com/tarantool/tarantool/-/jobs/577884238#L7 Fetching changes... 00:04 Reinitialized existing Git repository in /home/gitlab-runner/builds/zzyC6hh5/0/tarantool/tarantool/.git/ Checking out 8ff7f32 as ... warning: failed to remove CMakeFiles/Makefile.cmake Found the job that saved the directories with root permissions https://gitlab.com/tarantool/tarantool/-/jobs/577768553 The issue appeared because the issue caused job used the script runner to run docker in it as additional command. It caused the the gitlab-runner default workspace cleanup to be outside the docker (in opposite to the jobs when docker runner used, the cleanup runs inside the docker and no fails exists ever). Cleanup failed to remove files created by root inside the docker container before and which were shared to global host. To fix the issue need to run docker images using the gitlab-runner only in RO mode with Out-Of-Source builds in it. Either use the docker runners when docker containers are needed. Anyway script runner jobs with additional calls to dockers can't be control for the branches of developers, to avoid of it need to make local cleanup instead of default to the working paths for each job with script runners use. Decided to setup gitlab-runner configuration as described in: https://docs.gitlab.com/ce/ci/yaml/README.html#git-clean-flags Also got the issue with left data from previous builds at the submodule pathes, like here: https://gitlab.com/tarantool/tarantool/-/jobs/574199256#L3141 Undefined symbols for architecture x86_64: "_u_isprint_66", referenced from: _yaml_emitter_is_printable in libyaml_static.a(emitter.c.o) ld: symbol(s) not found for architecture x86_64 clang: error: linker command failed with exit code 1 (use -v to see invocation) To fix it was added the command to clean all available git submodules: git submodule foreach git clean -ffdx Closes #5036
avtikhon
added a commit
that referenced
this issue
Jun 3, 2020
Found the issue on regular testing hosts: https://gitlab.com/tarantool/tarantool/-/jobs/577884238#L7 Fetching changes... 00:04 Reinitialized existing Git repository in /home/gitlab-runner/builds/zzyC6hh5/0/tarantool/tarantool/.git/ Checking out 8ff7f32 as ... warning: failed to remove CMakeFiles/Makefile.cmake Found the job that saved the directories with root permissions https://gitlab.com/tarantool/tarantool/-/jobs/577768553 The issue appeared because the issue caused job used the script runner to run docker in it as additional command. It caused the the gitlab-runner default workspace cleanup to be outside the docker (in opposite to the jobs when docker runner used, the cleanup runs inside the docker and no fails exists ever). Cleanup failed to remove files created by root inside the docker container before and which were shared to global host. To fix the issue need to run docker images using the gitlab-runner only in RO mode with Out-Of-Source builds in it. Either use the docker runners when docker containers are needed. Anyway script runner jobs with additional calls to dockers can't be control for the branches of developers, to avoid of it need to make local cleanup instead of default to the working paths for each job with script runners use. Decided to setup gitlab-runner configuration as described in: https://docs.gitlab.com/ce/ci/yaml/README.html#git-clean-flags Also got the issue with left data from previous builds at the submodule pathes, like here: https://gitlab.com/tarantool/tarantool/-/jobs/574199256#L3141 Undefined symbols for architecture x86_64: "_u_isprint_66", referenced from: _yaml_emitter_is_printable in libyaml_static.a(emitter.c.o) ld: symbol(s) not found for architecture x86_64 clang: error: linker command failed with exit code 1 (...) To fix it was added the command to clean all available git submodules: git submodule foreach git clean -ffdx Closes #5036
avtikhon
added a commit
that referenced
this issue
Jun 4, 2020
Found the issue on regular testing hosts: https://gitlab.com/tarantool/tarantool/-/jobs/577884238#L7 Fetching changes... 00:04 Reinitialized existing Git repository in /home/gitlab-runner/builds/zzyC6hh5/0/tarantool/tarantool/.git/ Checking out 8ff7f32 as ... warning: failed to remove CMakeFiles/Makefile.cmake Found the job that saved the directories with root permissions https://gitlab.com/tarantool/tarantool/-/jobs/577768553 The issue appeared because the job that saved directories with root permissions used the 'shell' runner to run docker container inside. It caused the gitlab-runner to run the default workspace cleanup outside the docker container. In opposite to it, when 'docker' runner is used, the cleanup routine runs inside the docker container and no fails ever exist, because the root permissions are used in the docker container and this is the same root permissions for the host. As the result using 'shell' runner, cleanup routine failed to remove files created by root inside the docker container and which were shared to global host with the same permissions, because gitlab-runner runs the 'shell' runner by regular 'gitlab-runner' user, but not by root. To fix the issue need to run docker containers using the gitlab-runner only in RO mode with Out-Of-Source builds in it. Either use the 'docker' runners when docker containers are needed. Anyway 'shell' runner jobs with additional calls to docker containers can't be control for the branches of developers, to avoid of it need to make local cleanup routine instead of default to the working paths for each job with 'shell' runners use. Decided to setup gitlab-runner configuration as described in: https://docs.gitlab.com/ce/ci/yaml/README.html#git-clean-flags Also got the issue with left data from previous builds at the submodule pathes, like here: https://gitlab.com/tarantool/tarantool/-/jobs/574199256#L3141 Undefined symbols for architecture x86_64: "_u_isprint_66", referenced from: _yaml_emitter_is_printable in libyaml_static.a(emitter.c.o) ld: symbol(s) not found for architecture x86_64 clang: error: linker command failed with exit code 1 (...) To fix it was added the command to clean all available git submodules: git submodule foreach git clean -ffdx Closes #5036
avtikhon
added a commit
that referenced
this issue
Jun 10, 2020
Found the issue on regular testing hosts: https://gitlab.com/tarantool/tarantool/-/jobs/577884238#L7 Fetching changes... 00:04 Reinitialized existing Git repository in /home/gitlab-runner/builds/zzyC6hh5/0/tarantool/tarantool/.git/ Checking out 8ff7f32 as ... warning: failed to remove CMakeFiles/Makefile.cmake Found the job that saved the directories with root permissions https://gitlab.com/tarantool/tarantool/-/jobs/577768553 The issue appeared because the job that saved directories with root permissions used the 'shell' runner to run docker container inside. It caused the gitlab-runner to run the default workspace cleanup outside the docker container. In opposite to it, when 'docker' runner is used, the cleanup routine runs inside the docker container and no fails ever exist, because the root permissions are used in the docker container and this is the same root permissions for the host. As the result using 'shell' runner, cleanup routine failed to remove files created by root inside the docker container and which were shared to global host with the same permissions, because gitlab-runner runs the 'shell' runner by regular 'gitlab-runner' user, but not by root. To fix the issue need to run docker containers using the gitlab-runner only in RO mode with Out-Of-Source builds in it. Either use the 'docker' runners when docker containers are needed. Anyway 'shell' runner jobs with additional calls to docker containers can't be control for the branches of developers, to avoid of it need to make local cleanup routine instead of default to the working paths for each job with 'shell' runners use. Decided to setup gitlab-runner configuration as described in: https://docs.gitlab.com/ce/ci/yaml/README.html#git-clean-flags Also got the issue with left data from previous builds at the submodule pathes, like here: https://gitlab.com/tarantool/tarantool/-/jobs/574199256#L3141 Undefined symbols for architecture x86_64: "_u_isprint_66", referenced from: _yaml_emitter_is_printable in libyaml_static.a(emitter.c.o) ld: symbol(s) not found for architecture x86_64 clang: error: linker command failed with exit code 1 (...) To fix it was added the command to clean all available git submodules: git submodule foreach git clean -ffdx Closes #5036
avtikhon
added a commit
that referenced
this issue
Jun 10, 2020
Found the issue on regular testing hosts: https://gitlab.com/tarantool/tarantool/-/jobs/577884238#L7 Fetching changes... 00:04 Reinitialized existing Git repository in /home/gitlab-runner/builds/zzyC6hh5/0/tarantool/tarantool/.git/ Checking out 8ff7f32 as ... warning: failed to remove CMakeFiles/Makefile.cmake Found the job that saved the directories with root permissions https://gitlab.com/tarantool/tarantool/-/jobs/577768553 The issue appeared because the job that saved directories with root permissions used the 'shell' runner to run docker container inside. It caused the gitlab-runner to run the default workspace cleanup outside the docker container. In opposite to it, when 'docker' runner is used, the cleanup routine runs inside the docker container and no fails ever exist, because the root permissions are used in the docker container and this is the same root permissions for the host. As the result using 'shell' runner, cleanup routine failed to remove files created by root inside the docker container and which were shared to global host with the same permissions, because gitlab-runner runs the 'shell' runner by regular 'gitlab-runner' user, but not by root. To fix the issue need to run docker containers using the gitlab-runner only in RO mode with Out-Of-Source builds in it. Either use the 'docker' runners when docker containers are needed. Anyway 'shell' runner jobs with additional calls to docker containers can't be control for the branches of developers, to avoid of it need to make local cleanup routine instead of default to the working paths for each job with 'shell' runners use. Decided to setup gitlab-runner configuration as described in: https://docs.gitlab.com/ce/ci/yaml/README.html#git-clean-flags Also got the issue with left data from previous builds at the submodule pathes, like here: https://gitlab.com/tarantool/tarantool/-/jobs/574199256#L3141 Undefined symbols for architecture x86_64: "_u_isprint_66", referenced from: _yaml_emitter_is_printable in libyaml_static.a(emitter.c.o) ld: symbol(s) not found for architecture x86_64 clang: error: linker command failed with exit code 1 (...) To fix it was added the command to clean all available git submodules: git submodule foreach git clean -ffdx Closes #5036 (cherry picked from commit a654760)
avtikhon
added a commit
that referenced
this issue
Jun 11, 2020
Found the issue on regular testing hosts: https://gitlab.com/tarantool/tarantool/-/jobs/577884238#L7 Fetching changes... 00:04 Reinitialized existing Git repository in /home/gitlab-runner/builds/zzyC6hh5/0/tarantool/tarantool/.git/ Checking out 8ff7f32 as ... warning: failed to remove CMakeFiles/Makefile.cmake Found the job that saved the directories with root permissions https://gitlab.com/tarantool/tarantool/-/jobs/577768553 The issue appeared because the job that saved directories with root permissions used the 'shell' runner to run docker container inside. It caused the gitlab-runner to run the default workspace cleanup outside the docker container. In opposite to it, when 'docker' runner is used, the cleanup routine runs inside the docker container and no fails ever exist, because the root permissions are used in the docker container and this is the same root permissions for the host. As the result using 'shell' runner, cleanup routine failed to remove files created by root inside the docker container and which were shared to global host with the same permissions, because gitlab-runner runs the 'shell' runner by regular 'gitlab-runner' user, but not by root. To fix the issue need to run docker containers using the gitlab-runner only in RO mode with Out-Of-Source builds in it. Either use the 'docker' runners when docker containers are needed. Anyway 'shell' runner jobs with additional calls to docker containers can't be control for the branches of developers, to avoid of it need to make local cleanup routine instead of default to the working paths for each job with 'shell' runners use. Decided to setup gitlab-runner configuration as described in: https://docs.gitlab.com/ce/ci/yaml/README.html#git-clean-flags Also got the issue with left data from previous builds at the submodule pathes, like here: https://gitlab.com/tarantool/tarantool/-/jobs/574199256#L3141 Undefined symbols for architecture x86_64: "_u_isprint_66", referenced from: _yaml_emitter_is_printable in libyaml_static.a(emitter.c.o) ld: symbol(s) not found for architecture x86_64 clang: error: linker command failed with exit code 1 (...) To fix it was added the command to clean all available git submodules: git submodule foreach git clean -ffdx Closes #5036 (cherry picked from commit a654760)
avtikhon
added a commit
that referenced
this issue
Jun 11, 2020
Found the issue on regular testing hosts: https://gitlab.com/tarantool/tarantool/-/jobs/577884238#L7 Fetching changes... 00:04 Reinitialized existing Git repository in /home/gitlab-runner/builds/zzyC6hh5/0/tarantool/tarantool/.git/ Checking out 8ff7f32 as ... warning: failed to remove CMakeFiles/Makefile.cmake Found the job that saved the directories with root permissions https://gitlab.com/tarantool/tarantool/-/jobs/577768553 The issue appeared because the job that saved directories with root permissions used the 'shell' runner to run docker container inside. It caused the gitlab-runner to run the default workspace cleanup outside the docker container. In opposite to it, when 'docker' runner is used, the cleanup routine runs inside the docker container and no fails ever exist, because the root permissions are used in the docker container and this is the same root permissions for the host. As the result using 'shell' runner, cleanup routine failed to remove files created by root inside the docker container and which were shared to global host with the same permissions, because gitlab-runner runs the 'shell' runner by regular 'gitlab-runner' user, but not by root. To fix the issue need to run docker containers using the gitlab-runner only in RO mode with Out-Of-Source builds in it. Either use the 'docker' runners when docker containers are needed. Anyway 'shell' runner jobs with additional calls to docker containers can't be control for the branches of developers, to avoid of it need to make local cleanup routine instead of default to the working paths for each job with 'shell' runners use. Decided to setup gitlab-runner configuration as described in: https://docs.gitlab.com/ce/ci/yaml/README.html#git-clean-flags Also got the issue with left data from previous builds at the submodule pathes, like here: https://gitlab.com/tarantool/tarantool/-/jobs/574199256#L3141 Undefined symbols for architecture x86_64: "_u_isprint_66", referenced from: _yaml_emitter_is_printable in libyaml_static.a(emitter.c.o) ld: symbol(s) not found for architecture x86_64 clang: error: linker command failed with exit code 1 (...) Also got issues with left files from previously tested branches, like here: https://gitlab.com/tarantool/tarantool/-/jobs/590573606#L3718 [087] small/rlist.test [087] TAP13 parse failed (Missing plan in the TAP source). [087] [087] No result file (small/rlist.result) found. [087] Run the test with --update-result option to write the new result file. [087] [ fail ] [087] small/static.test [087] TAP13 parse failed (Missing plan in the TAP source). [087] [087] No result file (small/static.result) found. [087] Run the test with --update-result option to write the new result file. [087] [ fail ] To fix it was added the command to clean all available git submodules: git submodule foreach git clean -ffdx Closes #5036
avtikhon
added a commit
that referenced
this issue
Jun 11, 2020
Found the issue on regular testing hosts: https://gitlab.com/tarantool/tarantool/-/jobs/577884238#L7 Fetching changes... 00:04 Reinitialized existing Git repository in /home/gitlab-runner/builds/zzyC6hh5/0/tarantool/tarantool/.git/ Checking out 8ff7f32 as ... warning: failed to remove CMakeFiles/Makefile.cmake Found the job that saved the directories with root permissions https://gitlab.com/tarantool/tarantool/-/jobs/577768553 The issue appeared because the job that saved directories with root permissions used the 'shell' runner to run docker container inside. It caused the gitlab-runner to run the default workspace cleanup outside the docker container. In opposite to it, when 'docker' runner is used, the cleanup routine runs inside the docker container and no fails ever exist, because the root permissions are used in the docker container and this is the same root permissions for the host. As the result using 'shell' runner, cleanup routine failed to remove files created by root inside the docker container and which were shared to global host with the same permissions, because gitlab-runner runs the 'shell' runner by regular 'gitlab-runner' user, but not by root. To fix the issue need to run docker containers using the gitlab-runner only in RO mode with Out-Of-Source builds in it. Either use the 'docker' runners when docker containers are needed. Anyway 'shell' runner jobs with additional calls to docker containers can't be control for the branches of developers, to avoid of it need to make local cleanup routine instead of default to the working paths for each job with 'shell' runners use. Decided to setup gitlab-runner configuration as described in: https://docs.gitlab.com/ce/ci/yaml/README.html#git-clean-flags Also got the issue with left data from previous builds at the submodule pathes, like here: https://gitlab.com/tarantool/tarantool/-/jobs/574199256#L3141 Undefined symbols for architecture x86_64: "_u_isprint_66", referenced from: _yaml_emitter_is_printable in libyaml_static.a(emitter.c.o) ld: symbol(s) not found for architecture x86_64 clang: error: linker command failed with exit code 1 (...) Also got issues with left files from previously tested branches, like here: https://gitlab.com/tarantool/tarantool/-/jobs/590573606#L3718 [087] small/rlist.test [087] TAP13 parse failed (Missing plan in the TAP source). [087] [087] No result file (small/rlist.result) found. [087] Run the test with --update-result option to write the new result file. [087] [ fail ] [087] small/static.test [087] TAP13 parse failed (Missing plan in the TAP source). [087] [087] No result file (small/static.result) found. [087] Run the test with --update-result option to write the new result file. [087] [ fail ] To fix it was added the command to clean all available git submodules: git submodule foreach git clean -ffdx Closes #5036
avtikhon
added a commit
that referenced
this issue
Jun 11, 2020
Found the issue on regular testing hosts: https://gitlab.com/tarantool/tarantool/-/jobs/577884238#L7 Fetching changes... 00:04 Reinitialized existing Git repository in /home/gitlab-runner/builds/zzyC6hh5/0/tarantool/tarantool/.git/ Checking out 8ff7f32 as ... warning: failed to remove CMakeFiles/Makefile.cmake Found the job that saved the directories with root permissions https://gitlab.com/tarantool/tarantool/-/jobs/577768553 The issue appeared because the job that saved directories with root permissions used the 'shell' runner to run docker container inside. It caused the gitlab-runner to run the default workspace cleanup outside the docker container. In opposite to it, when 'docker' runner is used, the cleanup routine runs inside the docker container and no fails ever exist, because the root permissions are used in the docker container and this is the same root permissions for the host. As the result using 'shell' runner, cleanup routine failed to remove files created by root inside the docker container and which were shared to global host with the same permissions, because gitlab-runner runs the 'shell' runner by regular 'gitlab-runner' user, but not by root. To fix the issue need to run docker containers using the gitlab-runner only in RO mode with Out-Of-Source builds in it. Either use the 'docker' runners when docker containers are needed. Anyway 'shell' runner jobs with additional calls to docker containers can't be control for the branches of developers, to avoid of it need to make local cleanup routine instead of default to the working paths for each job with 'shell' runners use. Decided to setup gitlab-runner configuration as described in: https://docs.gitlab.com/ce/ci/yaml/README.html#git-clean-flags Also got the issue with left data from previous builds at the submodule pathes, like here: https://gitlab.com/tarantool/tarantool/-/jobs/574199256#L3141 Undefined symbols for architecture x86_64: "_u_isprint_66", referenced from: _yaml_emitter_is_printable in libyaml_static.a(emitter.c.o) ld: symbol(s) not found for architecture x86_64 clang: error: linker command failed with exit code 1 (...) Also got issues with left files from previously tested branches, like here: https://gitlab.com/tarantool/tarantool/-/jobs/590573606#L3718 [087] small/rlist.test [087] TAP13 parse failed (Missing plan in the TAP source). [087] [087] No result file (small/rlist.result) found. [087] Run the test with --update-result option to write the new result file. [087] [ fail ] [087] small/static.test [087] TAP13 parse failed (Missing plan in the TAP source). [087] [087] No result file (small/static.result) found. [087] Run the test with --update-result option to write the new result file. [087] [ fail ] To fix it was added the command to clean all available git submodules: git submodule foreach git clean -ffdx Closes #5036
avtikhon
added a commit
that referenced
this issue
Jun 11, 2020
Found the issue on regular testing hosts: https://gitlab.com/tarantool/tarantool/-/jobs/577884238#L7 Fetching changes... 00:04 Reinitialized existing Git repository in /home/gitlab-runner/builds/zzyC6hh5/0/tarantool/tarantool/.git/ Checking out 8ff7f32 as ... warning: failed to remove CMakeFiles/Makefile.cmake Found the job that saved the directories with root permissions https://gitlab.com/tarantool/tarantool/-/jobs/577768553 The issue appeared because the job that saved directories with root permissions used the 'shell' runner to run docker container inside. It caused the gitlab-runner to run the default workspace cleanup outside the docker container. In opposite to it, when 'docker' runner is used, the cleanup routine runs inside the docker container and no fails ever exist, because the root permissions are used in the docker container and this is the same root permissions for the host. As the result using 'shell' runner, cleanup routine failed to remove files created by root inside the docker container and which were shared to global host with the same permissions, because gitlab-runner runs the 'shell' runner by regular 'gitlab-runner' user, but not by root. To fix the issue need to run docker containers using the gitlab-runner only in RO mode with Out-Of-Source builds in it. Either use the 'docker' runners when docker containers are needed. Anyway 'shell' runner jobs with additional calls to docker containers can't be control for the branches of developers, to avoid of it need to make local cleanup routine instead of default to the working paths for each job with 'shell' runners use. Decided to setup gitlab-runner configuration as described in: https://docs.gitlab.com/ce/ci/yaml/README.html#git-clean-flags Also got the issue with left data from previous builds at the submodule pathes, like here: https://gitlab.com/tarantool/tarantool/-/jobs/574199256#L3141 Undefined symbols for architecture x86_64: "_u_isprint_66", referenced from: _yaml_emitter_is_printable in libyaml_static.a(emitter.c.o) ld: symbol(s) not found for architecture x86_64 clang: error: linker command failed with exit code 1 (...) Also got issues with left files from previously tested branches, like here: https://gitlab.com/tarantool/tarantool/-/jobs/590573606#L3718 [087] small/rlist.test [087] TAP13 parse failed (Missing plan in the TAP source). [087] [087] No result file (small/rlist.result) found. [087] Run the test with --update-result option to write the new result file. [087] [ fail ] [087] small/static.test [087] TAP13 parse failed (Missing plan in the TAP source). [087] [087] No result file (small/static.result) found. [087] Run the test with --update-result option to write the new result file. [087] [ fail ] To fix it was added the command to clean all available git submodules: git submodule foreach git clean -ffdx Closes #5036
|
avtikhon
added a commit
that referenced
this issue
Jun 11, 2020
Found the issue on regular testing hosts:
[1]
Fetching changes...
00:04
Reinitialized existing Git repository in
/home/gitlab-runner/builds/zzyC6hh5/0/tarantool/tarantool/.git/
Checking out 8ff7f32 as ...
warning: failed to remove CMakeFiles/Makefile.cmake
Found the job that saved the directories with root permissions
[2]
The issue appeared because the job that saved directories with root
permissions used the 'shell' runner to run docker container inside.
It caused the gitlab-runner to run the default workspace cleanup
outside the docker container. In opposite to it, when 'docker' runner
is used, the cleanup routine runs inside the docker container and no
fails ever exist, because the root permissions are used in the docker
container and this is the same root permissions for the host. As the
result using 'shell' runner, cleanup routine failed to remove files
created by root inside the docker container and which were shared to
global host with the same permissions, because gitlab-runner runs the
'shell' runner by regular 'gitlab-runner' user, but not by root.
To fix the issue need to run docker containers using the gitlab-runner
only in RO mode with Out-Of-Source builds in it. Either use the
'docker' runners when docker containers are needed. Anyway 'shell'
runner jobs with additional calls to docker containers can't be
control for the branches of developers, to avoid of it need to make
local cleanup routine instead of default to the working paths for each
job with 'shell' runners use.
Decided to setup gitlab-runner configuration as described in:
[3]
Also got the issue with left data from previous builds at the
submodule pathes, like here:
[4]
Undefined symbols for architecture x86_64:
"_u_isprint_66", referenced from:
_yaml_emitter_is_printable in libyaml_static.a(emitter.c.o)
ld: symbol(s) not found for architecture x86_64
clang: error: linker command failed with exit code 1 (...)
Also got issues with left files from previously tested branches,
like here:
[5]
[087] small/rlist.test
[087] TAP13 parse failed (Missing plan in the TAP source).
[087]
[087] No result file (small/rlist.result) found.
[087] Run the test with --update-result option to write the new result file.
[087] [ fail ]
[087] small/static.test
[087] TAP13 parse failed (Missing plan in the TAP source).
[087]
[087] No result file (small/static.result) found.
[087] Run the test with --update-result option to write the new result file.
[087] [ fail ]
To fix it was added the command to clean all available git submodules:
git submodule foreach git clean -ffdx
Closes #5036
[1] - https://gitlab.com/tarantool/tarantool/-/jobs/577884238#L7
[2] - https://gitlab.com/tarantool/tarantool/-/jobs/577768553
[3] - https://docs.gitlab.com/ce/ci/yaml/README.html#git-clean-flags
[4] - https://gitlab.com/tarantool/tarantool/-/jobs/574199256#L3141
[5] - https://gitlab.com/tarantool/tarantool/-/jobs/590573606#L3718
avtikhon
added a commit
that referenced
this issue
Jun 11, 2020
Found the issue on regular testing hosts:
[1]
Fetching changes...
00:04
Reinitialized existing Git repository in
/home/gitlab-runner/builds/zzyC6hh5/0/tarantool/tarantool/.git/
Checking out 8ff7f32 as ...
warning: failed to remove CMakeFiles/Makefile.cmake
Found the job that saved the directories with root permissions
[2]
The issue appeared because the job that saved directories with root
permissions used the 'shell' runner to run docker container inside.
It caused the gitlab-runner to run the default workspace cleanup
outside the docker container. In opposite to it, when 'docker' runner
is used, the cleanup routine runs inside the docker container and no
fails ever exist, because the root permissions are used in the docker
container and this is the same root permissions for the host. As the
result using 'shell' runner, cleanup routine failed to remove files
created by root inside the docker container and which were shared to
global host with the same permissions, because gitlab-runner runs the
'shell' runner by regular 'gitlab-runner' user, but not by root.
To fix the issue need to run docker containers using the gitlab-runner
only in RO mode with Out-Of-Source builds in it. Either use the
'docker' runners when docker containers are needed. Anyway 'shell'
runner jobs with additional calls to docker containers can't be
control for the branches of developers, to avoid of it need to make
local cleanup routine instead of default to the working paths for each
job with 'shell' runners use.
Decided to setup gitlab-runner configuration as described in:
[3]
Also got the issue with left data from previous builds at the
submodule pathes, like here:
[4]
Undefined symbols for architecture x86_64:
"_u_isprint_66", referenced from:
_yaml_emitter_is_printable in libyaml_static.a(emitter.c.o)
ld: symbol(s) not found for architecture x86_64
clang: error: linker command failed with exit code 1 (...)
Also got issues with left files from previously tested branches,
like here:
[5]
[087] small/rlist.test
[087] TAP13 parse failed (Missing plan in the TAP source).
[087]
[087] No result file (small/rlist.result) found.
[087] Run the test with --update-result option to write the new result file.
[087] [ fail ]
[087] small/static.test
[087] TAP13 parse failed (Missing plan in the TAP source).
[087]
[087] No result file (small/static.result) found.
[087] Run the test with --update-result option to write the new result file.
[087] [ fail ]
To fix it was added the command to clean all available git submodules:
git submodule foreach git clean -ffdx
Closes #5036
[1] - https://gitlab.com/tarantool/tarantool/-/jobs/577884238#L7
[2] - https://gitlab.com/tarantool/tarantool/-/jobs/577768553
[3] - https://docs.gitlab.com/ce/ci/yaml/README.html#git-clean-flags
[4] - https://gitlab.com/tarantool/tarantool/-/jobs/574199256#L3141
[5] - https://gitlab.com/tarantool/tarantool/-/jobs/590573606#L3718
avtikhon
added a commit
that referenced
this issue
Jun 11, 2020
Found the issue on regular testing hosts:
[1]
Fetching changes...
00:04
Reinitialized existing Git repository in
/home/gitlab-runner/builds/zzyC6hh5/0/tarantool/tarantool/.git/
Checking out 8ff7f32 as ...
warning: failed to remove CMakeFiles/Makefile.cmake
Found the job that saved the directories with root permissions
[2]
The issue appeared because the job that saved directories with root
permissions used the 'shell' runner to run docker container inside.
It caused the gitlab-runner to run the default workspace cleanup
outside the docker container. In opposite to it, when 'docker' runner
is used, the cleanup routine runs inside the docker container and no
fails ever exist, because the root permissions are used in the docker
container and this is the same root permissions for the host. As the
result using 'shell' runner, cleanup routine failed to remove files
created by root inside the docker container and which were shared to
global host with the same permissions, because gitlab-runner runs the
'shell' runner by regular 'gitlab-runner' user, but not by root.
To fix the issue need to run docker containers using the gitlab-runner
only in RO mode with Out-Of-Source builds in it. Either use the
'docker' runners when docker containers are needed. Anyway 'shell'
runner jobs with additional calls to docker containers can't be
control for the branches of developers, to avoid of it need to make
local cleanup routine instead of default to the working paths for each
job with 'shell' runners use.
Decided to setup gitlab-runner configuration as described in:
[3]
Also got the issue with left data from previous builds at the
submodule pathes, like here:
[4]
Undefined symbols for architecture x86_64:
"_u_isprint_66", referenced from:
_yaml_emitter_is_printable in libyaml_static.a(emitter.c.o)
ld: symbol(s) not found for architecture x86_64
clang: error: linker command failed with exit code 1 (...)
Also got issues with left files from previously tested branches,
like here:
[5]
[087] small/rlist.test
[087] TAP13 parse failed (Missing plan in the TAP source).
[087]
[087] No result file (small/rlist.result) found.
[087] Run the test with --update-result option to write the new result file.
[087] [ fail ]
[087] small/static.test
[087] TAP13 parse failed (Missing plan in the TAP source).
[087]
[087] No result file (small/static.result) found.
[087] Run the test with --update-result option to write the new result file.
[087] [ fail ]
To fix it was added the command to clean all available git submodules:
git submodule foreach git clean -ffdx
Closes #5036
[1] - https://gitlab.com/tarantool/tarantool/-/jobs/577884238#L7
[2] - https://gitlab.com/tarantool/tarantool/-/jobs/577768553
[3] - https://docs.gitlab.com/ce/ci/yaml/README.html#git-clean-flags
[4] - https://gitlab.com/tarantool/tarantool/-/jobs/574199256#L3141
[5] - https://gitlab.com/tarantool/tarantool/-/jobs/590573606#L3718
avtikhon
added a commit
that referenced
this issue
Jun 11, 2020
Found the issue on regular testing hosts:
[1]
Fetching changes...
00:04
Reinitialized existing Git repository in
/home/gitlab-runner/builds/zzyC6hh5/0/tarantool/tarantool/.git/
Checking out 8ff7f32 as ...
warning: failed to remove CMakeFiles/Makefile.cmake
Found the job that saved the directories with root permissions
[2]
The issue appeared because the job that saved directories with root
permissions used the 'shell' runner to run docker container inside.
It caused the gitlab-runner to run the default workspace cleanup
outside the docker container. In opposite to it, when 'docker' runner
is used, the cleanup routine runs inside the docker container and no
fails ever exist, because the root permissions are used in the docker
container and this is the same root permissions for the host. As the
result using 'shell' runner, cleanup routine failed to remove files
created by root inside the docker container and which were shared to
global host with the same permissions, because gitlab-runner runs the
'shell' runner by regular 'gitlab-runner' user, but not by root.
To fix the issue need to run docker containers using the gitlab-runner
only in RO mode with Out-Of-Source builds in it. Either use the
'docker' runners when docker containers are needed. Anyway 'shell'
runner jobs with additional calls to docker containers can't be
control for the branches of developers, to avoid of it need to make
local cleanup routine instead of default to the working paths for each
job with 'shell' runners use.
Decided to setup gitlab-runner configuration as described in:
[3]
Also got the issue with left data from previous builds at the
submodule pathes, like here:
[4]
Undefined symbols for architecture x86_64:
"_u_isprint_66", referenced from:
_yaml_emitter_is_printable in libyaml_static.a(emitter.c.o)
ld: symbol(s) not found for architecture x86_64
clang: error: linker command failed with exit code 1 (...)
Also got issues with left files from previously tested branches,
like here:
[5]
[087] small/rlist.test
[087] TAP13 parse failed (Missing plan in the TAP source).
[087]
[087] No result file (small/rlist.result) found.
[087] Run the test with --update-result option to write the new result file.
[087] [ fail ]
[087] small/static.test
[087] TAP13 parse failed (Missing plan in the TAP source).
[087]
[087] No result file (small/static.result) found.
[087] Run the test with --update-result option to write the new result file.
[087] [ fail ]
To fix it was added the command to clean all available git submodules:
git submodule foreach git clean -ffdx
Closes #5036
[1] - https://gitlab.com/tarantool/tarantool/-/jobs/577884238#L7
[2] - https://gitlab.com/tarantool/tarantool/-/jobs/577768553
[3] - https://docs.gitlab.com/ce/ci/yaml/README.html#git-clean-flags
[4] - https://gitlab.com/tarantool/tarantool/-/jobs/574199256#L3141
[5] - https://gitlab.com/tarantool/tarantool/-/jobs/590573606#L3718
avtikhon
added a commit
that referenced
this issue
Jun 11, 2020
Found the issue on regular testing hosts:
[1]
Fetching changes...
00:04
Reinitialized existing Git repository in
/home/gitlab-runner/builds/zzyC6hh5/0/tarantool/tarantool/.git/
Checking out 8ff7f32 as ...
warning: failed to remove CMakeFiles/Makefile.cmake
Found the job that saved the directories with root permissions
[2]
The issue appeared because the job that saved directories with root
permissions used the 'shell' runner to run docker container inside.
It caused the gitlab-runner to run the default workspace cleanup
outside the docker container. In opposite to it, when 'docker' runner
is used, the cleanup routine runs inside the docker container and no
fails ever exist, because the root permissions are used in the docker
container and this is the same root permissions for the host. As the
result using 'shell' runner, cleanup routine failed to remove files
created by root inside the docker container and which were shared to
global host with the same permissions, because gitlab-runner runs the
'shell' runner by regular 'gitlab-runner' user, but not by root.
To fix the issue need to run docker containers using the gitlab-runner
only in RO mode with Out-Of-Source builds in it. Either use the
'docker' runners when docker containers are needed. Anyway 'shell'
runner jobs with additional calls to docker containers can't be
control for the branches of developers, to avoid of it need to make
local cleanup routine instead of default to the working paths for each
job with 'shell' runners use.
Decided to setup gitlab-runner configuration as described in:
[3]
Also got the issue with left data from previous builds at the
submodule pathes, like here:
[4]
Undefined symbols for architecture x86_64:
"_u_isprint_66", referenced from:
_yaml_emitter_is_printable in libyaml_static.a(emitter.c.o)
ld: symbol(s) not found for architecture x86_64
clang: error: linker command failed with exit code 1 (...)
Also got issues with left files from previously tested branches,
like here:
[5]
[087] small/rlist.test
[087] TAP13 parse failed (Missing plan in the TAP source).
[087]
[087] No result file (small/rlist.result) found.
[087] Run the test with --update-result option to write the new result file.
[087] [ fail ]
[087] small/static.test
[087] TAP13 parse failed (Missing plan in the TAP source).
[087]
[087] No result file (small/static.result) found.
[087] Run the test with --update-result option to write the new result file.
[087] [ fail ]
To fix it was added the command to clean all available git submodules:
git submodule foreach git clean -ffdx
Closes #5036
[1] - https://gitlab.com/tarantool/tarantool/-/jobs/577884238#L7
[2] - https://gitlab.com/tarantool/tarantool/-/jobs/577768553
[3] - https://docs.gitlab.com/ce/ci/yaml/README.html#git-clean-flags
[4] - https://gitlab.com/tarantool/tarantool/-/jobs/574199256#L3141
[5] - https://gitlab.com/tarantool/tarantool/-/jobs/590573606#L3718
avtikhon
added a commit
that referenced
this issue
Jun 11, 2020
Found the issue on regular testing hosts: [1]
Fetching changes...
00:04
Reinitialized existing Git repository in
/home/gitlab-runner/builds/zzyC6hh5/0/tarantool/tarantool/.git/
Checking out 8ff7f32 as ...
warning: failed to remove CMakeFiles/Makefile.cmake
Found the job that saved the directories with root permissions: [2]
The issue appeared because the job that saved directories with root
permissions used the 'shell' runner to run docker container inside.
It caused the gitlab-runner to run the default workspace cleanup
outside the docker container. In opposite to it, when 'docker' runner
is used, the cleanup routine runs inside the docker container and no
fails ever exist, because the root permissions are used in the docker
container and this is the same root permissions for the host. As the
result using 'shell' runner, cleanup routine failed to remove files
created by root inside the docker container and which were shared to
global host with the same permissions, because gitlab-runner runs the
'shell' runner by regular 'gitlab-runner' user, but not by root.
To fix the issue need to run docker containers using the gitlab-runner
only in RO mode with Out-Of-Source builds in it. Either use the
'docker' runners when docker containers are needed. Anyway 'shell'
runner jobs with additional calls to docker containers can't be
control for the branches of developers, to avoid of it need to make
local cleanup routine instead of default to the working paths for each
job with 'shell' runners use.
Decided to setup gitlab-runner configuration as described in: [3]
Also got the issue with left data from previous builds at the
submodule pathes, like here: [4]
Undefined symbols for architecture x86_64:
"_u_isprint_66", referenced from:
_yaml_emitter_is_printable in libyaml_static.a(emitter.c.o)
ld: symbol(s) not found for architecture x86_64
clang: error: linker command failed with exit code 1 (...)
Also got issues with left files from previously tested branches,
like here: [5]
[087] small/rlist.test
[087] TAP13 parse failed (Missing plan in the TAP source).
[087]
[087] No result file (small/rlist.result) found.
[087] Run the test with --update-result option to write the new result file.
[087] [ fail ]
[087] small/static.test
[087] TAP13 parse failed (Missing plan in the TAP source).
[087]
[087] No result file (small/static.result) found.
[087] Run the test with --update-result option to write the new result file.
[087] [ fail ]
To fix it was added the command to clean all available git submodules:
git submodule foreach git clean -ffdx
Closes #5036
1. https://gitlab.com/tarantool/tarantool/-/jobs/577884238#L7
2. https://gitlab.com/tarantool/tarantool/-/jobs/577768553
3. https://docs.gitlab.com/ce/ci/yaml/README.html#git-clean-flags
4. https://gitlab.com/tarantool/tarantool/-/jobs/574199256#L3141
5. https://gitlab.com/tarantool/tarantool/-/jobs/590573606#L3718
avtikhon
added a commit
that referenced
this issue
Jun 12, 2020
Found the issue on regular testing hosts: [1]
Fetching changes...
00:04
Reinitialized existing Git repository in
/home/gitlab-runner/builds/zzyC6hh5/0/tarantool/tarantool/.git/
Checking out 8ff7f32 as ...
warning: failed to remove CMakeFiles/Makefile.cmake
Found the job that saved the directories with root permissions: [2]
The issue appeared because the job that saved directories with root
permissions used the 'shell' runner to run docker container inside.
It caused the gitlab-runner to run the default workspace cleanup
outside the docker container. In opposite to it, when 'docker' runner
is used, the cleanup routine runs inside the docker container and no
fails ever exist, because the root permissions are used in the docker
container and this is the same root permissions for the host. As the
result using 'shell' runner, cleanup routine failed to remove files
created by root inside the docker container and which were shared to
global host with the same permissions, because gitlab-runner runs the
'shell' runner by regular 'gitlab-runner' user, but not by root.
To fix the issue need to run docker containers using the gitlab-runner
only in RO mode with Out-Of-Source builds in it. Either use the
'docker' runners when docker containers are needed. Anyway 'shell'
runner jobs with additional calls to docker containers can't be
control for the branches of developers, to avoid of it need to make
local cleanup routine instead of default to the working paths for each
job with 'shell' runners use.
Decided to setup gitlab-runner configuration as described in: [3]
Also got the issue with left data from previous builds at the
submodule pathes, like here: [4]
Undefined symbols for architecture x86_64:
"_u_isprint_66", referenced from:
_yaml_emitter_is_printable in libyaml_static.a(emitter.c.o)
ld: symbol(s) not found for architecture x86_64
clang: error: linker command failed with exit code 1 (...)
Also got issues with left files from previously tested branches,
like here: [5]
[087] small/rlist.test
[087] TAP13 parse failed (Missing plan in the TAP source).
[087]
[087] No result file (small/rlist.result) found.
[087] Run the test with --update-result option to write the new result file.
[087] [ fail ]
[087] small/static.test
[087] TAP13 parse failed (Missing plan in the TAP source).
[087]
[087] No result file (small/static.result) found.
[087] Run the test with --update-result option to write the new result file.
[087] [ fail ]
To fix it was added the command to clean all available git submodules:
git submodule foreach git clean -ffdx
Closes #5036
1. https://gitlab.com/tarantool/tarantool/-/jobs/577884238#L7
2. https://gitlab.com/tarantool/tarantool/-/jobs/577768553
3. https://docs.gitlab.com/ce/ci/yaml/README.html#git-clean-flags
4. https://gitlab.com/tarantool/tarantool/-/jobs/574199256#L3141
5. https://gitlab.com/tarantool/tarantool/-/jobs/590573606#L3718
avtikhon
added a commit
that referenced
this issue
Jun 12, 2020
Found the issue on regular testing hosts: [1]
Fetching changes...
00:04
Reinitialized existing Git repository in
/home/gitlab-runner/builds/zzyC6hh5/0/tarantool/tarantool/.git/
Checking out 8ff7f32 as ...
warning: failed to remove CMakeFiles/Makefile.cmake
Found the job that saved the directories with root permissions: [2]
The issue appeared because the job that saved directories with root
permissions used the 'shell' runner to run docker container inside.
It caused the gitlab-runner to run the default workspace cleanup
outside the docker container. In opposite to it, when 'docker' runner
is used, the cleanup routine runs inside the docker container and no
fails ever exist, because the root permissions are used in the docker
container and this is the same root permissions for the host. As the
result using 'shell' runner, cleanup routine failed to remove files
created by root inside the docker container and which were shared to
global host with the same permissions, because gitlab-runner runs the
'shell' runner by regular 'gitlab-runner' user, but not by root.
To fix the issue need to run docker containers using the gitlab-runner
only in RO mode with Out-Of-Source builds in it. Either use the
'docker' runners when docker containers are needed. Anyway 'shell'
runner jobs with additional calls to docker containers can't be
control for the branches of developers, to avoid of it need to make
local cleanup routine instead of default to the working paths for each
job with 'shell' runners use.
Decided to setup gitlab-runner configuration as described in: [3]
Also got the issue with left data from previous builds at the
submodule pathes, like here: [4]
Undefined symbols for architecture x86_64:
"_u_isprint_66", referenced from:
_yaml_emitter_is_printable in libyaml_static.a(emitter.c.o)
ld: symbol(s) not found for architecture x86_64
clang: error: linker command failed with exit code 1 (...)
Also got issues with left files from previously tested branches,
like here: [5]
[087] small/rlist.test
[087] TAP13 parse failed (Missing plan in the TAP source).
[087]
[087] No result file (small/rlist.result) found.
[087] Run the test with --update-result option to write the new result file.
[087] [ fail ]
[087] small/static.test
[087] TAP13 parse failed (Missing plan in the TAP source).
[087]
[087] No result file (small/static.result) found.
[087] Run the test with --update-result option to write the new result file.
[087] [ fail ]
To fix it was added the command to clean all available git submodules:
git submodule foreach git clean -ffdx
Closes #5036
1. https://gitlab.com/tarantool/tarantool/-/jobs/577884238#L7
2. https://gitlab.com/tarantool/tarantool/-/jobs/577768553
3. https://docs.gitlab.com/ce/ci/yaml/README.html#git-clean-flags
4. https://gitlab.com/tarantool/tarantool/-/jobs/574199256#L3141
5. https://gitlab.com/tarantool/tarantool/-/jobs/590573606#L3718
kyukhin
pushed a commit
that referenced
this issue
Jun 15, 2020
Found the issue on regular testing hosts: [1]
Fetching changes...
00:04
Reinitialized existing Git repository in
/home/gitlab-runner/builds/zzyC6hh5/0/tarantool/tarantool/.git/
Checking out 8ff7f32 as ...
warning: failed to remove CMakeFiles/Makefile.cmake
Found the job that saved the directories with root permissions: [2]
The issue appeared because the job that saved directories with root
permissions used the 'shell' runner to run docker container inside.
It caused the gitlab-runner to run the default workspace cleanup
outside the docker container. In opposite to it, when 'docker' runner
is used, the cleanup routine runs inside the docker container and no
fails ever exist, because the root permissions are used in the docker
container and this is the same root permissions for the host. As the
result using 'shell' runner, cleanup routine failed to remove files
created by root inside the docker container and which were shared to
global host with the same permissions, because gitlab-runner runs the
'shell' runner by regular 'gitlab-runner' user, but not by root.
To fix the issue need to run docker containers using the gitlab-runner
only in RO mode with Out-Of-Source builds in it. Either use the
'docker' runners when docker containers are needed. Anyway 'shell'
runner jobs with additional calls to docker containers can't be
control for the branches of developers, to avoid of it need to make
local cleanup routine instead of default to the working paths for each
job with 'shell' runners use.
Decided to setup gitlab-runner configuration as described in: [3]
Also got the issue with left data from previous builds at the
submodule pathes, like here: [4]
Undefined symbols for architecture x86_64:
"_u_isprint_66", referenced from:
_yaml_emitter_is_printable in libyaml_static.a(emitter.c.o)
ld: symbol(s) not found for architecture x86_64
clang: error: linker command failed with exit code 1 (...)
Also got issues with left files from previously tested branches,
like here: [5]
[087] small/rlist.test
[087] TAP13 parse failed (Missing plan in the TAP source).
[087]
[087] No result file (small/rlist.result) found.
[087] Run the test with --update-result option to write the new result file.
[087] [ fail ]
[087] small/static.test
[087] TAP13 parse failed (Missing plan in the TAP source).
[087]
[087] No result file (small/static.result) found.
[087] Run the test with --update-result option to write the new result file.
[087] [ fail ]
To fix it was added the command to clean all available git submodules:
git submodule foreach git clean -ffdx
Closes #5036
1. https://gitlab.com/tarantool/tarantool/-/jobs/577884238#L7
2. https://gitlab.com/tarantool/tarantool/-/jobs/577768553
3. https://docs.gitlab.com/ce/ci/yaml/README.html#git-clean-flags
4. https://gitlab.com/tarantool/tarantool/-/jobs/574199256#L3141
5. https://gitlab.com/tarantool/tarantool/-/jobs/590573606#L3718
(cherry picked from commit 9640778)
avtikhon
added a commit
that referenced
this issue
Jul 15, 2020
Set cleanup based on docker for all of the jobs to avoid of fails when the job before change could use docker and reassigned the temporary files in the working directory. Follows up #5036
avtikhon
added a commit
that referenced
this issue
Jul 15, 2020
Set cleanup based on docker for all of the jobs to avoid of fails when the job before change could use docker and reassigned the temporary files in the working directory. If the docker is not available than no need to use it for cleanup and cleanup runs using shell. Follows up #5036
avtikhon
added a commit
that referenced
this issue
Jul 15, 2020
Set cleanup based on docker for all of the jobs to avoid of fails when the job before change could use docker and reassigned the temporary files in the working directory. If the docker is not available than no need to use it for cleanup and cleanup runs using shell. Follows up #5036
avtikhon
added a commit
that referenced
this issue
Jul 15, 2020
Set cleanup based on docker for all of the jobs to avoid of fails when the job before change could use docker and reassigned the temporary files in the working directory. If the docker is not available than no need to use it for cleanup and cleanup runs using shell. Follows up #5036
avtikhon
added a commit
that referenced
this issue
Jul 22, 2020
Set cleanup based on docker for all of the jobs to avoid of fails when the job before change could use docker and reassigned the temporary files in the working directory. If the docker is not available than no need to use it for cleanup and cleanup runs using shell. Follows up #5036
avtikhon
added a commit
that referenced
this issue
Jul 22, 2020
Set cleanup based on docker for all of the jobs to avoid of fails when the job before change could use docker and reassigned the temporary files in the working directory. If the docker is not available than no need to use it for cleanup and cleanup runs using shell. Follows up #5036
avtikhon
added a commit
that referenced
this issue
Jul 22, 2020
Set cleanup based on docker for all of the jobs to avoid of fails when the job before change could use docker and reassigned the temporary files in the working directory. If the docker is not available than no need to use it for cleanup and cleanup runs using shell. Gitlab-ci clean flags disabled and reorganized localy as shown at [1]. Disabled gitlab-ci checkout default process as shown at [2] to be able to fix repository before the checkout. Found that previously run gitlab-ci jobs could change permissions files at repository which broke the checkouts at the next jobs. Used for checkout strategy [3], for submodule update strategy [4]. List of steps made localy instead of gitlab-ci preparations: 1. Check/clone the Tarantool repository with submodules. 2. For shell based jobs change ownership of all the sources to 'gitlab-runner' user. (NOTE: in Docker based jobs the 'gitlab-runner' user is not known.) 3. Fetch Tarantool sources with branches and force checkout of the testing commit. 4. Update submodules recursively (use force where supports). 5. Cleanup all the sources from all files except from repository. [1] https://docs.gitlab.com/ee/ci/yaml/README.html#git-clean-flags [2] https://docs.gitlab.com/ee/ci/yaml/README.html#git-strategy [3] https://docs.gitlab.com/ee/ci/yaml/README.html#git-checkout [4] https://docs.gitlab.com/ee/ci/yaml/README.html#git-submodule-strategy Follows up #5036
avtikhon
added a commit
that referenced
this issue
Jul 22, 2020
Set cleanup based on docker for all of the jobs to avoid of fails when the job before change could use docker and reassigned the temporary files in the working directory. If the docker is not available than no need to use it for cleanup and cleanup runs using shell. Gitlab-ci clean flags disabled and reorganized localy as shown at [1]. Disabled gitlab-ci checkout default process as shown at [2] to be able to fix repository before the checkout. Found that previously run gitlab-ci jobs could change permissions files at repository which broke the checkouts at the next jobs. Used for checkout strategy [3], for submodule update strategy [4]. List of steps made localy instead of gitlab-ci preparations: 1. Check/clone the Tarantool repository with submodules. 2. For shell based jobs change ownership of all the sources to 'gitlab-runner' user. (NOTE: in Docker based jobs the 'gitlab-runner' user is not known.) 3. Fetch Tarantool sources with branches and force checkout of the testing commit. 4. Update submodules recursively (use force where supports). 5. Cleanup all the sources from all files except from repository. [1] https://docs.gitlab.com/ee/ci/yaml/README.html#git-clean-flags [2] https://docs.gitlab.com/ee/ci/yaml/README.html#git-strategy [3] https://docs.gitlab.com/ee/ci/yaml/README.html#git-checkout [4] https://docs.gitlab.com/ee/ci/yaml/README.html#git-submodule-strategy Follows up #5036
avtikhon
added a commit
that referenced
this issue
Jul 22, 2020
Set cleanup based on docker for all of the jobs to avoid of fails when the job before change could use docker and reassigned the temporary files in the working directory. If the docker is not available than no need to use it for cleanup and cleanup runs using shell. Gitlab-ci clean flags disabled and reorganized localy as shown at [1]. Disabled gitlab-ci checkout default process as shown at [2] to be able to fix repository before the checkout. Found that previously run gitlab-ci jobs could change permissions files at repository which broke the checkouts at the next jobs. Used for checkout strategy [3], for submodule update strategy [4]. List of steps made localy instead of gitlab-ci preparations: 1. Check/clone the Tarantool repository with submodules. 2. For shell based jobs change ownership of all the sources to 'gitlab-runner' user. (NOTE: in Docker based jobs the 'gitlab-runner' user is not known.) 3. Fetch Tarantool sources with branches and force checkout of the testing commit. 4. Update submodules recursively (use force where supports). 5. Cleanup all the sources from all files except from repository. [1] https://docs.gitlab.com/ee/ci/yaml/README.html#git-clean-flags [2] https://docs.gitlab.com/ee/ci/yaml/README.html#git-strategy [3] https://docs.gitlab.com/ee/ci/yaml/README.html#git-checkout [4] https://docs.gitlab.com/ee/ci/yaml/README.html#git-submodule-strategy Follows up #5036
avtikhon
added a commit
that referenced
this issue
Jul 22, 2020
Set cleanup based on docker for all of the jobs to avoid of fails when the job before change could use docker and reassigned the temporary files in the working directory. If the docker is not available than no need to use it for cleanup and cleanup runs using shell. Gitlab-ci clean flags disabled and reorganized localy as shown at [1]. Disabled gitlab-ci checkout default process as shown at [2] to be able to fix repository before the checkout. Found that previously run gitlab-ci jobs could change permissions files at repository which broke the checkouts at the next jobs. Used for checkout strategy [3], for submodule update strategy [4]. List of steps made localy instead of gitlab-ci preparations: 1. Check/clone the Tarantool repository with submodules. 2. For shell based jobs change ownership of all the sources to 'gitlab-runner' user. (NOTE: in Docker based jobs the 'gitlab-runner' user is not known.) 3. Fetch Tarantool sources with branches and force checkout of the testing commit. 4. Update submodules recursively (use force where supports). 5. Cleanup all the sources from all files except from repository. [1] https://docs.gitlab.com/ee/ci/yaml/README.html#git-clean-flags [2] https://docs.gitlab.com/ee/ci/yaml/README.html#git-strategy [3] https://docs.gitlab.com/ee/ci/yaml/README.html#git-checkout [4] https://docs.gitlab.com/ee/ci/yaml/README.html#git-submodule-strategy Follows up #5036 Check common before_script
avtikhon
added a commit
that referenced
this issue
Jul 22, 2020
Set cleanup based on docker for all of the jobs to avoid of fails when the job before change could use docker and reassigned the temporary files in the working directory. If the docker is not available than no need to use it for cleanup and cleanup runs using shell. Gitlab-ci clean flags disabled and reorganized localy as shown at [1]. Disabled gitlab-ci checkout default process as shown at [2] to be able to fix repository before the checkout. Found that previously run gitlab-ci jobs could change permissions files at repository which broke the checkouts at the next jobs. Used for checkout strategy [3], for submodule update strategy [4]. Submodules local update routine in .gitlab.mk file became unneeded and removed. List of steps made localy instead of gitlab-ci preparations: 1. Check/clone the Tarantool repository with submodules. 2. For shell based jobs change ownership of all the sources to 'gitlab-runner' user. (NOTE: in Docker based jobs the 'gitlab-runner' user is not known.) 3. Fetch Tarantool sources with branches and force checkout of the testing commit. 4. Update submodules recursively (use force where supports). 5. Cleanup all the sources from all files except from repository. [1] https://docs.gitlab.com/ee/ci/yaml/README.html#git-clean-flags [2] https://docs.gitlab.com/ee/ci/yaml/README.html#git-strategy [3] https://docs.gitlab.com/ee/ci/yaml/README.html#git-checkout [4] https://docs.gitlab.com/ee/ci/yaml/README.html#git-submodule-strategy Follows up #5036 Check common before_script
avtikhon
added a commit
that referenced
this issue
Jul 22, 2020
Set cleanup based on docker for all of the jobs to avoid of fails when the job before change could use docker and reassigned the temporary files in the working directory. If the docker is not available than no need to use it for cleanup and cleanup runs using shell. Gitlab-ci clean flags disabled and reorganized localy as shown at [1]. Disabled gitlab-ci checkout default process as shown at [2] to be able to fix repository before the checkout. Found that previously run gitlab-ci jobs could change permissions files at repository which broke the checkouts at the next jobs. Used for checkout strategy [3], for submodule update strategy [4]. Submodules local update routine in .gitlab.mk file became unneeded and removed. List of steps made localy instead of gitlab-ci preparations: 1. Check/clone the Tarantool repository with submodules. 2. For shell based jobs change ownership of all the sources to 'gitlab-runner' user. (NOTE: in Docker based jobs the 'gitlab-runner' user is not known.) 3. Fetch Tarantool sources with branches and force checkout of the testing commit. 4. Update submodules recursively (use force where supports). 5. Cleanup all the sources from all files except from repository. [1] https://docs.gitlab.com/ee/ci/yaml/README.html#git-clean-flags [2] https://docs.gitlab.com/ee/ci/yaml/README.html#git-strategy [3] https://docs.gitlab.com/ee/ci/yaml/README.html#git-checkout [4] https://docs.gitlab.com/ee/ci/yaml/README.html#git-submodule-strategy Follows up #5036 Check common before_script
avtikhon
added a commit
that referenced
this issue
Jul 22, 2020
Set cleanup based on docker for all of the jobs to avoid of fails when the job before change could use docker and reassigned the temporary files in the working directory. If the docker is not available than no need to use it for cleanup and cleanup runs using shell. Gitlab-ci clean flags disabled and reorganized localy as shown at [1]. Disabled gitlab-ci checkout default process as shown at [2] to be able to fix repository before the checkout. Found that previously run gitlab-ci jobs could change permissions files at repository which broke the checkouts at the next jobs. Used for checkout strategy [3], for submodule update strategy [4]. Submodules local update routine in .gitlab.mk file became unneeded and removed. List of steps made localy instead of gitlab-ci preparations: 1. Check/clone the Tarantool repository with submodules. 2. For shell based jobs change ownership of all the sources to 'gitlab-runner' user. (NOTE: in Docker based jobs the 'gitlab-runner' user is not known.) 3. Fetch Tarantool sources with branches and force checkout of the testing commit. 4. Update submodules recursively (use force where supports). 5. Cleanup all the sources from all files except from repository. [1] https://docs.gitlab.com/ee/ci/yaml/README.html#git-clean-flags [2] https://docs.gitlab.com/ee/ci/yaml/README.html#git-strategy [3] https://docs.gitlab.com/ee/ci/yaml/README.html#git-checkout [4] https://docs.gitlab.com/ee/ci/yaml/README.html#git-submodule-strategy Follows up #5036
kyukhin
pushed a commit
that referenced
this issue
Jul 22, 2020
Set cleanup based on docker for all of the jobs to avoid of fails when the job before change could use docker and reassigned the temporary files in the working directory. If the docker is not available than no need to use it for cleanup and cleanup runs using shell. Gitlab-ci clean flags disabled and reorganized localy as shown at [1]. Disabled gitlab-ci checkout default process as shown at [2] to be able to fix repository before the checkout. Found that previously run gitlab-ci jobs could change permissions files at repository which broke the checkouts at the next jobs. Used for checkout strategy [3], for submodule update strategy [4]. Submodules local update routine in .gitlab.mk file became unneeded and removed. List of steps made localy instead of gitlab-ci preparations: 1. Check/clone the Tarantool repository with submodules. 2. For shell based jobs change ownership of all the sources to 'gitlab-runner' user. (NOTE: in Docker based jobs the 'gitlab-runner' user is not known.) 3. Fetch Tarantool sources with branches and force checkout of the testing commit. 4. Update submodules recursively (use force where supports). 5. Cleanup all the sources from all files except from repository. [1] https://docs.gitlab.com/ee/ci/yaml/README.html#git-clean-flags [2] https://docs.gitlab.com/ee/ci/yaml/README.html#git-strategy [3] https://docs.gitlab.com/ee/ci/yaml/README.html#git-checkout [4] https://docs.gitlab.com/ee/ci/yaml/README.html#git-submodule-strategy Follows up #5036
kyukhin
pushed a commit
that referenced
this issue
Jul 22, 2020
Set cleanup based on docker for all of the jobs to avoid of fails when the job before change could use docker and reassigned the temporary files in the working directory. If the docker is not available than no need to use it for cleanup and cleanup runs using shell. Gitlab-ci clean flags disabled and reorganized localy as shown at [1]. Disabled gitlab-ci checkout default process as shown at [2] to be able to fix repository before the checkout. Found that previously run gitlab-ci jobs could change permissions files at repository which broke the checkouts at the next jobs. Used for checkout strategy [3], for submodule update strategy [4]. Submodules local update routine in .gitlab.mk file became unneeded and removed. List of steps made localy instead of gitlab-ci preparations: 1. Check/clone the Tarantool repository with submodules. 2. For shell based jobs change ownership of all the sources to 'gitlab-runner' user. (NOTE: in Docker based jobs the 'gitlab-runner' user is not known.) 3. Fetch Tarantool sources with branches and force checkout of the testing commit. 4. Update submodules recursively (use force where supports). 5. Cleanup all the sources from all files except from repository. [1] https://docs.gitlab.com/ee/ci/yaml/README.html#git-clean-flags [2] https://docs.gitlab.com/ee/ci/yaml/README.html#git-strategy [3] https://docs.gitlab.com/ee/ci/yaml/README.html#git-checkout [4] https://docs.gitlab.com/ee/ci/yaml/README.html#git-submodule-strategy Follows up #5036 (cherry picked from commit fca966a)
kyukhin
pushed a commit
that referenced
this issue
Jul 22, 2020
Set cleanup based on docker for all of the jobs to avoid of fails when the job before change could use docker and reassigned the temporary files in the working directory. If the docker is not available than no need to use it for cleanup and cleanup runs using shell. Gitlab-ci clean flags disabled and reorganized localy as shown at [1]. Disabled gitlab-ci checkout default process as shown at [2] to be able to fix repository before the checkout. Found that previously run gitlab-ci jobs could change permissions files at repository which broke the checkouts at the next jobs. Used for checkout strategy [3], for submodule update strategy [4]. Submodules local update routine in .gitlab.mk file became unneeded and removed. List of steps made localy instead of gitlab-ci preparations: 1. Check/clone the Tarantool repository with submodules. 2. For shell based jobs change ownership of all the sources to 'gitlab-runner' user. (NOTE: in Docker based jobs the 'gitlab-runner' user is not known.) 3. Fetch Tarantool sources with branches and force checkout of the testing commit. 4. Update submodules recursively (use force where supports). 5. Cleanup all the sources from all files except from repository. [1] https://docs.gitlab.com/ee/ci/yaml/README.html#git-clean-flags [2] https://docs.gitlab.com/ee/ci/yaml/README.html#git-strategy [3] https://docs.gitlab.com/ee/ci/yaml/README.html#git-checkout [4] https://docs.gitlab.com/ee/ci/yaml/README.html#git-submodule-strategy Follows up #5036 (cherry picked from commit fca966a)
kyukhin
pushed a commit
that referenced
this issue
Jul 22, 2020
Set cleanup based on docker for all of the jobs to avoid of fails when the job before change could use docker and reassigned the temporary files in the working directory. If the docker is not available than no need to use it for cleanup and cleanup runs using shell. Gitlab-ci clean flags disabled and reorganized localy as shown at [1]. Disabled gitlab-ci checkout default process as shown at [2] to be able to fix repository before the checkout. Found that previously run gitlab-ci jobs could change permissions files at repository which broke the checkouts at the next jobs. Used for checkout strategy [3], for submodule update strategy [4]. Submodules local update routine in .gitlab.mk file became unneeded and removed. List of steps made localy instead of gitlab-ci preparations: 1. Check/clone the Tarantool repository with submodules. 2. For shell based jobs change ownership of all the sources to 'gitlab-runner' user. (NOTE: in Docker based jobs the 'gitlab-runner' user is not known.) 3. Fetch Tarantool sources with branches and force checkout of the testing commit. 4. Update submodules recursively (use force where supports). 5. Cleanup all the sources from all files except from repository. [1] https://docs.gitlab.com/ee/ci/yaml/README.html#git-clean-flags [2] https://docs.gitlab.com/ee/ci/yaml/README.html#git-strategy [3] https://docs.gitlab.com/ee/ci/yaml/README.html#git-checkout [4] https://docs.gitlab.com/ee/ci/yaml/README.html#git-submodule-strategy Follows up #5036 (cherry picked from commit fca966a)
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Tarantool version:
OS version:
Bug description:
Found that after some patches runs in gitlab-ci, like:
https://gitlab.com/tarantool/tarantool/-/commit/95d27d7ab71d90a03f3e66962c52c948f85e42ae
the old workspaces were not completely fixed, which produced the following issues in build on OSX hosts:
Found that removing the build directories in sources didn't help to resolve the issue, but complete removal of the gitlab-ci workspaces with sources on the testing hosts helped. Due to gitlab-ci cleanups the sources itself with 'git clean -xfd' command, and seems that some git submodules in the sources left bad files which caused the issue, the only solution to fix it to use cleanup for each git submodule, like:
Steps to reproduce:
Optional (but very desirable):
The text was updated successfully, but these errors were encountered: