Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jenkins: Build Fails Often #2967

Open
sanssecours opened this issue Sep 16, 2019 · 14 comments

Comments

@sanssecours
Copy link
Member

commented Sep 16, 2019

Description

Currently the Jenkins build fails quite often for various reasons. This issue should list some of the problems that currently include:

.

Failures

Branch Failure Reason Failed Build Job/Stage
PR #2932 Maven build debian-unstable-clang-asan
master Homepage build Deploy Website
master Homepage build Deploy Website
PR #2945 Internal compiler error build-elektra-web-base
master Cmake install failure debian-stretch-full
master Workspace removal failure Main builds
master Workspace removal failure Main builds
master Workspace removal failure Main builds
master Workspace removal failure Main builds
master Workspace removal failure Main builds
master Workspace removal failure Main builds
PR #2945 Haskell build failure debian-stretch-full-optimizations-off
PR #2945 APT install failed build-elektra-website
PR #2932 Maven build debian-unstable-clang-asan
master Timeout debian-stretch-full-mmap-asan
PR #2975 Timeout debian-buster-mingw-w64
master Homepage build Deploy Website
master Homepage build Deploy Website
master Timeout debian-buster-full
master Haskell build failure debian-stretch-full-ini
master Timeout debian-unstable-full
master Failing tests debian-buster-full
master Internal compiler error build-elektra-web-base
master Homepage build Deploy Website
master Homepage build Deploy Website
master Homepage build Deploy Website
master Homepage build Deploy Website
PR #2998 Timeout, Connection problems build-elektra-web-base, debian-buster-full-i386
master Maven build debian-unstable-clang-asan
PR #2998 Timeout build-elektra-website-backend
master Connection problems build-elektra-web-base
master Homepage build Deploy Website
master Maven build debian-unstable-full-clang
master Git commit failure buildPackage/debian/buster
master Git commit failure buildPackage/debian/buster
master Git commit failure buildPackage/debian/buster, buildPackage/debian/stretch
master Git commit failure buildPackage/debian/buster
master Git commit failure buildPackage/debian/buster

Failing Tests

Test Location Times Failed
check_external_example_codegen_econf debian-buster-full 1
check_external_example_codegen_menu debian-buster-full 1
check_external_example_codegen_tree debian-buster-full 1
check_external_example_highlevel debian-buster-full 1
check_spec debian-buster-full 1
testkdb_ensure debian-buster-full 1
@markus2330

This comment has been minimized.

Copy link
Contributor

commented Sep 16, 2019

Thank you for collecting the issues!

For the maven builds we already have an issue: #2855

@sanssecours

This comment has been minimized.

Copy link
Member Author

commented Sep 16, 2019

For the maven builds we already have an issue: #2855

I know 😊. I already added a link in the issue description.

@markus2330

This comment has been minimized.

Copy link
Contributor

commented Sep 23, 2019

Thank you for this elaborate research. We now need to fix one issue after the other.

@markus2330

This comment has been minimized.

Copy link
Contributor

commented Sep 23, 2019

For the Haskell problems we can remove the haskell bindings/plugins. They are not maintained anyway.

@sanssecours sanssecours added this to the 1.0.0 milestone Sep 26, 2019
@markus2330 markus2330 removed this from the 1.0.0 milestone Sep 26, 2019
@markus2330

This comment has been minimized.

Copy link
Contributor

commented Sep 28, 2019

Haskell will be removed in #3017

@markus2330 markus2330 referenced this issue Oct 6, 2019
0 of 14 tasks complete
@markus2330

This comment has been minimized.

Copy link
Contributor

commented Oct 6, 2019

The failures with docker pull failing in the website stage occurs quite often now.

@markus2330 markus2330 referenced this issue Oct 6, 2019
18 of 35 tasks complete
@Chemin1

This comment has been minimized.

Copy link
Contributor

commented Oct 8, 2019

I just got connection problems for build-elektra-web-base, too.

3d070e3209ce: Retrying in 1 second

error creating overlay mount to /home/_docker/overlay2/e9563564b9365114c47d90b7e8d307565225097a525e6b1b866a2da2877b2aa8/merged: device or resource busy

script returned exit code 1

This is a full log.

@markus2330 markus2330 referenced this issue Oct 13, 2019
0 of 14 tasks complete
@Chemin1

This comment has been minimized.

Copy link
Contributor

commented Oct 16, 2019

The failures with docker pull failing in the website stage occurs quite often now.

Is this all the retrying and waiting after Pulling from build-elektra-web-base (log)?

Additionally, I think this error is new: test_service_convertengine fails during Starting build/hub.libelektra.org/build-elektra-website-backend (log 2)

@markus2330

This comment has been minimized.

Copy link
Contributor

commented Oct 17, 2019

Yes, I agree test_service_convertengine is not reported here yet. Actually we can disable the test as the service is not modified anyway.

@sanssecours is there some procedure how to add new tests in the above list?

@sanssecours

This comment has been minimized.

Copy link
Member Author

commented Oct 17, 2019

@sanssecours is there some procedure how to add new tests in the above list?

Nope. I already gave up on modifying the list, since the Jenkins build fails too often. I would recommend we just open an issue for each specific problem.

@markus2330

This comment has been minimized.

Copy link
Contributor

commented Oct 17, 2019

For issues related to source code I agree. For the issues related to docker/jenkins instability it is enough to collect issues here as it is very limited what we can do next to the migration we already do but unfortunately takes longer as expected. It would be nice if @Mistreated could give more information about the status, maybe in #160.

@markus2330

This comment has been minimized.

Copy link
Contributor

commented Oct 17, 2019

Additionally, I think this error is new: test_service_convertengine fails during Starting build/hub.libelektra.org/build-elektra-website-backend (log 2)

Can you please report that separately? The fix is to disable the tests.

@Chemin1

This comment has been minimized.

Copy link
Contributor

commented Oct 17, 2019

Can you please report that separately?

Done, see #3086

@markus2330

This comment has been minimized.

Copy link
Contributor

commented Oct 18, 2019

I think our best guess to make our lives much easier is to "fix" these problems using https://wiki.jenkins.io/display/JENKINS/Naginator+Plugin

Then Jenkins will restart failed jobs several times. I think we could try 5 restarts before giving up?

@Mistreated Can you implement this also on the old server? Or is this too risky?

Before we implement this, however, we need the new Jenkins Node as otherwise the queue will get too long.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
4 participants
You can’t perform that action at this time.