Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jenkins: docker pull fails with unexpected EOF #3317

Open
mpranj opened this issue Dec 3, 2019 · 12 comments
Open

Jenkins: docker pull fails with unexpected EOF #3317

mpranj opened this issue Dec 3, 2019 · 12 comments

Comments

@mpranj
Copy link
Member

@mpranj mpranj commented Dec 3, 2019

To continue the discussion from #160 separately:

I've built it manually again

@Mistreated how did you do it, only on one node or did you push the image to the hub? Do you think the issue is isolated to jenkinsNode3 and there is no problem on the hub?

@mpranj

This comment has been minimized.

Copy link
Member Author

@mpranj mpranj commented Dec 3, 2019

Ok, thank you for fixing it! Then it does not make sense for me to clear the docker registry.

I'll try rebuilding and see if everything works now.

@Mistreated

This comment has been minimized.

Copy link
Contributor

@Mistreated Mistreated commented Dec 3, 2019

I've seen this error before when I first tried to add a new agent. At the time it was the hetzner-jenkins1 agent. The mistake just went away after few retries, I had no idea why.

I cloned libelektra and executed the following on the jenkinsNode3:
docker build libelektra/scripts/docker/debian/stretch/.

Usually, agents are reusing the local image, once it is pulled, but I have no idea why would pulling from hub throw an occasional error.

@mpranj

This comment has been minimized.

Copy link
Member Author

@mpranj mpranj commented Dec 3, 2019

The images are rebuilt every month to my knowledge (such that software inside the images is kept somewhat up-to-date). This happened also for December now, so it's possible that the newest image was not already cached locally.

@markus2330

This comment has been minimized.

Copy link
Contributor

@markus2330 markus2330 commented Dec 3, 2019

I hope this (monthly rebuilding of all images) still happens, as @Mistreated did not add the libelektra-monthly build job.

@markus2330 markus2330 mentioned this issue Dec 3, 2019
18 of 35 tasks complete
@markus2330

This comment has been minimized.

Copy link
Contributor

@markus2330 markus2330 commented Dec 3, 2019

@ingwinlu do you still know what the monthly build job did and why it did not have a Jenkinsfile?

@Mistreated

This comment has been minimized.

Copy link
Contributor

@Mistreated Mistreated commented Dec 3, 2019

I recreated the old Jenkins server. I can now recreate the monthly job.

@Mistreated

This comment has been minimized.

Copy link
Contributor

@Mistreated Mistreated commented Dec 3, 2019

I can now recreate the monthly job.

Done.
I do not know it's exact steps, but I will look into them in the evening.

@markus2330

This comment has been minimized.

Copy link
Contributor

@markus2330 markus2330 commented Dec 3, 2019

I recreated the old Jenkins server. I can now recreate the monthly job.

Good job, it might be helpful to keep the old Jenkins server for a while. Simply stop the container after use and do not start it on bootup.

@mpranj

This comment has been minimized.

Copy link
Member Author

@mpranj mpranj commented Dec 3, 2019

Now also push fails with device or resource busy (I have seen this before): https://build.libelektra.org/blue/organizations/jenkins/libelektra/detail/PR-3319/2/pipeline

On jenkinsNode3VM:

docker push hub.libelektra.org/build-elektra-website-backend:PR-3319_2
[...]
860ee8d82838: Retrying in 1 second
6bbb813c7d87: Retrying in 1 second
error creating overlay mount to /var/lib/docker/overlay2/8535169ca4de05e069978de34233d82158d1831fb4d980772411f59de2d370a5/merged: device or resource busy
script returned exit code 1
@markus2330

This comment has been minimized.

Copy link
Contributor

@markus2330 markus2330 commented Dec 3, 2019

Maybe increasing the timeout in the nginx of a7 helps: moby/moby#22188 (comment)

@mpranj

This comment has been minimized.

Copy link
Member Author

@mpranj mpranj commented Dec 3, 2019

I increased the timeout, let's see.

@mpranj

This comment has been minimized.

Copy link
Member Author

@mpranj mpranj commented Dec 3, 2019

Increasing the timeout did not help. I noticed that (only) jenkinsNode3VM was running debian packaged docker 18.x so I upgraded to the upstream docker packaged 19.x version (which was used by all other nodes already).

jenkinsNode3VM was now able to push and pull images without a problem. I hope it remains like that.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
3 participants
You can’t perform that action at this time.