New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

apt-get upgrade in plain Ubuntu precise image fails #1724

Closed
colszowka opened this Issue Aug 29, 2013 · 28 comments

Comments

Projects
None yet
@colszowka

colszowka commented Aug 29, 2013

I'm having trouble with running apt-get upgrade on a custom-built precise base box. The box is built using debootstrap as in the contrib scripts, like so:

debootstrap --verbose --variant=minbase --include='iproute,iputils-ping,lsb-release,curl,nano' --arch=amd64 precise

I already fixed another issue on the upgrade thanks to the workaround mentioned in #1024, but this here still shows up:

Reading package lists... Done
Building dependency tree       
Reading state information... Done
0 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.
1 not fully installed or removed.
After this operation, 0 B of additional disk space will be used.
Setting up initscripts (2.88dsf-13.10ubuntu11.1) ...
mount: block device /dev/shm is write-protected, mounting read-only
mount: cannot mount block device /dev/shm read-only
dpkg: error processing initscripts (--configure):
 subprocess installed post-installation script returned error exit status 1
Errors were encountered while processing:
 initscripts
E: Sub-process /usr/bin/dpkg returned an error code (1)

I dug up some lxc mailing list discussion on this from a year ago which you can find here, but the proposed workaround (rmdir /dev/shm && ln -s /run/shm /dev/shm) does not cut it for me as the dir removal fails with failed to remove '/dev/shm': Device or resource busy.

Any ideas how I could work around this?

@colszowka

This comment has been minimized.

colszowka commented Aug 29, 2013

Well, I did come up with a solution - I just told apt not to upgrade the initscripts package at any times via apt-mark hold initscripts - still wondering if there is some better solution for this

@snarlysodboxer

This comment has been minimized.

snarlysodboxer commented Oct 13, 2013

I am having this exact same problem just starting up ubuntu:quantal from the Docker Index and then doing
apt-get update && apt-get upgrade && apt-get dist-upgrade.
(I'm wanting to build a 13.04 image.)

@alambike

This comment has been minimized.

Contributor

alambike commented Oct 14, 2013

I have similar issue trying to update my ubuntu12.04 image, solved with similar solution like @colszowka did and with the workaround proposed in #1024 plus this tips

apt-mark hold initscripts udev plymouth mountall
dpkg-divert --local --rename --add /sbin/initctl
ln -s /bin/true /sbin/initctl

cat  > /usr/sbin/policy-rc.d << EOF
#!/bin/sh

exit 101
EOF

chmod +x /usr/sbin/policy-rc.d

@dodobas

This comment has been minimized.

dodobas commented Oct 15, 2013

It seems that the problem with initscripts update is because the postinstall script will fail when trying to detect the process running context (metal/virtualization/paravritualization) using ischroot utility.

So the way to solve the problems with initscripts update is to:

  • execute apt-get upgrade, which will fail
  • manually edit file /var/lib/dpkg/info/initscripts.postinst and on the line 248 replace ischroot with true, basically force the initscripts postinst to think is running in chroot environment
  • execute dpkg --configure -a

And it works, until the next initscripts update :)

@kiorky

This comment has been minimized.

Contributor

kiorky commented Oct 29, 2013

Or you can make an image/containers that starts correctly upstart, see #2276

@dstrctrng

This comment has been minimized.

dstrctrng commented Nov 23, 2013

Another workaround is to symlink ischroot to true. Just make sure to redo the link if debianutils is upgraded. Maybe put that package on hold.

@crosbymichael

This comment has been minimized.

Member

crosbymichael commented Dec 5, 2013

ping @tianon

@snarlysodboxer You cannot run dist-upgrade in a container. It is not the proper way to upgrade the distro.

@tianon

This comment has been minimized.

Member

tianon commented Dec 5, 2013

Indeed, your best bet is to generate an updated base image and rebuild the dependant images on top of the new base.

If there's a particular package you need updated, just apt-get install that one package and it'll update.

@snarlysodboxer

This comment has been minimized.

snarlysodboxer commented Jan 3, 2014

@crosbymichael Thank you. I was able to use debootstrap to build my own raring install as described here: http://docs.docker.io/en/latest/use/baseimages/

@tobstarr

This comment has been minimized.

Contributor

tobstarr commented Jan 29, 2014

I just ran into a similar issue. That was right after booting a new EC2 instance on which I installed linux-image-extra-$(uname -r) to get AUFs working. After rebooting the EC2 instance the upgrade went through. Could that be some kind of kernel issue?

@tobstarr

This comment has been minimized.

Contributor

tobstarr commented Jan 29, 2014

Well, and here I am again back at square one.

@tianon

This comment has been minimized.

Member

tianon commented Feb 6, 2014

@tobstarr that's a separate issue entirely - this issue is about running apt-get upgrade or apt-get dist-upgrade inside a container

@garthk

This comment has been minimized.

garthk commented Feb 7, 2014

Indeed, your best bet is to generate an updated base image

Didn't the publishers of the ubuntu and stackbrew/ubuntu images take on that responsibility, @tianon? If not, we need to add warnings to the documentation.

The business driver behind apt-get upgrade is to ensure my Docker images match the compulsively-patched VMs on which my software will be deployed. It makes Docker seem more time-costly and less reliable if that means I need to to manually update every package except a few, or add lines to my Dockerfile to chain apt-get down, or build my own base images.

@garthk

This comment has been minimized.

garthk commented Feb 7, 2014

Atoned for my snarking in IRC by testing @alambike's workarounds (thankyou!) and documenting them in Dockerfile syntax for the ease of copy and paste:

# hopefully temporary work-around of http://git.io/Ke_Meg#1724 
RUN apt-mark hold initscripts udev plymouth mountall

EDIT: as @tianon pointed out in IRC, these lines aren't necessary in the latest base images:

RUN dpkg-divert --local --rename --add /sbin/initctl
RUN ln -sf /bin/true /sbin/initctl
RUN ln -sf /bin/false /usr/sbin/policy-rc.d
@tianon

This comment has been minimized.

Member

tianon commented Feb 7, 2014

Ok, so this clearly deserves more explanation, since it's obviously something very strongly (and in several cases emotionally) desired.

I've got a much more legitimate and simple workaround than holding packages, but I want you guys to promise not to use it in general Dockerfiles that you might subject other people to, especially because most of those other people won't want to wait anywhere from 15-60 minutes while the entire base image upgrades (which is one huge reason why we try not to update the base images willy-nilly). I will concede that there are places where it makes sense to do this, which is why I'm sharing it, but it is still "wrong" in the general case so it won't be added to the base images (especially since our base images are used for so many different things).

So, without further ado, the magic one-liner is:

RUN dpkg-divert --local --rename /usr/bin/ischroot && ln -sf /bin/true /usr/bin/ischroot

After this one line, apt-get dist-upgrade should complete just fine, regardless of which packages update, because "dpkg-divert" literally exists just to overwrite system files in a way that won't be destroyed by upgrades. I've been personally testing this against "ubuntu:12.04", but if there are images or packages this doesn't "fix" it for, please let me know and I will be happy to look into it.

@tianon

This comment has been minimized.

Member

tianon commented Feb 7, 2014

As another note, in my testing with "ubuntu:12.04", this creates a new ~100MB layer on top of the already ~200MB base image. For those of you who need this, this probably isn't a problem, but it's worth considering. Cheers.

mccahill added a commit to mccahill/docker-swift that referenced this issue Feb 15, 2014

new workaround for apt-get upgrade problem in current Ubuntu 12.04 ba…
…se image

the previous workaround doesn't work because of changes in the base image. I used a newer workaround described here:

moby/moby#1724
Signed-off-by: Mark McCahill <mark.mccahill@duke.edu>
@stucki

This comment has been minimized.

stucki commented Feb 25, 2014

@tianon Thanks for the workaround, it works perfect. I checked the manpage of "ischroot", and to me it seems very logical that this workaround will solve most problems for Docker images. So I'm wondering, why are you saying that it's a wrong fix? Please enlighten me! 😄

On another note, I am wondering if there is any way for me to find out the version of a base image which someone is using? It seems like there have been several changes made to the ubuntu:12.04 base image, so lots of workarounds are no longer needed. However I don't know how to detect if the image is up to date. Any hints for me?

@tianon

This comment has been minimized.

Member

tianon commented Feb 25, 2014

The proper fix would be made to ischroot itself to properly detect being in a container, IMO.

The easiest way to make sure your "ubuntu:12.04" is up to date is to run docker pull ubuntu:12.04.

@stucki

This comment has been minimized.

stucki commented Feb 26, 2014

I understand. So this means that the fix works just fine for any Docker image and can easily be used as a permanent workaround as long as ischroot isn't fixed, right?
I just don't understand why you mention that this should not be used in generic Dockerfiles if it works so well. Maybe I just didn't get the joke of it?

Regarding the base image: I know that I can run docker pull ubuntu:12.04 for my own images, however if I am providing a Dockerfile to other people, how can I know what version they are using?

stucki added a commit to stucki/docker-lineageos that referenced this issue Mar 2, 2014

@tianon

This comment has been minimized.

Member

tianon commented Mar 2, 2014

It wasn't a joke, it was an honest plea. I quite like running other people's Dockerfiles, and I don't want to wait for 15 - 20 minutes (on the optimistic end) to then waste 200mb before I can actually test or play with your Dockerfile. Also, I'd really rather not have to modify each Dockerfile before I run it just to make sure it's not doing a full upgrade. That makes Dockerfiles less portable in practice, IMO, because nobody will want to wait while they run.

Anyhow, these are just my personal thoughts.

To discuss keeping images themselves up to date, that'd be issue #1860 and #4238.

@stucki

This comment has been minimized.

stucki commented Mar 3, 2014

Ok I understand that you don't want to update images during build, fine. However, it should still be possible to update a container if someone wants to do so during a run (not build). Therefore, I think your fix is very welcome and needed by default.

I agree that upgrading is probably not everyone's wish, but on the other hand the updates will usually contain important security fixes, so you should have an interest in applying them...

Meanwhile I subscribed to #1860 and #4238. Let's agree that this needs some more thinking...

neofreko added a commit to neofreko/docker-lampstack that referenced this issue Mar 8, 2014

assumes ubuntu is the latest
thus requires no manual upgrade. otherwise we will face initscript issue moby/moby#1724

hamiltont added a commit to hamiltont/docker-lampstack that referenced this issue Apr 18, 2014

Remove apt-get upgrade
Currently upgrade breaks the container build. Upgrading is 
unrecommended inside a container, as there is no init system and 
some packages have to be held back. 

To see the official issue: moby/moby#1724
To see workarounds: https://github.com/racker/docker-ubuntu-with-updates/blob/master/precise/Dockerfile

hamiltont added a commit to hamiltont/ZoneMinder that referenced this issue Apr 18, 2014

Remove apt-get upgrade
Convention is to avoid this if not needed, as there is no init 
system and some packages have to be held back. upgrade will 
sometimes break due to the state of ubuntu:latest

To see the official issue: moby/moby#1724
To see workarounds: https://github.com/racker/docker-ubuntu-with-updates/blob/master/precise/Dockerfile
@vincentwoo

This comment has been minimized.

Contributor

vincentwoo commented Apr 22, 2014

@tianon is there any hope of not needing this sort of thing in the long haul? I'm loathe to add more bloat but it looks like I don't have much of a choice if I want to be able to upgrade in a build.

@thaJeztah

This comment has been minimized.

Member

thaJeztah commented Apr 22, 2014

@vincentwoo if I understand correctly, the idea is that the base images (e.g. ubuntu:12.04, ubuntu:13.10) will be kept up-to-date by the maintainer of those images. Run docker pull ubuntu to make sure you have the latest version. If your Dockerfile uses those images as 'FROM', there should be no reason to perform an upgrade.

@vincentwoo

This comment has been minimized.

Contributor

vincentwoo commented Apr 22, 2014

@thaJeztah understood. From here it looks like ubuntu:12.04 was last updated a couple weeks ago, so you might be right. Do you know where I can go to find clarity about why these base images were updated and what the differences between versions are?

@jamtur01

This comment has been minimized.

Contributor

jamtur01 commented Apr 22, 2014

The last update to the base images was to address the Heartbleed vulnerability from memory - https://groups.google.com/forum/#!searchin/docker-user/heartbleed/docker-user/F7P_MOKtKAk/WrQy1r29lvoJ - we do need to make this process more transparent though.

@thaJeztah

This comment has been minimized.

Member

thaJeztah commented Apr 22, 2014

@jamtur01 should there be an (automated) re-build of base images so that users can always know that the base images are up-to-date and there's no reason to perform 'apt-get upgrade'?

@stucki

This comment has been minimized.

stucki commented Apr 22, 2014

@jamtur01 @thaJeztah More important, there needs to be some way to find out how old an image is, or what features it lacks. I imagine a version number for images, or something similar...

@unclejack

This comment has been minimized.

Contributor

unclejack commented Dec 15, 2014

The ubuntu:14.04 and ubuntu:12.04 official images have been updated to make it possible to run apt-get dist-upgrade in a container. This should also work properly when building images with the mkimage scripts.

I'm going to close this issue since it's been solved.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment