apt-get upgrade in plain Ubuntu precise image fails #1724

Closed
colszowka opened this Issue Aug 29, 2013 · 28 comments

Projects

None yet
@colszowka

I'm having trouble with running apt-get upgrade on a custom-built precise base box. The box is built using debootstrap as in the contrib scripts, like so:

debootstrap --verbose --variant=minbase --include='iproute,iputils-ping,lsb-release,curl,nano' --arch=amd64 precise

I already fixed another issue on the upgrade thanks to the workaround mentioned in #1024, but this here still shows up:

Reading package lists... Done
Building dependency tree       
Reading state information... Done
0 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.
1 not fully installed or removed.
After this operation, 0 B of additional disk space will be used.
Setting up initscripts (2.88dsf-13.10ubuntu11.1) ...
mount: block device /dev/shm is write-protected, mounting read-only
mount: cannot mount block device /dev/shm read-only
dpkg: error processing initscripts (--configure):
 subprocess installed post-installation script returned error exit status 1
Errors were encountered while processing:
 initscripts
E: Sub-process /usr/bin/dpkg returned an error code (1)

I dug up some lxc mailing list discussion on this from a year ago which you can find here, but the proposed workaround (rmdir /dev/shm && ln -s /run/shm /dev/shm) does not cut it for me as the dir removal fails with failed to remove '/dev/shm': Device or resource busy.

Any ideas how I could work around this?

@colszowka

Well, I did come up with a solution - I just told apt not to upgrade the initscripts package at any times via apt-mark hold initscripts - still wondering if there is some better solution for this

@snarlysodboxer

I am having this exact same problem just starting up ubuntu:quantal from the Docker Index and then doing
apt-get update && apt-get upgrade && apt-get dist-upgrade.
(I'm wanting to build a 13.04 image.)

@alambike
Contributor

I have similar issue trying to update my ubuntu12.04 image, solved with similar solution like @colszowka did and with the workaround proposed in #1024 plus this tips

apt-mark hold initscripts udev plymouth mountall
dpkg-divert --local --rename --add /sbin/initctl
ln -s /bin/true /sbin/initctl

cat  > /usr/sbin/policy-rc.d << EOF
#!/bin/sh

exit 101
EOF

chmod +x /usr/sbin/policy-rc.d

@dodobas
dodobas commented Oct 15, 2013

It seems that the problem with initscripts update is because the postinstall script will fail when trying to detect the process running context (metal/virtualization/paravritualization) using ischroot utility.

So the way to solve the problems with initscripts update is to:

  • execute apt-get upgrade, which will fail
  • manually edit file /var/lib/dpkg/info/initscripts.postinst and on the line 248 replace ischroot with true, basically force the initscripts postinst to think is running in chroot environment
  • execute dpkg --configure -a

And it works, until the next initscripts update :)

@kiorky
Contributor
kiorky commented Oct 29, 2013

Or you can make an image/containers that starts correctly upstart, see #2276

@dstrctrng

Another workaround is to symlink ischroot to true. Just make sure to redo the link if debianutils is upgraded. Maybe put that package on hold.

@crosbymichael
Member

ping @tianon

@snarlysodboxer You cannot run dist-upgrade in a container. It is not the proper way to upgrade the distro.

@tianon
Member
tianon commented Dec 5, 2013

Indeed, your best bet is to generate an updated base image and rebuild the dependant images on top of the new base.

If there's a particular package you need updated, just apt-get install that one package and it'll update.

@snarlysodboxer

@crosbymichael Thank you. I was able to use debootstrap to build my own raring install as described here: http://docs.docker.io/en/latest/use/baseimages/

@tobstarr
Contributor

I just ran into a similar issue. That was right after booting a new EC2 instance on which I installed linux-image-extra-$(uname -r) to get AUFs working. After rebooting the EC2 instance the upgrade went through. Could that be some kind of kernel issue?

@tobstarr
Contributor

Well, and here I am again back at square one.

@tianon
Member
tianon commented Feb 6, 2014

@tobstarr that's a separate issue entirely - this issue is about running apt-get upgrade or apt-get dist-upgrade inside a container

@garthk
garthk commented Feb 7, 2014

Indeed, your best bet is to generate an updated base image

Didn't the publishers of the ubuntu and stackbrew/ubuntu images take on that responsibility, @tianon? If not, we need to add warnings to the documentation.

The business driver behind apt-get upgrade is to ensure my Docker images match the compulsively-patched VMs on which my software will be deployed. It makes Docker seem more time-costly and less reliable if that means I need to to manually update every package except a few, or add lines to my Dockerfile to chain apt-get down, or build my own base images.

@garthk
garthk commented Feb 7, 2014

Atoned for my snarking in IRC by testing @alambike's workarounds (thankyou!) and documenting them in Dockerfile syntax for the ease of copy and paste:

# hopefully temporary work-around of http://git.io/Ke_Meg#1724 
RUN apt-mark hold initscripts udev plymouth mountall

EDIT: as @tianon pointed out in IRC, these lines aren't necessary in the latest base images:

RUN dpkg-divert --local --rename --add /sbin/initctl
RUN ln -sf /bin/true /sbin/initctl
RUN ln -sf /bin/false /usr/sbin/policy-rc.d
@tianon
Member
tianon commented Feb 7, 2014

Ok, so this clearly deserves more explanation, since it's obviously something very strongly (and in several cases emotionally) desired.

I've got a much more legitimate and simple workaround than holding packages, but I want you guys to promise not to use it in general Dockerfiles that you might subject other people to, especially because most of those other people won't want to wait anywhere from 15-60 minutes while the entire base image upgrades (which is one huge reason why we try not to update the base images willy-nilly). I will concede that there are places where it makes sense to do this, which is why I'm sharing it, but it is still "wrong" in the general case so it won't be added to the base images (especially since our base images are used for so many different things).

So, without further ado, the magic one-liner is:

RUN dpkg-divert --local --rename /usr/bin/ischroot && ln -sf /bin/true /usr/bin/ischroot

After this one line, apt-get dist-upgrade should complete just fine, regardless of which packages update, because "dpkg-divert" literally exists just to overwrite system files in a way that won't be destroyed by upgrades. I've been personally testing this against "ubuntu:12.04", but if there are images or packages this doesn't "fix" it for, please let me know and I will be happy to look into it.

@tianon
Member
tianon commented Feb 7, 2014

As another note, in my testing with "ubuntu:12.04", this creates a new ~100MB layer on top of the already ~200MB base image. For those of you who need this, this probably isn't a problem, but it's worth considering. Cheers.

@mccahill mccahill added a commit to mccahill/docker-swift that referenced this issue Feb 15, 2014
@mccahill mccahill new workaround for apt-get upgrade problem in current Ubuntu 12.04 ba…
…se image

the previous workaround doesn't work because of changes in the base image. I used a newer workaround described here:

docker/docker#1724
Signed-off-by: Mark McCahill <mark.mccahill@duke.edu>
2ca16d1
@stucki
stucki commented Feb 25, 2014

@tianon Thanks for the workaround, it works perfect. I checked the manpage of "ischroot", and to me it seems very logical that this workaround will solve most problems for Docker images. So I'm wondering, why are you saying that it's a wrong fix? Please enlighten me! 😄

On another note, I am wondering if there is any way for me to find out the version of a base image which someone is using? It seems like there have been several changes made to the ubuntu:12.04 base image, so lots of workarounds are no longer needed. However I don't know how to detect if the image is up to date. Any hints for me?

@tianon
Member
tianon commented Feb 25, 2014

The proper fix would be made to ischroot itself to properly detect being in a container, IMO.

The easiest way to make sure your "ubuntu:12.04" is up to date is to run docker pull ubuntu:12.04.

@stucki
stucki commented Feb 26, 2014

I understand. So this means that the fix works just fine for any Docker image and can easily be used as a permanent workaround as long as ischroot isn't fixed, right?
I just don't understand why you mention that this should not be used in generic Dockerfiles if it works so well. Maybe I just didn't get the joke of it?

Regarding the base image: I know that I can run docker pull ubuntu:12.04 for my own images, however if I am providing a Dockerfile to other people, how can I know what version they are using?

@gwelr gwelr referenced this issue in hectcastro/docker-riak Feb 26, 2014
Closed

apt-get upgrade returns an error #4

@stucki stucki added a commit to stucki/docker-lineageos that referenced this issue Mar 2, 2014
@stucki stucki Fix ischroot detection (see docker/docker#1724) 7900497
@tianon
Member
tianon commented Mar 2, 2014

It wasn't a joke, it was an honest plea. I quite like running other people's Dockerfiles, and I don't want to wait for 15 - 20 minutes (on the optimistic end) to then waste 200mb before I can actually test or play with your Dockerfile. Also, I'd really rather not have to modify each Dockerfile before I run it just to make sure it's not doing a full upgrade. That makes Dockerfiles less portable in practice, IMO, because nobody will want to wait while they run.

Anyhow, these are just my personal thoughts.

To discuss keeping images themselves up to date, that'd be issue #1860 and #4238.

@stucki
stucki commented Mar 3, 2014

Ok I understand that you don't want to update images during build, fine. However, it should still be possible to update a container if someone wants to do so during a run (not build). Therefore, I think your fix is very welcome and needed by default.

I agree that upgrading is probably not everyone's wish, but on the other hand the updates will usually contain important security fixes, so you should have an interest in applying them...

Meanwhile I subscribed to #1860 and #4238. Let's agree that this needs some more thinking...

@neofreko neofreko added a commit to neofreko/docker-lampstack that referenced this issue Mar 8, 2014
@neofreko neofreko assumes ubuntu is the latest
thus requires no manual upgrade. otherwise we will face initscript issue docker/docker#1724
95bdb9a
@neofreko neofreko referenced this issue in jbfink/docker-lampstack Mar 8, 2014
Open

failed build: RUN apt-get -y upgrade #2

@hamiltont hamiltont added a commit to hamiltont/docker-lampstack that referenced this issue Apr 18, 2014
@hamiltont hamiltont Remove apt-get upgrade
Currently upgrade breaks the container build. Upgrading is 
unrecommended inside a container, as there is no init system and 
some packages have to be held back. 

To see the official issue: docker/docker#1724
To see workarounds: https://github.com/racker/docker-ubuntu-with-updates/blob/master/precise/Dockerfile
28b0801
@hamiltont hamiltont referenced this issue in jbfink/docker-lampstack Apr 18, 2014
Open

Remove apt-get upgrade #3

@hamiltont hamiltont added a commit to hamiltont/ZoneMinder that referenced this issue Apr 18, 2014
@hamiltont hamiltont Remove apt-get upgrade
Convention is to avoid this if not needed, as there is no init 
system and some packages have to be held back. upgrade will 
sometimes break due to the state of ubuntu:latest

To see the official issue: docker/docker#1724
To see workarounds: https://github.com/racker/docker-ubuntu-with-updates/blob/master/precise/Dockerfile
7b62780
@hamiltont hamiltont referenced this issue in ZoneMinder/ZoneMinder Apr 18, 2014
Merged

Remove apt-get upgrade #390

@vincentwoo
Contributor

@tianon is there any hope of not needing this sort of thing in the long haul? I'm loathe to add more bloat but it looks like I don't have much of a choice if I want to be able to upgrade in a build.

@thaJeztah
Member

@vincentwoo if I understand correctly, the idea is that the base images (e.g. ubuntu:12.04, ubuntu:13.10) will be kept up-to-date by the maintainer of those images. Run docker pull ubuntu to make sure you have the latest version. If your Dockerfile uses those images as 'FROM', there should be no reason to perform an upgrade.

@vincentwoo
Contributor

@thaJeztah understood. From here it looks like ubuntu:12.04 was last updated a couple weeks ago, so you might be right. Do you know where I can go to find clarity about why these base images were updated and what the differences between versions are?

@jamtur01
Contributor

The last update to the base images was to address the Heartbleed vulnerability from memory - https://groups.google.com/forum/#!searchin/docker-user/heartbleed/docker-user/F7P_MOKtKAk/WrQy1r29lvoJ - we do need to make this process more transparent though.

@thaJeztah
Member

@jamtur01 should there be an (automated) re-build of base images so that users can always know that the base images are up-to-date and there's no reason to perform 'apt-get upgrade'?

@stucki
stucki commented Apr 22, 2014

@jamtur01 @thaJeztah More important, there needs to be some way to find out how old an image is, or what features it lacks. I imagine a version number for images, or something similar...

@pshouse pshouse added a commit to pshouse/docker-ubuntu-vnc-desktop that referenced this issue Oct 13, 2014
@pshouse pshouse Update Dockerfile 265073d
@unclejack
Contributor

The ubuntu:14.04 and ubuntu:12.04 official images have been updated to make it possible to run apt-get dist-upgrade in a container. This should also work properly when building images with the mkimage scripts.

I'm going to close this issue since it's been solved.

@unclejack unclejack closed this Dec 15, 2014
@issue-reporter issue-reporter referenced this issue in Shippable/support Feb 9, 2015
Closed

apt-get dist-upgrade fails #1043

@davidsansome davidsansome added a commit to clementine-player/Buildbot that referenced this issue Dec 28, 2015
@davidsansome davidsansome Build 32-bit precise base images properly (docker/docker#1724) d13e1d3
@ninjabiscuit ninjabiscuit added a commit to madebymany/base-builds that referenced this issue Oct 8, 2016
@ninjabiscuit ninjabiscuit don't run in a container docker/docker#1724 712933e
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment