Ubuntu image kernel updates aren't found by dist-upgrade

[edmorley]

Hi!

STR:

1. mkdir bento-test && cd bento-test
2. vagrant init bento/ubuntu-16.04
3. vagrant up
4. vagrant ssh
5. In the guest: uname -r
6. In the guest: sudo apt-get update
7. In the guest: sudo apt-get dist-upgrade

(Using: Windows 10 host, Vagrant 2.0.1, Virtualbox 5.2.2, bento/ubuntu-16.04 v201801.02.0)

Expected:
That the dist-upgrade command offer to upgrade the existing 4.4.0-87-generic kernel to the newly released 4.4.0-109-generic - ie the list of packages to be updated should include linux-image-generic and linux-image-4.4.0-109-generic.

Actual:
No kernel package updates are offered:

$ sudo apt-get dist-upgrade
...
The following NEW packages will be installed:
  libdrm-common
The following packages will be upgraded:
  apport libdrm2 linux-libc-dev python3-apport python3-problem-report unattended-upgrades
6 upgraded, 1 newly installed, 0 to remove and 0 not upgraded.
Need to get 1,107 kB of archives.
After this operation, 42.0 kB of additional disk space will be used.

Initially I thought that perhaps it's just too soon after the packages were released, however they are available:

$ sudo apt-cache policy linux-image-generic
linux-image-generic:
  Installed: (none)
  Candidate: 4.4.0.109.114
  Version table:
     4.4.0.109.114 500
        500 http://archive.ubuntu.com/ubuntu xenial-updates/main amd64 Packages
        500 http://security.ubuntu.com/ubuntu xenial-security/main amd64 Packages
...

Manually running sudo apt-get install linux-image-generic --no-install-recommends and then rebooting successfully installs the latest kernel.

The problem appears to be that the Ubuntu build cleanup step also removes linux-image-generic when removing/purging linux-firmware here:

bento/ubuntu/scripts/cleanup.sh

Lines 45 to 46 in 4e5a5a5

	# Delete the massive firmware packages
	apt-get -y purge linux-firmware

Without this virtual package, dist-upgrade has no way of knowing that the installed linux-image-4.4.0-87-generic package is now superseded by another package with a different name (linux-image-4.4.0-109-generic) - so doesn't think there are any kernel updates available.

To prove it's that cleanup step causing the issue, manual install linux-image-generic (which sadly has a dep on linux-firmware, so re-installs that too), and then repeat what the cleanup script does, like so:

$ sudo apt-get purge linux-firmware
...
The following packages will be REMOVED:
  linux-firmware* linux-image-generic*
0 upgraded, 0 newly installed, 2 to remove and 6 not upgraded.
After this operation, 203 MB disk space will be freed.

It may be that this is unavoidable (if wanting to reduce image size by removing the firmware), but if it's seen as an intentional/acceptable trade-off (given new Bento box versions will come with the newer kernel releases anyway) it would be good to document in the README.

Many thanks :-)

[edmorley]

I don't think it's going to be possible to prevent the installation of linux-firmware, since even if the necessary arguments are passed to APT in the cleanup script to ignore the missing linux-image-generic dependency on linux-firmware, future APT invocations by the end user will still report errors.

The best approach I can think of is using the dpkg path-exclude option to prevent the bloat from the files under /lib/firmware/ and /usr/share/doc/linux-firmware/, similar to:
https://raphaelhertzog.com/2010/11/15/save-disk-space-by-excluding-useless-files-with-dpkg/

(path-exclude seems preferable to a one-off rm -rf /lib/firmware /usr/share/doc/linux-firmware in cleanup.sh, since it will also apply if the user's later dist-upgrade pulls down a newer linux-firmware package)

[cheeseplus]

Great repro and definitely something for us to think about as I don't think the consequences of that line were ever taken into account especially with longer lived boxes. I expect that the dpkg exclude is probably the best call but we'll give this a look for the next release.

[cheeseplus]

Finally getting around to working on this today and should have this tested/PRed by the end of the week.

[cheeseplus]

This is looking promising, running a few more tests but this looks good so far #987

[edmorley]

Many thanks!