Running update-grub-legacy-ec2 doesn't update kernel list in /boot/grub/menu.lst

[ubuntu-server-builder]

This bug was originally filed in Launchpad as LP: #1309079

Launchpad details

affected_projects = ['grub-legacy-ec2 (Ubuntu)']
assignee = None
assignee_name = None
date_closed = None
date_created = 2014-04-17T15:50:51.802617+00:00
date_fix_committed = None
date_fix_released = None
id = 1309079
importance = undecided
is_complete = False
lp_url = https://bugs.launchpad.net/cloud-init/+bug/1309079
milestone = None
owner = peter.waller
owner_name = Peter Waller
private = False
status = confirmed
submitter = peter.waller
submitter_name = Peter Waller
tags = []
duplicates = []

Launchpad user Peter Waller(peter.waller) wrote on 2014-04-17T15:50:51.802617+00:00

This bug has been observed with at least EC2 AMI: ami-ec50a19b "ubuntu/images/ebs/ubuntu-saucy-13.10-amd64-server-20140212", and other Ubuntu saucy AMIs.

Steps to reproduce:

1. sudo apt-get update
2. sudo apt-get dist-upgrade
3. Get prompted for one thing, whether to install the maintainer's version of /boot/grub/menu.lst (choose to use the maintainer's file)
4. observe updated file
5. modify the file to remove the new entries
6. run update-grub-legacy-ec2
7. observe file not updated

This is especially insidious because it claims the file is updated, with output like this:

Found kernel: /boot/vmlinuz-3.11.0-19-generic
Found kernel: /boot/vmlinuz-3.11.0-18-generic
Found kernel: /boot/vmlinuz-3.11.0-17-generic
Found kernel: /boot/vmlinuz-3.11.0-15-generic
Found kernel: /boot/vmlinuz-3.11.0-14-generic
Found kernel: /boot/vmlinuz-3.11.0-12-generic
Found kernel: /boot/memtest86+.bin
Updating /boot/grub/menu.lst ... done

And in fact, /boot/grub/menu.lst has its modified time updated to now, but it doesn't actually change the contents of the file.

This is extremely confusing.

I note that UCF_FORCE_CONFFNEW=1 and DEBIAN_PRIORITY=low and:

debconf-get-selections | grep grub-l

grub-legacy-ec2 grub/update_grub_changeprompt_threeway select install_new

don't have any useful effect.

I've emailed Ben Howard who confirmed he could reproduce this bug.

[ubuntu-server-builder]

Launchpad user Ben Howard(darkmuggle-deactivatedaccount) wrote on 2014-04-28T16:49:16.163566+00:00

I've confirmed that this happens on non-EC2 systems. I was able to replicate this on Canonistack (OpenStack) but not on EC2.

[ubuntu-server-builder]

Launchpad user Ben Howard(darkmuggle-deactivatedaccount) wrote on 2014-05-21T13:49:27.065495+00:00

Peter,

Can you tell me more about how you are hitting this in EC2? Are you booting an HVM instance-store AMI and then running ec2-bundle-vol? The only way that I can see this happening is if you boot with BIOS/GPT (i.e. HVM instance-store) and then try to create a volume and expect it to boot using PVGrub (which uses legacy Grub menu.lst).

[ubuntu-server-builder]

Launchpad user Peter Waller(peter.waller) wrote on 2014-05-21T18:37:02.318407+00:00

It's not a HVM instance. The AMI is supplied at the top of the bug report.

Note that this bug has nothing to do with which kernel is running on the machine after reboot, except that the file /boot/grub/menu.lst has the wrong contents.

The buggy behaviour is that /boot/grub/menu.lst IS NOT BEING UPDATED even though the message on the console actually says "Updating /boot/grub/menu.lst ... done".

To reproduce the behaviour all you have to do is modify /boot/grub/menu.lst once ever. Then it will never be updated again, even though update-grub-legacy-ec2 claims it is updating the file.

Unless I'm missing something, it appears that the debconf setting is being ignored.

The net effect is that we don't get kernel updates without manually intervening on /boot/grub/menu.lst.

[ubuntu-server-builder]

Launchpad user Peter Waller(peter.waller) wrote on 2014-05-21T18:44:48.898724+00:00

Please subtract "Note that this bug has nothing to do with which kernel is running on the machine after reboot, except that" from my last comment. I'm somewhat tired and I'm not sure what I meant by it.

Further digesting what you're saying, /boot/grub/menu.lst is not used for EBS AMIs? This doesn't seem consistent with what I've observed, which is that machines only boot to the new kernel if I modify this file manually.

It is possible that some other interaction is fooling me into thinking this, though, and it has been hard to pin down consistent behaviour.

It seems in any case that the buggy (or at least confusing) behaviour of update-grub-legacy-ec2 is problematic since it has wasted many man hours as we try and figure out why it refuses to update on some machines but not others.

[ubuntu-server-builder]

Launchpad user Peter Waller(peter.waller) wrote on 2014-06-26T17:24:31.824479+00:00

I've just discovered the doing a sudo rm /boot/grub/menu.lst and sudo update-grub-legacy-ec2 fixes the problem, at least temporarily.

[ubuntu-server-builder]

Launchpad user glance(glance-acc) wrote on 2014-06-27T08:23:19.217472+00:00

This have started to affect me to.

Older machine based on ubuntu-trusty-14.04-amd64-server-20140416.1 isn't affected by this problem but newer machines as of ubuntu-trusty-14.04-amd64-server-20140607.1 are affected.

[ubuntu-server-builder]

Launchpad user Stratos Zolotas(baskin) wrote on 2014-07-24T11:04:29.349641+00:00

I'm experiencing the same issue on an EC2 (AWS) Ubuntu-14.04 instance.

Although the following kernels are installed:

Found linux image: /boot/vmlinuz-3.13.0-32-generic
Found initrd image: /boot/initrd.img-3.13.0-32-generic
Found linux image: /boot/vmlinuz-3.13.0-30-generic
Found initrd image: /boot/initrd.img-3.13.0-30-generic
Found linux image: /boot/vmlinuz-3.13.0-29-generic
Found initrd image: /boot/initrd.img-3.13.0-29-generic

the system boots with 3.13.0-29 and the menu.lst is not updated.

[ubuntu-server-builder]

Launchpad user Peter Waller(peter.waller) wrote on 2014-07-24T11:27:05.648312+00:00

For what it's worth, HVM machines appear to be unaffected.

It seems that the fix of deleting the menu.lst only works once must be deleted to update it subsequently.

[ubuntu-server-builder]

Launchpad user Anders Hall(a.hall) wrote on 2014-07-25T11:50:21.627014+00:00

We have a similar bug in https://bugs.launchpad.net/ubuntu/+source/apt/+bug/1323772 you might want to join.

[ubuntu-server-builder]

Launchpad user Cam Cope(ccope) wrote on 2015-10-14T02:35:29.437567+00:00

This affects me on Ubuntu 12.04 with grub-legacy-ec2 0.7.5-0ubuntu1.12

[ubuntu-server-builder]

Launchpad user Launchpad Janitor(janitor) wrote on 2019-09-24T17:42:54.930018+00:00

Status changed to 'Confirmed' because the bug affects multiple users.

[ubuntu-server-builder]

Launchpad user Nathan Stratton Treadway(nathanst) wrote on 2019-09-24T18:29:06.903770+00:00

I ran into a problem with grub-legacy-ec2 v1:1 in Bionic (on an EC2 instance built from a Canonical Bionic AMI)... but the problem I found relates to the switch from /var/run to /run, so it seems like it could be related the earlier reports as well.

Specifically, what I noticed was the "The first time ucf sees the file" check around line 1349 of update-grub-legacy-ec2, which is:

=====

The first time ucf sees the file, we can only assume any difference

between the magic comments and the kernel options is a result of local

mods, so this will result in a ucf prompt for anyone whose first

invocation of update-grub is as a result of updating the magic comments.

if ! ucfq grub | grep -q $ucf_menu_file; then
otherbuffer=$(tempfile)
cat $buffer > $otherbuffer

    sortedKernels=`sed -n -e "
    /$endopt/,/$end/ {
            s/^kernel[[:space:]]\+\([^[:space:]]\+\).*/\1/p
    }" < $menu | grep -vE "memtest86|$grub2name|xen" | uniq`
    xenKernels=`sed -n -e "
    /$endopt/,/$end/ {
            s/^module[[:space:]]\+\([^[:space:]]*vmlinuz[^[:space:]]\+\).*/\

1/p
}" < $menu | uniq`

=====

When this is called, $ucf_menu_file contains "/var/run/grub/menu.lst"... but because of the /var/run -> /run switch, UCF doesn't actually have registered a file with that path:

=====

ucfq grub

Configuration file Package Exists Changed
/run/grub/menu.lst grub Yes Yes

if ! ucfq grub | grep -q /var/run/grub/menu.lst; then echo "yes"; fi

yes

So the net effect is that the script assumes that the file has not been registered with UCF before and rebuilds it from the kernel list taken out of the existing /boot/grub/menu.lst... but since that list is stale, there is a ucf conflict and the new file is not installed -- and then later when the script tries to to update the file with the currently-installed kernel list, there is still a ucf conflict and the updated menu.lst file is not installed.

I was able to get past this by running the kernel-image install manually (using aptitude) and thus getting the ucf "A new version of /boot/grub/menu.lst is available, but the version installed currently has been locally modified" dialog box. When I chose the "install the package maintainer's version" version, the script was then able to proceed to install that (outdated) menu.lst file, and then went on to detect the current kernel list and update menu.lst reflect them...

(I have not yet tried a second kernel update to see if the problem reoccurs.)

[TheRealFalcon]

No longer related to cloud-init