Failure having to do with grub and upgrading

[Michagogo]

Some recent changes have to do with grub and/or upgrading the guest system before gbuilding. Not sure what's going on exactly, but I'm having a problem where the Upgrading system, may take a while section took a while, and then failed with this message: ./bin/gbuild:21:in 'system!': failed to run on-target -u root bash < target-bin/upgrade-system.sh > var/install.log 2>&1 (RuntimeError).

Editing upgrade-system.sh to remove the redirections to files and /dev/null reveals that the purge of grub works, and the dist-upgrade seems to proceed normally, until this happens: https://www.irccloud.com/pastebin/xLjCr9YR

The weird thing is that I used gbuild last night and it worked without a problem, so I don't know what changed between last night and now.

Right now I've removed the base container and am rerunning make-base-vm, hoping that then there won't be anything to upgrade and I'll skip over this issue, but that shouldn't be necessary. Also, I noticed that there was some stuff that looked weird in upgrade-system.sh -- there seems to be two different redirects, both to /dev/null and to a log file. What's up with that?

[Michagogo]

Update: Rerunning make-base-vm worked, and I was able to build without that problem, and as of right now I was again able to build, still without this problem.

[Michagogo]

/me pings @devrandom

[devrandom]

This seems to be related to 7d1e7c5 . Before this commit, the grub package was being picked up, which was causing issues with Ubuntu trusty. However, it looks like the package change is not backwards compatible with existing VM images. I don't have the bandwidth to look into the root cause right now. The workaround is to recreate the base VM image.

[Michagogo]

Oh, I see. That's good to know... I was worried that I'd have to redo that every time package x got updated or something.

[Michagogo]

Wait, I thought grub is removed before the upgrade?

[devrandom]

Yes, grub is removed, but look like the replacement, grub-pc fails to install properly.

[Michagogo]

@devrandom Looks like it's not just a matter of backwards compatibility. It's happening again. Looking at the list of packages:

The following NEW packages will be installed:
  grub-gfxpayload-lists grub-pc grub2-common linux-image-3.2.0-83-generic
  linux-image-3.2.0-83-virtual
The following packages will be upgraded:
  apt apt-utils debootstrap dnsmasq-base dpkg dpkg-dev libapt-inst1.4
  libapt-pkg4.12 libcurl3-gnutls libdpkg-perl libssl1.0.0 libtasn1-3 libx11-6
  libx11-data libxext6 linux-image-generic linux-image-virtual linux-libc-dev
  ntpdate openssl tzdata
21 upgraded, 5 newly installed, 0 to remove and 0 not upgraded.

I'm trying to think what might be pulling it in, and I suspect it may have something to do with upgrading Linux...

[devrandom]

Please let me know here if this happens again.

[devrandom]

It seems like the root issue is that we don't need a kernel and grub on LXC, but vmbuilder doesn't know that.

Perhaps it's time to move to a different bootstrapping method, at least for LXC.

[Michagogo]

There's lxc-create, though when I tried figuring out how it works I ended
up confusing myself. There's a whole set of tools for LXC creation, use,
and management, a little more sophisticated than converting from a KVM
image and using cp to reset... If someone who actually knew how all that
worked were to take a look at gitian's LXC support, I suspect it would be
possible to make it work really well, as opposed to having a bunch of hacks
layered on top of each other.

On Sunday, May 17, 2015, Dev Random notifications@github.com wrote:

It seems like the root issue is that we don't need a kernel and grub on
LXC, but vmbuilder doesn't know that.

Perhaps it's time to move to a different bootstrapping method, at least
for LXC.

—
Reply to this email directly or view it on GitHub
#86 (comment)
.

[gurnec]

FWIW, I think I see what's going on here, at least in my case (an Ubuntu 12.04 container).

The dist-upgrade in upgrade-system.sh pulls in a new kernel image, and the kernel has a dependency for a bootloader. Since a purge grub is done right before the dist-upgrade (and grub is one alternative for a bootloader), the preferred bootloader is brought in instead (grub-pc, another bootloader alternative).

[Michagogo]

Wait, but in that case, shouldn't it refuse to remove grub? Or rather,
automatically remove the kernel? I thought that if you remove a package it
also removes packages that depend on it (for an example, try telling
apt-get to remove something like Python or libc).

On Sunday, May 17, 2015, Christopher Gurnee notifications@github.com
wrote:

FWIW, I think I see what's going on here, at least in my case (an Ubuntu
12.04 container).

The dist-upgrade in upgrade-system.sh pulls in a new kernel image, and
the kernel has a dependency for a bootloader
http://packages.ubuntu.com/precise/linux-image-3.2.0-82-generic. Since
a purge grub is done right before the dist-upgrade (and grub is one
alternative for a bootloader), the preferred bootloader is brought in
instead (grub-pc, another bootloader alternative).

—
Reply to this email directly or view it on GitHub
#86 (comment)
.

[gurnec]

@Michagogo That's the last time I'll try to analyze something on a Saturday night ;-)

You're right again, I just checked the link above, and it's not a dependency, just a recommendation. I don't have time to test this at the moment, but I suspect a much better idea would be to revert my entire commit, and then just add --no-install-recommends to the apt-get dist-upgrade. That would be much simpler and cleaner, if it works.

[devrandom]

I have a revamped support for LXC in the new lxc branch of this repo. Please try it out, I think it might solve all our LXC issues. It eliminates grub, the linux kernel and holds several packages that can cause issues on upgrade.

[gurnec]

@devrandom This looks great, thanks!

I tried building Bitcoin 0.10.1 (12.04 container on a 14.04 host), and ran into one blocking issue and two minor annoyances. I added comments to the commit with proposed small updates to fix what I ran into.

Although I only built the Windows binaries, I can confirm that they match the official release.

[devrandom]

I made some updates to the lxc branch...

[Michagogo]

Does debootstrap pull from -updates and -security? I suspect not, since on first use of a freshly created container it needs to upgrade 122 packages and install 9 new ones. (and I assume it'll need to do that every time, and it's taking ages... and also, it failed. I'm trying it again with a >/dev/null removed to see where.)

[devrandom]

@Michagogo that's correct, it unfortunately does not. Is that a concern? You can use gbuild -i to reuse the updated container during development.

[Michagogo]

Well, it's just really annoying that it takes several times longer now to
gbuild Bitcoin Core releases, which is 3 different builds for each of the 3
OSes. And there's also the issue of it just failing, which I should be able
to tell you more about as soon as it happens again (assuming that happens).
It's running right now with a /dev/null redirect removed...

On Tue, May 19, 2015 at 1:00 AM, Dev Random notifications@github.com
wrote:

@Michagogo https://github.com/Michagogo that's correct, it
unfortunately does not. Is that a concern? You can use gbuild -i to reuse
the updated container during development.

—
Reply to this email directly or view it on GitHub
#86 (comment)
.

[devrandom]

Will look into improving this.