Avoid exposing the host's special mounts #294

nbraud · 2016-02-24T19:32:30Z

This should avoid the kind of problem encountered in #293.

~~Currently, this breaks InstallGrub_1_99 and InstallGrub_2, which need to be fixed to call install-grub from outside the chroot.~~

nbraud · 2016-02-24T20:55:30Z

Actually, I'm not sure calling install-grub inside the chroot is the right idea.

nbraud · 2016-02-24T21:26:26Z

@andsens The latest commit I added rely on nbd having been loaded with max_part being 1.
Also, I wasn't able to check whether that indeed lets install-grub run successfully. I will check tomorrow, in case you don't have time by then.

nbraud · 2016-02-24T21:40:40Z

@andsens Slightly unrelated, but do you think the “fake /dev” approach would be relevant for debootstrap too?

andsens · 2016-02-24T22:59:47Z

@andsens Slightly unrelated, but do you think the “fake /dev” approach would be relevant for debootstrap too?

It wouldn't. Whoever coded that part was the right kind of lazy about it. debootstrap just extracts a very special tar.gz directly into /dev and calls it a day. It's a file you really shouldn't try extracting anywhere else :-)

Actually, I'm not sure calling install-grub inside the chroot is the right idea.

On of the reasons for doing this might be a leftover from grub 1.99. When run on the host, the grub helper scripts make all kinds of assumptions that you can't configure your way around. More importantly though is that you have to run grub inside the guest. What if the host boots from extlinux or a different version of grub?

andsens · 2016-02-24T23:05:01Z

In the previous comment I forgot to mention how impressive this PR is. That is some really cool stuff!!
I love how it demystifies the whole /dev magic :-)

However, before merging, this will need some heavy testing and a lot more comments.

nbraud · 2016-02-25T12:24:30Z

Whoever coded that part was the right kind of lazy about it. debootstrap just extracts a very special tar.gz directly into /dev and calls it a day.

Ok, I misunderstood your comment in #293 and thought that debootstrap actually mounts a devfs there.

this will need some heavy testing and a lot more comments.

Definitely. That's just a first stab at it because I got incredibly annoyed by #293.
I guess it gave me incentive to fix it, though :-)

nbraud · 2016-02-25T18:57:40Z

@andsens I added some commits, now the grub2 install works.
Grub1.99 should work too, but I didn't check

nbraud · 2016-02-29T22:20:43Z

@andsens Did you have the opportunity to try this?

andsens · 2016-02-29T23:08:06Z

Hoooly crap. How much documentation did you have to sift through to get this right? Wow.
I have not had the time to test this yet. Sorry. Maybe we could pull in some help? (I'll definitely test it myself as well, though).
Paging @JamesBromberger, @myhro, @srivasta and @zmarano. Could you guys try this out? This PR has the potential to fix loads of annoying errors and could make the bootstrapping process a lot more robust to failure. It also removes unnecessary interaction with the host and could potentially provide a more "clean" system.

andsens · 2016-02-29T23:14:52Z

bootstrapvz/common/tasks/filesystem.py

+                                      makedev(43, nbd_max_part * i + j+1))
+
+                                path = '/dev/mapper/nbd%ip%i' % (i,j+1)
+                                if os.path.exists(path):


Under which circumstances would this path not be a link? Shouldn't we consider that an error? We expect something to be there, but it's not, which means somethings fishy.

Unfortunately, I couldn't find a good reference on the internals of device-mapper.
Basically, I'm unaware of any case where the files in /dev/mapper are not symlinks (except control), I'm just being extra-cautious.

nbraud · 2016-02-29T23:54:51Z

How much documentation did you have to sift through to get this right?

It wasn't quite that bad ;-)
Also, I just woke up and realised that a PID namespace might be the right answer for /proc, so I added that before going back to sleep.

I have not had the time to test this yet.

No problem, it's not quite urgent. I guess one would have to be extra unlucky to hit #293.

myhro · 2016-03-01T11:53:31Z

@andsens this is on my TODO list. Unfortunately I've been sick (it's true what they say about Zika virus) in the past few days and the time I was able to stand on the PC was used for work-related stuff. :-(

This makes the assumption that nbd was loaded with max_part=1

The only block devices we are exposing are /dev/sbd* Also, we are *mounting* /dev, /sys and /proc, none of which are block devices

andsens · 2016-03-13T19:56:12Z

bootstrapvz/common/tasks/filesystem.py

+
+                # The nbd devices
+                os.makedirs(join(dev, 'mapper'), 0o755)
+                nbd_max_part = int(info.volume._module_param('nbd', 'max_part'))


This breaks on EC2, since the _module_param part only works for QEMU volumes.

andsens · 2016-03-13T20:37:39Z

I'm having trouble getting grub2 to install:

[remote] Installing grub 2
[remote] Executing: chroot /target/5fec2bf1/root grub-install /dev/xvdf
[remote] Installing for i386-pc platform.
[remote] grub-install: error: failed to get canonical path of `/dev/mapper/xvdf2'.
[remote] Command 'chroot /target/5fec2bf1/root grub-install /dev/xvdf' returned non-zero exit status 1

nbraud · 2016-03-14T22:05:01Z

@andsens Could I have the manifest which makes it fail?

andsens · 2016-03-14T23:43:18Z

Ah, yes of course. Though it isn't a manifest. It's the gpt_grub_stable ec2 test:
tox -e system tests.system.ec2_ebs_hvm_tests:test_gpt_grub_stable

Or via code:

#!/usr/bin/env python

from tests.system.ec2_ebs_hvm_tests import test_gpt_grub_stable
from bootstrapvz.base.main import setup_loggers

setup_loggers({'--log': '-', '--color': 'default', '--debug': True})
test_gpt_grub_stable()

You'll need a build-servers.yml for the test to be able to run.

andsens · 2016-04-02T09:21:50Z

Hehe, soo, that was easy:

for name, partition in get_partitions().items():
    mknod(join(dev, name), 0o666 | stat.S_IFBLK, makedev(int(partition['major']), int(partition['minor'])))

I think you can even replace the nbd stuff with this (minus the symlinks). The code simply re-creates all the block devices mentioned in /proc/partitions, which you can fetch with get_partitions().

nbraud · 2016-04-06T17:48:57Z

@andsens Thanks for having a look, I was unable to make the time those last weeks.

I will see about using your snippet and testing.

andsens · 2016-06-05T08:28:33Z

OK, so I have tried getting this to work, but there are a few problems:

After isolating the process it seems python can't start new threads. It fails with thread.error: can't start new thread in /usr/lib/python2.7/threading.py:745
Some tasks use unmount() (e.g. the prebootstrapped plugin) to do things to the volume. After remounting /dev is of course not populated any longer, so we need to either re-create it or create /dev outside root and mount --bind it into root

I have improved on your method by simply walking through /dev and re-creating everything in it, you can check it out here.
It's on the special-devices branch.

I am closing this PR, let me know if you find a solution to these problems. I'd love to get this working, but there is just too much other stuff for me to work on.

nbraud · 2016-06-05T14:34:57Z

@andsens Duplicating the entirety of /dev still comes with the risk that the build/installation of the guest might clobber something in the host (for instance during grub-install).

I also didn't have time lately to dedicate to this PR, but when I do I will try sending you separate PRs.
I realise that trying to work simultaneously on /dev, /proc and /sys here didn't help making it very manageable.

P.S.: Actually, it seems everything except /proc isolation made it to your branch (modulo my previous caveat with duplication the entirety of /dev)

andsens reviewed Feb 29, 2016
View reviewed changes

andsens mentioned this pull request Mar 3, 2016

Fails to unmount /proc #302

Open

Nicolas Braud-Santoni added 11 commits March 4, 2016 01:08

MountSpecials: Create a “fake” /dev

9b7f842

MountSpecials: Use a separate devpts instance

2db2470

MountSpecials: Mount sysfs read-only

7c8aa74

Update change log

63a4f8a

MountSpecials: Expose nbd* devices

5592428

This makes the assumption that nbd was loaded with max_part=1

MountSpecials: Fix description

5ad7cef

The only block devices we are exposing are /dev/sbd* Also, we are *mounting* /dev, /sys and /proc, none of which are block devices

MountSpecials: Comments

78ef1ce

MountSpecials: More comments

7829e61

NBD fixup

3de2cfe

Use CLONE_NEWPID and CLONE_NEWIPC to isolate /proc

08eb598

Merge branch 'master' into special-devices

9d565fd

andsens reviewed Mar 13, 2016
View reviewed changes

andsens mentioned this pull request Mar 13, 2016

Enable setting apt.conf(5) options #305

Merged

andsens closed this Jun 5, 2016

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Avoid exposing the host's special mounts #294

Avoid exposing the host's special mounts #294

nbraud commented Feb 24, 2016

nbraud commented Feb 24, 2016

nbraud commented Feb 24, 2016

nbraud commented Feb 24, 2016

andsens commented Feb 24, 2016

andsens commented Feb 24, 2016

nbraud commented Feb 25, 2016

nbraud commented Feb 25, 2016

nbraud commented Feb 29, 2016

andsens commented Feb 29, 2016

andsens Feb 29, 2016

nbraud Feb 29, 2016

nbraud commented Feb 29, 2016

myhro commented Mar 1, 2016

andsens Mar 13, 2016

andsens commented Mar 13, 2016

nbraud commented Mar 14, 2016

andsens commented Mar 14, 2016

andsens commented Apr 2, 2016

nbraud commented Apr 6, 2016

andsens commented Jun 5, 2016

nbraud commented Jun 5, 2016 •

edited

Avoid exposing the host's special mounts #294

Avoid exposing the host's special mounts #294

Conversation

nbraud commented Feb 24, 2016

nbraud commented Feb 24, 2016

nbraud commented Feb 24, 2016

nbraud commented Feb 24, 2016

andsens commented Feb 24, 2016

andsens commented Feb 24, 2016

nbraud commented Feb 25, 2016

nbraud commented Feb 25, 2016

nbraud commented Feb 29, 2016

andsens commented Feb 29, 2016

andsens Feb 29, 2016

Choose a reason for hiding this comment

nbraud Feb 29, 2016

Choose a reason for hiding this comment

nbraud commented Feb 29, 2016

myhro commented Mar 1, 2016

andsens Mar 13, 2016

Choose a reason for hiding this comment

andsens commented Mar 13, 2016

nbraud commented Mar 14, 2016

andsens commented Mar 14, 2016

andsens commented Apr 2, 2016

nbraud commented Apr 6, 2016

andsens commented Jun 5, 2016

nbraud commented Jun 5, 2016 • edited

nbraud commented Jun 5, 2016 •

edited