cloud-init does not mount ephemeral0 on /mnt in nova

[ubuntu-server-builder]

This bug was originally filed in Launchpad as LP: #827590

Launchpad details

affected_projects = ['nova', 'cloud-init (Ubuntu)', 'nova (Ubuntu)', 'cloud-init (Ubuntu Oneiric)', 'nova (Ubuntu Oneiric)']
assignee = None
assignee_name = None
date_closed = 2011-09-13T18:21:00.874190+00:00
date_created = 2011-08-16T19:22:28.284423+00:00
date_fix_committed = None
date_fix_released = None
id = 827590
importance = undecided
is_complete = True
lp_url = https://bugs.launchpad.net/cloud-init/+bug/827590
milestone = None
owner = smoser
owner_name = Scott Moser
private = False
status = invalid
submitter = smoser
submitter_name = Scott Moser
tags = ['apport-bug', 'ec2-images', 'i386', 'oneiric', 'rls-mgr-o-tracking', 'server-o-rs']
duplicates = []

Launchpad user Scott Moser(smoser) wrote on 2011-08-16T19:22:28.284423+00:00

related bug:
bug 828357: request to add a label to the filesystem for ephemeral devices
bug 827598: ephemeral device does not have a filesystem

$ python -c 'import boto.utils; print boto.utils.get_instance_metadata()["block-device-mapping"]'
{'ami': 'sda1', 'root': '/dev/sda1', 'ephemeral0': 'sda2', 'swap': 'sda3'}

$ cat /proc/partitions
$ cat /proc/partitions
major minor #blocks name

 253 0 10485760 vda
 253 16 83886080 vdb

So there is an issue here, at very least the MD should say "/dev/sda" and "/dev/sdb" would be a better guess. Additionally, there is no third disk (as reported in 'swap') present at all.

ProblemType: Bug
DistroRelease: Ubuntu 11.10
Package: cloud-init 0.6.1-0ubuntu14
ProcVersionSignature: User Name 3.0.0-8.11-virtual 3.0.1
Uname: Linux 3.0.0-8-virtual i686
Architecture: i386
Date: Tue Aug 16 19:17:00 2011
Ec2AMI: ami-00000011
Ec2AMIManifest: FIXME
Ec2AvailabilityZone: nova
Ec2InstanceType: <nova.db.sqlalchemy.models.InstanceTypes object at 0x4f32090>
Ec2Kernel: aki-00000010
Ec2Ramdisk: unavailable
PackageArchitecture: all
ProcEnviron:
 LANG=en_US.UTF-8
 SHELL=/bin/bash
SourcePackage: cloud-init
UpgradeStatus: No upgrade log present (probably fresh install)

[ubuntu-server-builder]

Launchpad user Scott Moser(smoser) wrote on 2011-08-16T19:22:28.284423+00:00

Launchpad attachments: Dependencies.txt

[ubuntu-server-builder]

Launchpad user Isaku Yamahata(yamahata) wrote on 2011-08-17T10:46:14.448830+00:00

So far nova lied. It is fixed by nova revision of 1400 in the repository.
Here is the example
ubuntu@ubuntu:~$ cat /proc/partitions
major minor #blocks name

252 0 10485760 vda
252 16 20971520 vdb
252 32 1048576 vdc

ubuntu@ubuntu:~$ python -c 'import boto.utils; print boto.utils.get_instance_metadata()["block-device-mapping"]'
{'ami': 'vda', 'root': '/dev/vda', 'ephemeral0': '/dev/vdd'}

[ubuntu-server-builder]

Launchpad user Thierry Carrez(ttx) wrote on 2011-08-19T09:26:05.445735+00:00

@scott, could you confirm the fix ?

[ubuntu-server-builder]

Launchpad user Scott Moser(smoser) wrote on 2011-08-22T13:42:08.824184+00:00

Just from reading comment 2, the issue is clearly not fixed correctly.

/proc/partitions shows vda, vdb, and vdc, but the block-device-mapping shows ephemeral0 mapped to vdd.

[ubuntu-server-builder]

Launchpad user Isaku Yamahata(yamahata) wrote on 2011-08-23T03:46:12+00:00

On Mon, Aug 22, 2011 at 01:42:08PM -0000, Scott Moser wrote:

Just from reading comment 2, the issue is clearly not fixed correctly.

/proc/partitions shows vda, vdb, and vdc, but the block-device-mapping
shows ephemeral0 mapped to vdd.

Yes, you're right.
The create kvm command line that libvirt created and the created xml for
the guest is as follows. You can see that those target device is specified
correctly in the xml, but the mapping information is lost in the command line.
libvirt doesn't have any way to force the guest to use the setting in
qemu-kvm case because qemu-kvm creates pci devices for each virtio-disk
and the guests understands it sequentially from 'a'. In order to fix it,
we need to pass those mapping information to guest from nova somehow and
teach OS initialization sequence (cloud-init?/udev?) the mapping.

The command line to launch kvm guest.

/usr/bin/kvm -S -M pc-0.14 -enable-kvm -m 512 -smp 1,sockets=1,cores=1,threads=1 -name instance-000000e4 -uuid 8882f5d5-d448-29e3-e124-6739e6fd22d6 -nodefconfig -nodefaults -chardev socket,id=charmonitor,path=/var/lib/libvirt/qemu/instance-000000e4.monitor,server,nowait -mon chardev=charmonitor,id=monitor,mode=readline -rtc base=utc -boot c -kernel /var/lib/nova/instances/instance-000000e4/kernel -append root=/dev/vda console=ttyS0 -drive file=/var/lib/nova/instances/instance-000000e4/disk,if=none,id=drive-virtio-disk0,format=qcow2 -device virtio-blk-pci,bus=pci.0,addr=0x4,drive=drive-virtio-disk0,id=virtio-disk0 -drive file=/var/lib/nova/instances/instance-000000e4/disk.eph0,if=none,id=drive-virtio-disk3, format=qcow2 -device virtio-blk-pci,bus=pci.0,addr=0x5,drive=drive-virtio-disk3, id=virtio-disk3 -drive file=/dev/mapper/nova--volumes-volume--00000040,if=none,id=drive-virtio-disk2,format=raw -device virtio-blk-pci,bus=pci.0,addr=0x6,drive=drive-virtio-disk2,id=virtio-disk2 -netdev tap,fd=18,id=hostnet0 -device rtl8139,netdev=hostnet0,id=net0,mac=02:16:3e:1e:ff:c0,bus=pci.0,addr=0x3 -chardev file, id=charserial0,path=/var/lib/nova/instances/instance-000000e4/console.log -device isa-serial,chardev=charserial0,id=serial0 -chardev pty,id=charserial1 -device isa-serial,chardev=charserial1,id=serial1 -usb -vnc 0.0.0.0:0 -k en-us -vga cirrus -device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x7

the create xml for the guest

instance-000000e4
524288

hvm
/var/lib/nova/instances/instance-000000e4/kernel
root=/dev/vda console=ttyS0




1

    <interface type='bridge'>
        <source bridge='br100'/>
        <mac address='02:16:3e:1e:ff:c0'/>
        <!--   <model type='virtio'/>  CANT RUN virtio network right now -->
        <filterref filter="nova-instance-instance-000000e4-02163e1effc0">
            <parameter name="IP" value="172.17.220.4" />
            <parameter name="DHCPSERVER" value="172.17.220.1" />
        </filterref>
    </interface>

    <!-- The order is significant here.  File must be defined first -->
    <serial type="file">
        <source path='/var/lib/nova/instances/instance-000000e4/console.log'

/>

    <console type='pty' tty='/dev/pts/2'>
        <source path='/dev/pts/2'/>
        <target port='0'/>
    </console>

    <serial type='pty'>
        <source path='/dev/pts/2'/>
        <target port='0'/>
    </serial>

    <graphics type='vnc' port='-1' autoport='yes' keymap='en-us' listen='0.0

.0.0'/>

[ubuntu-server-builder]

Launchpad user Scott Moser(smoser) wrote on 2011-08-23T12:32:16.557260+00:00

I understand that actually mapping a device name from hypervisor to guest is actually impossible, but hopefully we can come up with a better guess.

Its unfortunate that libvirt doesn't even seem to be using the "dev=" as a suggestion of device ordering along the virtio bus. Instead, I'm guessing it is using order present in xml. I was hoping at very least that suggesting 'vda', 'vdb', 'vdc' would render in kvm command line to addr=0x4, addr=0x5, addr=0x6 respectively.

Since libvirt seems to be relying on order in the xml, is it possible to specify the "correct" order and also sane "dev=" attributes?

Ie, what would happen if you had xml like:

|
|
|
|
|
|
|
|
|
|
|
|
|
|
|

And then specified block device mapping like:
{
'ami': 'vda',
'root': '/dev/vda',
'ephemeral0': '/dev/vdb'
'vol0': '/dev/vdc'
}

Again, I understand, you can't force the guest to name devices in any way, but hopefully we can at least map "vda" to "first device on virtio bus", and "vdb" to "second disk on virtio bus".

[ubuntu-server-builder]

Launchpad user Scott Moser(smoser) wrote on 2011-09-13T18:23:49.498787+00:00

With Isaku's branch this is fixed with no cloud-init changes, so I've marked those tasks as invalid.

[ubuntu-server-builder]

Launchpad user Thierry Carrez(ttx) wrote on 2011-09-15T12:07:59.684148+00:00

let's do a backport for diablo