Cannot install VM on Gluster bankend storage

[Ganeti-Issues-Migrator]

Originally reported of Google Code with ID 1064.

What software version are you running? Please provide the output of "gnt-
cluster --version", "gnt-cluster version", and "hspace --version".

root@node1:~# gnt-cluster --version
gnt-cluster (ganeti v2.10.7-70-g70f22ba) 2.11.6

<b>What distribution are you using?</b>

root@node1:~# lsb_release -a
No LSB modules are available.
Distributor ID:	Debian
Description:	Debian GNU/Linux 7.8 (wheezy)
Release:	7.8
Codename:	wheezy

<b>What steps will reproduce the problem?</b>
1, root@node1:~# gluster volume create gv0 replica 2 transport tcp node1:/root/tmp1 node1:/root/tmp2
Creation of volume gv0 has been successful. Please start the volume to access data.

2, root@node1:~# gluster volume start gv0
Starting volume gv0 has been successful

3, root@node1:~# gnt-cluster init --vg-name=xenvg --master-netdev=eth0 --gluster-storage-dir=/var/run/ganeti/gluster -D gluster -H kvm:kernel_path=/boot/vmlinuz-3-kvmU,initrd_path=/boot/initrd-3-kvmU -N link=xen-br0 --enabled-hypervisor=kvm --enabled-disk-templates=gluster,drbd,plain cluster


4, root@node1:~# sudo gnt-instance add -n node1 -o debootstrap+default -t gluster -s 5G instance1
Wed Apr 15 05:30:13 2015  - INFO: Resolved given name 'instance1' to 'instance1.ganeti.com'
Wed Apr 15 05:30:17 2015 * disk 0, size 5.0G
Wed Apr 15 05:30:17 2015 * creating instance disks...
Wed Apr 15 05:30:22 2015  - WARNING: Device creation failed
Failure: command execution error:
Can't create block device <Disk(type=gluster, logical_id=('loop', 'ganeti/ce24daf6-15fc-4450-abd3-f893b342b601.0'), children=None, visible as /dev/disk/0, size=5120m)> on node node1.ganeti.com for instance instance1.ganeti.com: Error while executing backend function: Path '/var/run/ganeti/gluster/ganeti/ce24daf6-15fc-4450-abd3-f893b342b601.0' is not acceptable for file storage

<b>What is the expected output? What do you see instead?</b>

Expected: Create VM on gluster backend storage successfully.
Instead: Cannot install, error happened.

<b>Please provide any additional information below.</b>

I can install VM successfully with plain disk template as follows.

root@node1:~# sudo gnt-instance add -n node1 -o debootstrap+default -t plain -s 5G instance1
Wed Apr 15 05:37:53 2015  - INFO: Resolved given name 'instance1' to 'instance1.ganeti.com'
Wed Apr 15 05:37:57 2015 * disk 0, size 5.0G
Wed Apr 15 05:37:57 2015 * creating instance disks...
Wed Apr 15 05:37:58 2015 adding instance instance1.ganeti.com to cluster config
Wed Apr 15 05:37:59 2015  - INFO: Waiting for instance instance1.ganeti.com to sync disks
Wed Apr 15 05:37:59 2015  - INFO: Instance instance1.ganeti.com's disks are in sync
Wed Apr 15 05:38:00 2015 * running the instance OS create scripts...
Wed Apr 15 05:38:10 2015 * starting instance...
root@node1:~# gnt-instance list
Instance             Hypervisor OS                  Primary_node     Status  Memory
instance1.ganeti.com kvm        debootstrap+default node1.ganeti.com running   128M


Cluster infos are as follows.

root@node1:~# gnt-cluster verify
Submitted jobs 3, 4
Waiting for job 3 ...
Wed Apr 15 05:29:53 2015 * Verifying cluster config
Wed Apr 15 05:29:53 2015 * Verifying cluster certificate files
Wed Apr 15 05:29:53 2015 * Verifying hypervisor parameters
Wed Apr 15 05:29:53 2015 * Verifying all nodes belong to an existing group
Waiting for job 4 ...
Wed Apr 15 05:29:53 2015 * Verifying group 'default'
Wed Apr 15 05:29:53 2015 * Gathering data (1 nodes)
Wed Apr 15 05:29:54 2015 * Gathering disk information (1 nodes)
Wed Apr 15 05:29:54 2015 * Verifying configuration file consistency
Wed Apr 15 05:29:54 2015 * Verifying node status
Wed Apr 15 05:29:54 2015 * Verifying instance status
Wed Apr 15 05:29:54 2015 * Verifying orphan volumes
Wed Apr 15 05:29:54 2015 * Verifying N+1 Memory redundancy
Wed Apr 15 05:29:54 2015 * Other Notes
Wed Apr 15 05:29:54 2015 * Hooks Results

root@node1:~# gnt-cluster info
Cluster name: cluster.ganeti.com
Cluster UUID: db2ddf84-451a-462e-87d7-c92c5c161d7e
Creation time: 2015-04-15 05:29:39
Modification time: 2015-04-15 05:29:46
Master node: node1.ganeti.com
Architecture (this node): 64bits (x86_64)
Tags: (none)
Default hypervisor: kvm
Enabled hypervisors: kvm
Hypervisor parameters: 
  kvm: 
    acpi: True
    boot_order: disk
    cdrom2_image_path: 
    cdrom_disk_type: 
    cdrom_image_path: 
    cpu_cores: 0
    cpu_mask: all
    cpu_sockets: 0
    cpu_threads: 0
    cpu_type: 
    disk_cache: default
    disk_type: paravirtual
    floppy_image_path: 
    initrd_path: /boot/initrd-3-kvmU
    kernel_args: ro
    kernel_path: /boot/vmlinuz-3-kvmU
    keymap: 
    kvm_extra: 
    kvm_flag: 
    kvm_path: /usr/bin/kvm
    machine_version: 
    mem_path: 
    migration_bandwidth: 32
    migration_downtime: 30
    migration_mode: live
    migration_port: 8102
    nic_type: paravirtual
    reboot_behavior: reboot
    root_path: /dev/vda1
    security_domain: 
    security_model: none
    serial_console: True
    serial_speed: 38400
    soundhw: 
    spice_bind: 
    spice_image_compression: 
    spice_ip_version: 0
    spice_jpeg_wan_compression: 
    spice_password_file: 
    spice_playback_compression: True
    spice_streaming_video: 
    spice_tls_ciphers: HIGH:-DES:-3DES:-EXPORT:-ADH
    spice_use_tls: False
    spice_use_vdagent: True
    spice_zlib_glz_wan_compression: 
    usb_devices: 
    usb_mouse: 
    use_chroot: False
    use_localtime: False
    user_shutdown: False
    vga: 
    vhost_net: False
    vnc_bind_address: 
    vnc_password_file: 
    vnc_tls: False
    vnc_x509_path: 
    vnc_x509_verify: False
    vnet_hdr: True
OS-specific hypervisor parameters: 
OS parameters: 
Hidden OSes: 
Blacklisted OSes: 
Cluster parameters: 
  candidate pool size: 10
  maximal number of jobs running simultaneously: 20
  master netdev: eth0
  master netmask: 32
  use external master IP address setup script: False
  lvm volume group: xenvg
  lvm reserved volumes: (none)
  drbd usermode helper: /bin/true
  file storage path: /srv/ganeti/file-storage
  shared file storage path: /srv/ganeti/shared-file-storage
  gluster storage path: /var/run/ganeti/gluster
  maintenance of node health: False
  uid pool: 
  default instance allocator: hail
  default instance allocator parameters: 
  primary ip version: 4
  preallocation wipe disks: False
  OS search path: /srv/ganeti/os
  ExtStorage Providers search path: /srv/ganeti/extstorage
  enabled disk templates: gluster, drbd, plain
  enabled user shutdown: False
Default node parameters: 
  exclusive_storage: False
  oob_program: 
  ovs: False
  ovs_link: 
  ovs_name: switch1
  spindle_count: 1
  ssh_port: 22
Default instance parameters: 
  default: 
    always_failover: False
    auto_balance: True
    maxmem: 128
    minmem: 128
    spindle_use: 1
    vcpus: 1
Default nic parameters: 
  default: 
    link: xen-br0
    mode: bridged
    vlan: 
Default disk parameters: 
  blockdev: 
  diskless: 
  drbd: 
    c-delay-target: 1
    c-fill-target: 0
    c-max-rate: 61440
    c-min-rate: 4096
    c-plan-ahead: 20
    data-stripes: 1
    disk-barriers: n
    disk-custom: 
    dynamic-resync: False
    meta-barriers: False
    meta-stripes: 1
    metavg: xenvg
    net-custom: 
    protocol: C
    resync-rate: 61440
  ext: 
  file: 
  gluster: 
    access: kernelspace
    host: 127.0.0.1
    port: 24007
    volume: gv0
  plain: 
    stripes: 1
  rbd: 
    access: kernelspace
    pool: rbd
  sharedfile: 
Instance policy - limits for instances: 
  bounds specs: 
    - max/0: 
        cpu-count: 8
        disk-count: 16
        disk-size: 1048576
        memory-size: 32768
        nic-count: 8
        spindle-use: 12
      min/0: 
        cpu-count: 1
        disk-count: 1
        disk-size: 1024
        memory-size: 128
        nic-count: 1
        spindle-use: 1
  std: 
    cpu-count: 1
    disk-count: 1
    disk-size: 1024
    memory-size: 128
    nic-count: 1
    spindle-use: 1
  allowed disk templates: drbd, plain, gluster
  vcpu-ratio: 4
  spindle-ratio: 32


Some logs related to gluster are in the following.
root@node1:/var/log/ganeti# grep gluster * -R
[...]
node-daemon.log:2015-04-15 05:30:17,745: ganeti-noded pid=1054 INFO RunCmd mount -t glusterfs -o server-port=24007 127.0.0.1:/gv0 /var/run/ganeti/gluster
node-daemon.log:2015-04-15 05:30:21,946: ganeti-noded pid=1054 INFO RunCmd umount /var/run/ganeti/gluster
node-daemon.log:  File "/usr/local/share/ganeti/2.11/ganeti/storage/gluster.py", line 456, in Create
node-daemon.log:FileStoragePathError: Path '/var/run/ganeti/gluster/ganeti/ce24daf6-15fc-4450-abd3-f893b342b601.0' is not acceptable for file storage

Originally added on 2015-04-15 09:42:36 +0000 UTC.

[Ganeti-Issues-Migrator]

Is /var/run/ganeti/gluster/ganeti whitelisted in /etc/ganeti/file-storage-paths ?

Note that for security reasons, Ganeti refuses to write to files/directories
that are not whitelisted for this purpose on the node itself. The error indicates
that this witelisting did not happen.

Originally added on 2015-04-15 09:51:15 +0000 UTC.

Changed State: NeedFeedback

[Ganeti-Issues-Migrator]

After I add /var/run/ganeti/gluster/ganeti whitelisted in /etc/ganeti/file-storage-paths. When I add an instance, I happen to following errors.

root@node1:/var/log/ganeti# sudo gnt-instance add -n node1 -o debootstrap+default -t gluster -s 5G instance1
Wed Apr 15 07:11:42 2015  - INFO: Resolved given name 'instance1' to 'instance1.ganeti.com'
Wed Apr 15 07:11:46 2015 * disk 0, size 5.0G
Wed Apr 15 07:11:46 2015 * creating instance disks...
Wed Apr 15 07:11:55 2015 adding instance instance1.ganeti.com to cluster config
Wed Apr 15 07:11:56 2015  - INFO: Waiting for instance instance1.ganeti.com to sync disks
Wed Apr 15 07:11:56 2015  - INFO: Instance instance1.ganeti.com's disks are in sync
Wed Apr 15 07:11:56 2015 * running the instance OS create scripts...
Failure: command execution error:
Could not add os for instance instance1.ganeti.com on node node1.ganeti.com: OS create script failed (exited with exit code 1), last lines in the log file:
sfdisk: ERROR: sector 0 does not have an msdos signature
 /dev/loop0: unrecognized partition table type
Old situation:
No partitions found
New situation:
Units = sectors of 512 bytes, counting from 0

   Device Boot    Start       End   #sectors  Id  System
/dev/loop0p1   *      2048  10485759   10483712  83  Linux
/dev/loop0p2             0         -          0   0  Empty
/dev/loop0p3             0         -          0   0  Empty
/dev/loop0p4             0         -          0   0  Empty
Successfully wrote the new partition table

Re-reading the partition table ...
BLKRRPART: Invalid argument

passwd: password expiry information changed.
loop: can't delete device /dev/loop0: Device or resource busy
umount: /tmp/tmp.ExMAXTrGE1: not found

root@node1:/var/log/ganeti# gnt-instance startup instance1
Waiting for job 6 for instance1.ganeti.com ...
Job 6 for instance1.ganeti.com has failed: Failure: command execution error:
Could not start instance 'instance1.ganeti.com': Hypervisor error: Failed to start instance instance1.ganeti.com: exited with exit code 1 (kvm: -drive file=/var/run/ganeti/instance-disks/instance1.ganeti.com:0,format=raw,if=none,cache=none,id=hotdisk-00d45331-pci-4,bus=0,unit=4: could not open disk image /var/run/ganeti/instance-disks/instance1.ganeti.com:0: Invalid argument
)

Originally added on 2015-04-15 11:21:40 +0000 UTC.

[Ganeti-Issues-Migrator]

It seems that there is no "/var/run/ganeti/instance-disks/instance1.ganeti.com:0" disk image file so the error happened.

Originally added on 2015-04-15 11:30:22 +0000 UTC.

[Ganeti-Issues-Migrator]

It seems that there is no "/var/run/ganeti/instance-disks/instance1.ganeti.com:0" disk image file so the error happened.

Some logs are as follows.

root@node1:/var/log/ganeti# cat node-daemon.log 
[...]
2015-04-15 07:32:47,816: ganeti-noded pid=3240 INFO 192.168.1.101:57593 POST /bridges_exist HTTP/1.1 200
2015-04-15 07:32:47,906: ganeti-noded pid=3241 INFO 192.168.1.101:57594 POST /instance_info HTTP/1.1 200
2015-04-15 07:32:47,996: ganeti-noded pid=3242 INFO RunCmd /usr/bin/kvm --help
2015-04-15 07:32:48,045: ganeti-noded pid=3242 INFO 192.168.1.101:57595 POST /node_info HTTP/1.1 200
2015-04-15 07:32:48,141: ganeti-noded pid=3244 INFO 192.168.1.101:57596 POST /hooks_runner HTTP/1.1 200
2015-04-15 07:32:49,403: ganeti-noded pid=3245 INFO 192.168.1.101:57597 POST /write_ssconf_files HTTP/1.1 200
2015-04-15 07:32:49,494: ganeti-noded pid=3246 INFO 192.168.1.101:57598 POST /blockdev_assemble HTTP/1.1 200
2015-04-15 07:32:49,600: ganeti-noded pid=3247 INFO 192.168.1.101:57599 POST /blockdev_assemble HTTP/1.1 200
2015-04-15 07:32:49,694: ganeti-noded pid=3248 INFO RunCmd /usr/bin/kvm --help
2015-04-15 07:32:49,743: ganeti-noded pid=3248 INFO RunCmd /usr/bin/kvm -M '?'
2015-04-15 07:32:49,796: ganeti-noded pid=3248 INFO RunCmd /usr/bin/kvm -device '?'
2015-04-15 07:32:49,890: ganeti-noded pid=3248 INFO RunCmd /usr/local/lib/ganeti/kvm-ifup tap0
2015-04-15 07:32:49,941: ganeti-noded pid=3248 WARNING KVM: overriding disk_cache setting 'default' with 'none' to prevent shared storage corruption on migration
2015-04-15 07:32:49,943: ganeti-noded pid=3248 INFO RunCmd /usr/bin/kvm -name instance1.ganeti.com -m 128 -smp 1 -pidfile /var/run/ganeti/kvm-hypervisor/pid/instance1.ganeti.com -balloon virtio,id=balloon,bus=pci.0,addr=0x3 -daemonize -machine pc-1.1 -kernel /boot/vmlinuz-3-kvmU -initrd /boot/initrd-3-kvmU -append 'root=/dev/vda1 ro console=ttyS0,38400' -monitor unix:/var/run/ganeti/kvm-hypervisor/ctrl/instance1.ganeti.com.monitor,server,nowait -serial unix:/var/run/ganeti/kvm-hypervisor/ctrl/instance1.ganeti.com.serial,server,nowait -usb -display none -uuid 27282764-b6cd-468f-930f-3d93dca14716 -netdev type=tap,id=hotnic-8a08c976-pci-5,fd=8 -device virtio-net-pci,mac=aa:00:00:3f:bd:44,id=hotnic-8a08c976-pci-5,bus=pci.0,addr=0x5,netdev=hotnic-8a08c976-pci-5 -qmp unix:/var/run/ganeti/kvm-hypervisor/ctrl/instance1.ganeti.com.qmp,server,nowait -qmp unix:/var/run/ganeti/kvm-hypervisor/ctrl/instance1.ganeti.com.kvmd,server,nowait -device virtio-blk-pci,drive=hotdisk-00d45331-pci-4,id=hotdisk-00d45331-pci-4,bus=pci.0,addr=0x4 -drive file=/var/run/ganeti/instance-disks/instance1.ganeti.com:0,format=raw,if=none,cache=none,id=hotdisk-00d45331-pci-4,bus=0,unit=4 -S
2015-04-15 07:32:50,101: ganeti-noded pid=3248 ERROR Hypervisor error: Failed to start instance instance1.ganeti.com: exited with exit code 1 (kvm: -drive file=/var/run/ganeti/instance-disks/instance1.ganeti.com:0,format=raw,if=none,cache=none,id=hotdisk-00d45331-pci-4,bus=0,unit=4: could not open disk image /var/run/ganeti/instance-disks/instance1.ganeti.com:0: Invalid argument
)
Traceback (most recent call last):
  File "/usr/local/share/ganeti/2.11/ganeti/backend.py", line 1883, in StartInstance
    hyper.StartInstance(instance, block_devices, startup_paused)
  File "/usr/local/share/ganeti/2.11/ganeti/hypervisor/hv_kvm.py", line 2095, in StartInstance
    self._ExecuteKVMRuntime(instance, kvm_runtime, kvmhelp)
  File "/usr/local/share/ganeti/2.11/ganeti/hypervisor/hv_kvm.py", line 2016, in _ExecuteKVMRuntime
    self._RunKVMCmd(name, kvm_cmd, tapfds)
  File "/usr/local/share/ganeti/2.11/ganeti/hypervisor/hv_kvm.py", line 1831, in _RunKVMCmd
    (name, result.fail_reason, result.output))
HypervisorError: Failed to start instance instance1.ganeti.com: exited with exit code 1 (kvm: -drive file=/var/run/ganeti/instance-disks/instance1.ganeti.com:0,format=raw,if=none,cache=none,id=hotdisk-00d45331-pci-4,bus=0,unit=4: could not open disk image /var/run/ganeti/instance-disks/instance1.ganeti.com:0: Invalid argument
)
2015-04-15 07:32:50,109: ganeti-noded pid=3248 INFO 192.168.1.101:57600 POST /instance_start HTTP/1.1 200
2015-04-15 07:32:50,296: ganeti-noded pid=3280 INFO 192.168.1.101:57601 POST /blockdev_shutdown HTTP/1.1 200
2015-04-15 07:32:50,389: ganeti-noded pid=3281 INFO 192.168.1.101:57602 POST /hooks_runner HTTP/1.1 200

Originally added on 2015-04-15 11:36:48 +0000 UTC.

[Ganeti-Issues-Migrator]

Looking at Comment #2, it seems that the main issue is that the OS creation script failed. This might be similar to Issue 1037. Please try to update the relevant OS creation script (most likely /srv/ganeti/os/debootstrap/) with this patch (must be done on all nodes!):

diff --git a/create b/create
index 6565176..b6c1754 100755
--- a/create
+++ b/create
@@ -37,7 +37,7 @@ CACHE_FILE="$CACHE_DIR/cache-${SUITE}-${DPKG_ARCH}.tar"
 if [ ! -b $blockdev ]; then
   ORIGINAL_BLOCKDEV=$blockdev
   blockdev=$(losetup --show -f $blockdev)
-  CLEANUP+=("losetup -d $blockdev")
+  CLEANUP+=("udevadm settle ; losetup -d $blockdev")
 fi
 
 DEVICE_SIZE=$(blockdev --getsize64 $blockdev)

Since I don't know which version of the OS installation script you're using, I'm giving it as a reference - it might not apply cleanly, so in that case just edit the scripts manually. The important thing is that 'udevadm settle' is run during the cleanup just before 'losetup -d ...'.

Originally added on 2015-04-15 15:34:17 +0000 UTC.

Mentioned Issue with GoogleCode ID 1037 has been migrated to GitHub with ID #1093.

[Ganeti-Issues-Migrator]

After I update create script under "/srv/ganeti/os/debootstrap/" as what you said above, I start an instance again. However, there is still a problem as follows. I am trying to fix this problem.

root@node1:~# sudo gnt-instance add -n node1 -o debootstrap+default -t gluster -s 5G instance1
Wed Apr 15 21:45:28 2015  - INFO: Resolved given name 'instance1' to 'instance1.ganeti.com'
Wed Apr 15 21:45:32 2015 * disk 0, size 5.0G
Wed Apr 15 21:45:32 2015 * creating instance disks...
Wed Apr 15 21:45:32 2015 adding instance instance1.ganeti.com to cluster config
Wed Apr 15 21:45:33 2015  - INFO: Waiting for instance instance1.ganeti.com to sync disks
Wed Apr 15 21:45:33 2015  - INFO: Instance instance1.ganeti.com's disks are in sync
Wed Apr 15 21:45:33 2015 * running the instance OS create scripts...
Failure: command execution error:
Could not add os for instance instance1.ganeti.com on node node1.ganeti.com: OS create script failed (exited with exit code 1), last lines in the log file:
sfdisk: ERROR: sector 0 does not have an msdos signature
 /dev/loop5: unrecognized partition table type
Old situation:
No partitions found
New situation:
Units = sectors of 512 bytes, counting from 0

   Device Boot    Start       End   #sectors  Id  System
/dev/loop5p1   *      2048  10485759   10483712  83  Linux
/dev/loop5p2             0         -          0   0  Empty
/dev/loop5p3             0         -          0   0  Empty
/dev/loop5p4             0         -          0   0  Empty
Successfully wrote the new partition table

Re-reading the partition table ...
BLKRRPART: Invalid argument

passwd: password expiry information changed.
settle: invalid option -- 'd'
umount: /tmp/tmp.q9FdTXtK05: not found



The verion of OS creation script:
  # git clone git://git.ganeti.org/instance-debootstrap.git
  # cd instance-debootstrap/
  # ./autogen.sh 
  # ./configure --with-os-dir=/srv/ganeti/os
  # make
  # make install
  # apt-get install debootstrap dump kpartx
root@node1:/var/log/ganeti/os# debootstrap --version
debootstrap 1.0.48+deb7u3

Originally added on 2015-04-16 03:20:51 +0000 UTC.

[Ganeti-Issues-Migrator]

Can you please your current 'create' script, to help to investigate? Thanks, Petr

Originally added on 2015-04-16 07:49:07 +0000 UTC.

[Ganeti-Issues-Migrator]

Can you please your current, patched version of the 'create' script, to help to investigate? Thanks, Petr

Originally added on 2015-04-16 07:54:00 +0000 UTC.

[Ganeti-Issues-Migrator]

I have found the problem is caused by function `format_disk0` in create script.

root@node1:/srv/ganeti/os/debootstrap# cat create 
[...]
 45 
 46 DEVICE_SIZE=$(blockdev --getsize64 $blockdev)
 47 if [ "$DEVICE_SIZE" -lt $MIN_DEV_SIZE ]; then 
 48   echo "Device size is too small ($((DEVICE_SIZE/1048576)) MB)" 1>&2
 49   echo "Required size is at least 256MB" 1>&2
 50   exit 1
 51 fi
 52 
 53 echo "999 DEVICE_SIZE is $DEVICE_SIZE"
 54 
 55 if [ "$PARTITION_STYLE" = "none" ]; then
 56   echo "999 style none"
 57   filesystem_dev=$blockdev
 58   echo "999 filesystem_dev is $blockdev"
 59 elif [ "$PARTITION_STYLE" = "msdos" ]; then
 60   echo "999 style msdos"                   <---------------------------
 61   # Create one big partition, and make it bootable
 62   format_disk0 $blockdev
 63   echo "999 call                           <--------------------------- format_disk0"                                                                                                    
 64   filesystem_dev=$(map_disk0 $blockdev)
 65   echo "999 filesystem_dev is $blockdev"
 66   CLEANUP+=("unmap_disk0 $blockdev")
 67 else
 68   echo "Unknown partition style $PARTITION_STYLE" 1>&2
 69   exit 1
 70 fi
 71 
 72 mke2fs -Fjqt $OSP_FILESYSTEM $filesystem_dev
 73 root_uuid=$($VOL_ID $filesystem_dev )
[...]

root@node1:/var/log/ganeti/os# cat add-debootstrap+default-instance2.ganeti.com-2015-04-16_03_33_41.log 
999 enter create script
999 added udevadm here!!!
999 DEVICE_SIZE is 5368709120
999 style msdos                                 <---------------------------
BLKRRPART: Invalid argument
Disk /dev/loop2: cannot get geometry

Disk /dev/loop2: 5120 cylinders, 64 heads, 32 sectors/track

sfdisk: ERROR: sector 0 does not have an msdos signature
 /dev/loop2: unrecognized partition table type
Old situation:
No partitions found
New situation:
Units = sectors of 512 bytes, counting from 0

   Device Boot    Start       End   #sectors  Id  System
/dev/loop2p1   *      2048  10485759   10483712  83  Linux
/dev/loop2p2             0         -          0   0  Empty
/dev/loop2p3             0         -          0   0  Empty
/dev/loop2p4             0         -          0   0  Empty
Successfully wrote the new partition table

Re-reading the partition table ...
BLKRRPART: Invalid argument

999 call format_disk0                            <--------------------------
999 filesystem_dev is /dev/loop2
999 run mke2fs here!!!
999 reset passwd
passwd: password expiry information changed.
999 leave create script
settle: invalid option -- 'd'
umount: /tmp/tmp.0xN0xEkiIW: not found


The related codes in create is "format_disk0 $blockdev ". $blockdev is "/dev/loop2". It seems that something wrong with "/dev/loop2". $blockdev is passed into format_disk0 to execute as the command "sfdisk -H 64 -S 32 -u S --quiet --Linux $blockdev". When I run this command as follows, it shows me the same error.

root@node1:/var/log/ganeti/os# sfdisk -H 64 -S 32 -u S --quiet --Linux /dev/loop2
BLKRRPART: Invalid argument
Disk /dev/loop2: cannot get geometry

Disk /dev/loop2: 5120 cylinders, 64 heads, 32 sectors/track
Old situation:
Units = sectors of 512 bytes, counting from 0

   Device Boot    Start       End   #sectors  Id  System
/dev/loop2p1   *      2048  10485759   10483712  83  Linux
/dev/loop2p2             0         -          0   0  Empty
/dev/loop2p3             0         -          0   0  Empty
/dev/loop2p4             0         -          0   0  Empty
^C

I am trying to fix this problem.

Originally added on 2015-04-16 07:59:53 +0000 UTC.

[Ganeti-Issues-Migrator]

Hi Petr,

Thanks for your reply. Actually I have not changed the main part of 'create' script. I installed up-to-date "git://git.ganeti.org/instance-debootstrap.git" tree.

$ git show
commit 5a8a67b7579d89488a9de4c2f44a15d61a5eee2b
Author: Apollon Oikonomopoulos <apoikos@gmail.com>
Date:   Mon Mar 24 22:28:15 2014 +0200

root@node1:~# git diff instance-debootstrap/create /srv/ganeti/os/debootstrap/create
diff --git a/instance-debootstrap/create b/srv/ganeti/os/debootstrap/create
index 6565176..59e3ea0 100755
--- a/instance-debootstrap/create
+++ b/srv/ganeti/os/debootstrap/create
@@ -17,6 +17,8 @@
 # Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA
 # 02110-1301, USA.
 
+echo "999 enter create script"
+
 set -e
 
 # minimum device size is 256 MB, but we use 255 to account for
@@ -37,7 +39,8 @@ CACHE_FILE="$CACHE_DIR/cache-${SUITE}-${DPKG_ARCH}.tar"
 if [ ! -b $blockdev ]; then
   ORIGINAL_BLOCKDEV=$blockdev
   blockdev=$(losetup --show -f $blockdev)
-  CLEANUP+=("losetup -d $blockdev")
+  echo "999 added udevadm here!!!"
+  CLEANUP+=("udevadm settle ; losetup -d $blockdev")
 fi
 
 DEVICE_SIZE=$(blockdev --getsize64 $blockdev)
@@ -47,12 +50,19 @@ if [ "$DEVICE_SIZE" -lt $MIN_DEV_SIZE ]; then
   exit 1
 fi
 
+echo "999 DEVICE_SIZE is $DEVICE_SIZE"
+
 if [ "$PARTITION_STYLE" = "none" ]; then
+  echo "999 style none"
   filesystem_dev=$blockdev
+  echo "999 filesystem_dev is $blockdev"
 elif [ "$PARTITION_STYLE" = "msdos" ]; then
+  echo "999 style msdos"
   # Create one big partition, and make it bootable
   format_disk0 $blockdev
+  echo "999 call format_disk0"
   filesystem_dev=$(map_disk0 $blockdev)
+  echo "999 filesystem_dev is $blockdev"
   CLEANUP+=("unmap_disk0 $blockdev")
 else
   echo "Unknown partition style $PARTITION_STYLE" 1>&2
@@ -62,6 +72,8 @@ fi
 mke2fs -Fjqt $OSP_FILESYSTEM $filesystem_dev
 root_uuid=$($VOL_ID $filesystem_dev )
 
+echo "999 run mke2fs here!!!"
+
 if [ -n "$swapdev" ]; then
   mkswap $swapdev
   swap_uuid=$($VOL_ID $swapdev || true )
@@ -113,6 +125,7 @@ else
   fi
 fi
 
+echo "999 reset passwd"
 # reset the root password
 chroot $TMPDIR passwd -d root
 
@@ -189,6 +202,8 @@ if [ -n "$RUN_PARTS" -a -n "$CUSTOMIZE_DIR" -a -d "$CUSTOMIZE_DIR" ]; then
   $RUN_PARTS $CUSTOMIZE_DIR
 fi
 
+echo "999 leave create script"
+
 # execute cleanups
 cleanup
 trap - EXIT 

Originally added on 2015-04-16 08:11:19 +0000 UTC.

[Ganeti-Issues-Migrator]

As I'm not able to reproduce the problem, the previous fix I suggested wasn't probably wasn't the correct one. Please try this one instead:

diff --git a/create b/create
index 6565176..08a2cb3 100755
--- a/create
+++ b/create
@@ -38,6 +38,7 @@ if [ ! -b $blockdev ]; then
   ORIGINAL_BLOCKDEV=$blockdev
   blockdev=$(losetup --show -f $blockdev)
   CLEANUP+=("losetup -d $blockdev")
+  CLEANUP+=("udevadm settle")
 fi
 
 DEVICE_SIZE=$(blockdev --getsize64 $blockdev)

That is, add CLEANUP+=("udevadm settle") _after_ the 'losetup -d ...' command (the cleanup commands are executed in reverse order).

Originally added on 2015-04-17 09:22:26 +0000 UTC.

[Ganeti-Issues-Migrator]

Hi Petr, after I update create script as what you said above. I happen to following error.

root@node1:/srv/ganeti/os/debootstrap# sudo gnt-instance add -n node1 -o debootstrap+default -t gluster -s 5G instance2
Fri Apr 17 05:37:28 2015  - INFO: Resolved given name 'instance2' to 'instance2.ganeti.com'
Fri Apr 17 05:37:32 2015 * disk 0, size 5.0G
Fri Apr 17 05:37:32 2015 * creating instance disks...
Fri Apr 17 05:37:33 2015 adding instance instance2.ganeti.com to cluster config
Fri Apr 17 05:37:34 2015  - INFO: Waiting for instance instance2.ganeti.com to sync disks
Fri Apr 17 05:37:34 2015  - INFO: Instance instance2.ganeti.com's disks are in sync
Fri Apr 17 05:37:34 2015 * running the instance OS create scripts...
Fri Apr 17 05:37:51 2015 * starting instance...
Failure: command execution error:
Could not start instance: Hypervisor error: Failed to start instance instance2.ganeti.com: exited with exit code 1 (kvm: -drive file=/var/run/ganeti/instance-disks/instance2.ganeti.com:0,format=raw,if=none,cache=none,id=hotdisk-8771fb57-pci-4,bus=0,unit=4: could not open disk image /var/run/ganeti/instance-disks/instance2.ganeti.com:0: Invalid argument
)


root@node1:/var/log/ganeti# cat node-daemon.log | tail -30
2015-04-17 05:37:32,666: ganeti-noded pid=6616 INFO 192.168.1.101:37117 POST /hooks_runner HTTP/1.1 200
2015-04-17 05:37:32,960: ganeti-noded pid=6617 INFO 192.168.1.101:37118 POST /file_storage_dir_create HTTP/1.1 200
2015-04-17 05:37:33,064: ganeti-noded pid=6618 INFO RunCmd umount /var/run/ganeti/gluster
2015-04-17 05:37:33,081: ganeti-noded pid=6618 WARNING Failed to unmount GlusterVolume("127.0.0.1", 24007, "gv0") from '/var/run/ganeti/gluster': exited with exit code 1
2015-04-17 05:37:33,087: ganeti-noded pid=6618 INFO 192.168.1.101:37119 POST /blockdev_create HTTP/1.1 200
2015-04-17 05:37:34,392: ganeti-noded pid=6620 INFO 192.168.1.101:37120 POST /write_ssconf_files HTTP/1.1 200
2015-04-17 05:37:34,515: ganeti-noded pid=6621 INFO 192.168.1.101:37121 POST /blockdev_getmirrorstatus HTTP/1.1 200
2015-04-17 05:37:34,678: ganeti-noded pid=6622 INFO RunCmd /srv/ganeti/os/debootstrap/create, output file '/var/log/ganeti/os/add-debootstrap+default-instance2.ganeti.com-2015-04-17_05_37_34.log'
2015-04-17 05:37:51,499: ganeti-noded pid=6622 INFO 192.168.1.101:37122 POST /instance_os_add HTTP/1.1 200
2015-04-17 05:37:51,708: ganeti-noded pid=6702 INFO RunCmd /usr/bin/kvm --help
2015-04-17 05:37:52,455: ganeti-noded pid=6702 INFO RunCmd /usr/bin/kvm -M '?'
2015-04-17 05:37:52,509: ganeti-noded pid=6702 INFO RunCmd /usr/bin/kvm -device '?'
2015-04-17 05:37:52,650: ganeti-noded pid=6702 INFO RunCmd /usr/local/lib/ganeti/kvm-ifup tap0
2015-04-17 05:37:52,886: ganeti-noded pid=6702 WARNING KVM: overriding disk_cache setting 'default' with 'none' to prevent shared storage corruption on migration
2015-04-17 05:37:52,887: ganeti-noded pid=6702 INFO RunCmd /usr/bin/kvm -name instance2.ganeti.com -m 128 -smp 1 -pidfile /var/run/ganeti/kvm-hypervisor/pid/instance2.ganeti.com -balloon virtio,id=balloon,bus=pci.0,addr=0x3 -daemonize -machine pc-1.1 -kernel /boot/vmlinuz-3-kvmU -initrd /boot/initrd-3-kvmU -append 'root=/dev/vda1 ro console=ttyS0,38400' -monitor unix:/var/run/ganeti/kvm-hypervisor/ctrl/instance2.ganeti.com.monitor,server,nowait -serial unix:/var/run/ganeti/kvm-hypervisor/ctrl/instance2.ganeti.com.serial,server,nowait -usb -display none -uuid c820b0a2-b300-4f8f-a61a-1975786c35f2 -netdev type=tap,id=hotnic-af38ca83-pci-5,fd=8 -device virtio-net-pci,mac=aa:00:00:cd:35:34,id=hotnic-af38ca83-pci-5,bus=pci.0,addr=0x5,netdev=hotnic-af38ca83-pci-5 -qmp unix:/var/run/ganeti/kvm-hypervisor/ctrl/instance2.ganeti.com.qmp,server,nowait -qmp unix:/var/run/ganeti/kvm-hypervisor/ctrl/instance2.ganeti.com.kvmd,server,nowait -device virtio-blk-pci,drive=hotdisk-8771fb57-pci-4,id=hotdisk-8771fb57-pci-4,bus=pci.0,addr=0x4 -drive file=/var/run/ganeti/instance-disks/instance2.ganeti.com:0,format=raw,if=none,cache=none,id=hotdisk-8771fb57-pci-4,bus=0,unit=4 -S
2015-04-17 05:37:53,165: ganeti-noded pid=6702 ERROR Hypervisor error: Failed to start instance instance2.ganeti.com: exited with exit code 1 (kvm: -drive file=/var/run/ganeti/instance-disks/instance2.ganeti.com:0,format=raw,if=none,cache=none,id=hotdisk-8771fb57-pci-4,bus=0,unit=4: could not open disk image /var/run/ganeti/instance-disks/instance2.ganeti.com:0: Invalid argument
)
Traceback (most recent call last):
  File "/usr/local/share/ganeti/2.11/ganeti/backend.py", line 1883, in StartInstance
    hyper.StartInstance(instance, block_devices, startup_paused)
  File "/usr/local/share/ganeti/2.11/ganeti/hypervisor/hv_kvm.py", line 2095, in StartInstance
    self._ExecuteKVMRuntime(instance, kvm_runtime, kvmhelp)
  File "/usr/local/share/ganeti/2.11/ganeti/hypervisor/hv_kvm.py", line 2016, in _ExecuteKVMRuntime
    self._RunKVMCmd(name, kvm_cmd, tapfds)
  File "/usr/local/share/ganeti/2.11/ganeti/hypervisor/hv_kvm.py", line 1831, in _RunKVMCmd
    (name, result.fail_reason, result.output))
HypervisorError: Failed to start instance instance2.ganeti.com: exited with exit code 1 (kvm: -drive file=/var/run/ganeti/instance-disks/instance2.ganeti.com:0,format=raw,if=none,cache=none,id=hotdisk-8771fb57-pci-4,bus=0,unit=4: could not open disk image /var/run/ganeti/instance-disks/instance2.ganeti.com:0: Invalid argument
)
2015-04-17 05:37:53,196: ganeti-noded pid=6702 INFO 192.168.1.101:37123 POST /instance_start HTTP/1.1 200
2015-04-17 05:37:53,291: ganeti-noded pid=6734 INFO 192.168.1.101:37124 POST /hooks_runner HTTP/1.1 200

Originally added on 2015-04-17 09:41:15 +0000 UTC.

[Ganeti-Issues-Migrator]

Hi Petr,
Actually, I use "gnt-cluster upgrate" to upgrade my previous ganeti cluster from 2.10 to 2.11 and then I wanna reproduce Issue 1030 (https://code.google.com/p/ganeti/issues/detail?id=1030). However, I happen to this issue. Now, I would solve this issue and reproduce Issue 1030 (deploy a new 2.11 ganeti cluster on Ubuntu 14.04 to solve Issue 1030) concurrently.

Thanks very much, Gangbiao ;-)

Originally added on 2015-04-17 09:47:18 +0000 UTC.

Mentioned Issue with GoogleCode ID 1030 has been migrated to GitHub with ID #1086.

[Ganeti-Issues-Migrator]

Since I don't have a Gluster setup, I can't directly reproduce it. Do you get the same error for other disk templates (plain, drbd) or just for Gluster? I didn't understand your comment #13, did you try to create a 2.11 cluster from scratch? Does the error happen both when you have a new 2.11 cluster and when you have a 2.10 cluster and upgrade it to 2.11?

Originally added on 2015-04-17 12:35:32 +0000 UTC.

[Ganeti-Issues-Migrator]

Hi petr,

Q: Do you get the same error for other disk templates (plain, drbd) or just for Gluster?
No, I don't have the same error for disk templates plain. it works well for plain.

root@node1:~# sudo gnt-instance add -n node1 -o debootstrap+default -t plain -s 5G instance2
Fri Apr 17 08:57:20 2015  - INFO: Resolved given name 'instance2' to 'instance2.ganeti.com'
Fri Apr 17 08:57:24 2015 * disk 0, size 5.0G
Fri Apr 17 08:57:24 2015 * creating instance disks...
Fri Apr 17 08:57:25 2015 adding instance instance2.ganeti.com to cluster config
Fri Apr 17 08:57:26 2015  - INFO: Waiting for instance instance2.ganeti.com to sync disks
Fri Apr 17 08:57:27 2015  - INFO: Instance instance2.ganeti.com's disks are in sync
Fri Apr 17 08:57:27 2015 * running the instance OS create scripts...
Fri Apr 17 08:57:34 2015 * starting instance...
root@node1:~# gnt-instance list
Instance             Hypervisor OS                  Primary_node     Status  Memory
instance2.ganeti.com kvm        debootstrap+default node1.ganeti.com running   128M

Q: I didn't understand your comment #13, did you try to create a 2.11 cluster from scratch?
Previously, my ganeti cluster version is 2.10. When I wanna solve Issue 1030 (https://code.google.com/p/ganeti/issues/detail?id=1030), I need to setup a cluster with ganeti version 2.11 (So I just compile/install ganeti stable-2.11, then I run command 'gnt-cluster upgrade --to 2.11') because Issue 1030 is based on ganeti cluster version 2.11. Then, I install gluster (Issue 1030 needs) and add an instance with gluster disk template. At last, the error happens.

Does the error happen both when you have a new 2.11 cluster and when you have a 2.10 cluster and upgrade it to 2.11?
No, I have not tried disk-template gluster when the version is 2.10. I try plain disk template and it works well for me(version 2.10 and 2.11). I only try gluster when version is 2.11.


Thanks, Gangbiao 

Originally added on 2015-04-17 13:10:44 +0000 UTC.

Mentioned Issue with GoogleCode ID 1030 has been migrated to GitHub with ID #1086.

[Ganeti-Issues-Migrator]

Just to double check: Did you follow the guide for installing Ganeti with Gluster here
http://docs.ganeti.org/ganeti/2.11/html/install.html#installing-gluster
?

Other than that, I would suggest you to create a new cluster from scratch, that is:

- Remove your current cluster and Ganeti installation completely.
- Install Gluster
- Compile and install Ganeti 2.11 from scratch.
- Create a brand new 2.11 cluster.

If the error persists, I'd suggest to write to the mailing list and ask if someone using Gluster encountered a similar issue.

Originally added on 2015-04-17 13:51:21 +0000 UTC.

[Ganeti-Issues-Migrator]

Hi Petr,

Thanks very much for your suggestions, I am now trying to deploy Ganeti cluster 2.11 (As what you said above) upon Ubuntu 14.04.

Best, Gangbiao

Originally added on 2015-04-18 03:20:41 +0000 UTC.

[Ganeti-Issues-Migrator]

Hi Petr,

After I deploy Ganeti cluster again in Ubuntu 14.04, I could add instances successfully with gluster storage as follows. Now, let me fix Issue 1030 (https://code.google.com/p/ganeti/issues/detail?id=1030). Thanks very much for your help.

root@node2:~# lsb_release -a
No LSB modules are available.
Distributor ID:	Ubuntu
Description:	Ubuntu 14.04 LTS
Release:	14.04
Codename:	trusty

root@node2:~# sudo gnt-instance add -n node2 -o debootstrap+default -t gluster -s 5G instance3
Mon Apr 20 09:58:48 2015  - INFO: Resolved given name 'instance3' to 'instance3.ganeti.com'
Mon Apr 20 09:58:52 2015 * disk 0, size 5.0G
Mon Apr 20 09:58:52 2015 * creating instance disks...
Mon Apr 20 09:58:52 2015 adding instance instance3.ganeti.com to cluster config
Mon Apr 20 09:58:54 2015  - INFO: Waiting for instance instance3.ganeti.com to sync disks
Mon Apr 20 09:58:54 2015  - INFO: Instance instance3.ganeti.com's disks are in sync
Mon Apr 20 09:58:54 2015 * running the instance OS create scripts...
Mon Apr 20 09:59:34 2015 * starting instance...

root@node2:~# gnt-instance list
Instance             Hypervisor OS                  Primary_node     Status  Memory
instance3.ganeti.com kvm        debootstrap+default node2.ganeti.com running   128M

root@node2:~# gnt-instance info instance3
- Instance name: instance3.ganeti.com
  UUID: 4b87b3b0-a7b5-43dc-a707-e295e00e1120
  Serial number: 2
  Creation time: 2015-04-20 09:58:52
  Modification time: 2015-04-20 09:59:34
  State: configured to be up, actual state is up
  Nodes: 
    - primary: node2.ganeti.com
      group: default (UUID 11f06efd-4400-46e9-8f59-585181d2df08)
    - secondaries: 
  Operating system: debootstrap+default
  Operating system parameters: 
  Allocated network port: 11002
  Hypervisor: kvm
  Hypervisor parameters: 
    acpi: default (True)
    boot_order: default (disk)
    cdrom2_image_path: default ()
    cdrom_disk_type: default ()
    cdrom_image_path: default ()
    cpu_cores: default (0)
    cpu_mask: default (all)
    cpu_sockets: default (0)
    cpu_threads: default (0)
    cpu_type: default ()
    disk_cache: default (default)
    disk_type: default (paravirtual)
    floppy_image_path: default ()
    initrd_path: default (/boot/initrd-3-kvmU)
    kernel_args: default (ro)
    kernel_path: default (/boot/vmlinuz-3-kvmU)
    keymap: default ()
    kvm_extra: default ()
    kvm_flag: default ()
    kvm_path: default (/usr/bin/kvm)
    machine_version: default ()
    mem_path: default ()
    migration_downtime: default (30)
    nic_type: default (paravirtual)
    reboot_behavior: default (reboot)
    root_path: default (/dev/vda1)
    security_domain: default ()
    security_model: default (none)
    serial_console: default (True)
    serial_speed: default (38400)
    soundhw: default ()
    spice_bind: default ()
    spice_image_compression: default ()
    spice_ip_version: default (0)
    spice_jpeg_wan_compression: default ()
    spice_password_file: default ()
    spice_playback_compression: default (True)
    spice_streaming_video: default ()
    spice_tls_ciphers: default (HIGH:-DES:-3DES:-EXPORT:-ADH)
    spice_use_tls: default (False)
    spice_use_vdagent: default (True)
    spice_zlib_glz_wan_compression: default ()
    usb_devices: default ()
    usb_mouse: default ()
    use_chroot: default (False)
    use_localtime: default (False)
    user_shutdown: default (False)
    vga: default ()
    vhost_net: default (False)
    vnc_bind_address: default ()
    vnc_password_file: default ()
    vnc_tls: default (False)
    vnc_x509_path: default ()
    vnc_x509_verify: default (False)
    vnet_hdr: default (True)
  Back-end parameters: 
    always_failover: default (False)
    auto_balance: default (True)
    maxmem: default (128)
    memory: default (128)
    minmem: default (128)
    spindle_use: default (1)
    vcpus: default (1)
  NICs: 
    - nic/0: 
      MAC: aa:00:00:ff:5d:0b
      IP: None
      mode: bridged
      link: xen-br0
      vlan: 
      network: None
      UUID: 27481bd3-6a81-43bf-b2ca-e584705c4d6a
      name: None
  Disk template: gluster
  Disks: 
    - disk/0: gluster, size 5.0G
      access mode: rw
      logical_id: ['loop', 'ganeti/4b87b3b0-a7b5-43dc-a707-e295e00e1120.0']
      on primary: /var/run/ganeti/gluster/ganeti/4b87b3b0-a7b5-43dc-a707-e295e00e1120.0 (N/A:N/A)
      name: None
      UUID: 1803964f-f178-4bd7-8330-438fc89eb4da

Originally added on 2015-04-20 02:04:00 +0000 UTC.

Mentioned Issue with GoogleCode ID 1030 has been migrated to GitHub with ID #1086.

[Ganeti-Issues-Migrator]

BTW, ganeti installed version is 2.11.

root@node2:~# gnt-cluster --version
gnt-cluster (ganeti v2.11.7) 2.11.7

Originally added on 2015-04-20 02:21:41 +0000 UTC.

[Ganeti-Issues-Migrator]

As you were able to successfully install instances, I'm closing as "working as intended". If problems arise again, please reopen. Thanks.

Originally added on 2015-04-20 08:42:17 +0000 UTC.

Changed State: Invalid

[Ganeti-Issues-Migrator]

Ack. I am fixing Issue 1030. Thanks very much ;-)

Originally added on 2015-04-20 09:36:15 +0000 UTC.

Mentioned Issue with GoogleCode ID 1030 has been migrated to GitHub with ID #1086.