CEPH Deployments for large VM images failing due to high image conversion time and job timeout

[poonam-agarwal28]

ISSUE TYPE

• Improvement Request

COMPONENT NAME

CEPH, API

CLOUDSTACK VERSION

4.15

CONFIGURATION

Cloudstack 4.15 with KVM Hypervisors and RBD/CEPH as Primary Storage. Secondary Storage on NFS.

OS / ENVIRONMENT

CentOS 7.8

SUMMARY

The current VM deployment process on CEPH as Primary Store runs below command during the deployment to convert the image to raw and also write to the CEPH Pool at the same time. This takes huge amount of time eg below has a 15GB template which took around 3+ Hours. This causes the VM deployment fail due to Job Timeout

If this process is converted to a 2-Step Process as below

Step 1: Image converted to raw
Step 2: Raw image moved to CEPH either using qemu-img convert or rbd as below takes much lower time.

STEPS TO REPRODUCE

Upload a big template with root volume size of 300GB or higher ( NFS as secondary Store )
Deploy a VM with the template with CEPH as the backend primary store.

[root@TEST]# time qemu-img convert -p -O raw /mnt/f268aaee-6090-3fc9-9e9d-41b807bfa8c5.qcow2 rbd:cloudstack/test450:mon_host=VXILAB1-CMON.ceph.local:auth_supported=cephx:id=cloudstack:key=AQDicxxxxxxxxxxxxxxxxFrNxwwxwxw==:rbd_default_format=2:client_mount_timeout=30
    (100.00/100%)

real    114m40.176s
user    10m34.924s
sys     4m3.139s

[root@VXIMUM1-CHVZR-2 mnt]# time qemu-img convert -p -O raw /media/ISETest12.raw rbd:cloudstack/testimage:mon_host=VXILAB1-CMON.ceph.local:auth_supported=cephx:id=cloudstack:key=AQxxxxxxxxxxxxxxrNDqlJueQ==:rbd_default_format=2:client_mount_timeout=30
    (0.00/100%)
    (32.02/100%)

    (100.00/100%)

real    74m1.770s
user    4m40.347s
sys     11m20.207s

[root@VXIMUM1-CHVZR-1 media]# time rbd import ISETest12.raw --dest-pool cloudstack

Importing image: 100% complete...done. 

real    7m58.793s

user    1m30.641s

sys     2m39.082s

[root@VXIMUM1-CHVZR-2 mnt]# time qemu-img convert -p -O raw /media/f268aaee-6090-3fc9-9e9d-41b807bfa8c5.qcow2 /media/ISETest12.raw
    (100.00/100%)

real    9m3.571s
user    4m8.513s
sys     2m41.161s

EXPECTED RESULTS

The expected result is that VM deployment on CEPH should complete in a reasonable time before the job times out.

ACTUAL RESULTS

The actual result is that the first time VM deployment on CEPH takes a long time and eventually times out due to the conversion process being extremely slow.

[rohityadavcloud]

Hi @poonam-agarwal28 thanks for sharing the issue and the benchmarks. Can you share output of:

qemu-img --version
virsh version

Can you also run adn share the benchmark for converting your raw or qcow2 template to rbd with flags -t none -n, something like:

time qemu-img convert -p -n -t none -O raw /media/ISETest12.raw rbd:cloudstack/testimage:mon_host=VXILAB1-CMON.ceph.local:auth_supported=cephx:id=cloudstack:key=AQxxxxxxxxxxxxxxrNDqlJueQ==:rbd_default_format=2:client_mount_timeout=30

[poonam-agarwal28]

@rhtyd Please note below O/P
[root@VXIMUM1-CHVZR-1 media]# time qemu-img convert -p -n -t none -O raw /media/ISETest12.raw rbd:cloudstack/testimage:mon_host=VXILAB1-CMON.ceph.local:auth_supported=cephx:id=cloudstack:key=AQDicAVfG2lpIhAAYu4bnFtZjy04FrNDqlJueQ==:rbd_default_format=2:client_mount_timeout=30
(32.02/100%)
(33.02/100%)
(100.00/100%)
real 78m42.581s
user 1m48.733s
sys 3m59.782s

[rohityadavcloud]

Thanks for sharing @poonam-agarwal28 can you also share your qemu-img and ceph versions?

[wido]

The bigger templates become the longer it takes for them to be converted. It's the way it currently works that we need to use qemu-img to convert the template.

A template of 300GB is a rather large file and can indeed take a lot of time.

You can increase the job timeout if that helps for you. What would a different suggestion be?

[poonam-agarwal28]

@rhtyd
[root@VXIMUM1-CHVZR-1 ~]# qemu-img --version
qemu-img version 2.10.0(qemu-kvm-ev-2.10.0-21.el7_5.7.1)
Copyright (c) 2003-2017 Fabrice Bellard and the QEMU Project developers
[root@VXIMUM1-CHVZR-1 ~]# ceph --version
ceph version 10.2.11 (e4b061b47f07f583c92a050d9e84b1813a35671e)
[root@VXIMUM1-CHVZR-1 ~]# virsh version
Compiled against library: libvirt 4.5.0
Using library: libvirt 4.5.0
Using API: QEMU 4.5.0
Running hypervisor: QEMU 2.10.0

[poonam-agarwal28]

The bigger templates become the longer it takes for them to be converted. It's the way it currently works that we need to use qemu-img to convert the template.

A template of 300GB is a rather large file and can indeed take a lot of time.

You can increase the job timeout if that helps for you. What would a different suggestion be?

@wido
The issue already details proposed solutions , instead to directly converting and writing the image to CEPH Pool , if we convert the image locally and then upload to the secondary store using the rbd client the results are much better

[rohityadavcloud]

Thanks for sharing the details @poonam-agarwal28, we'll use your env details to setup a test env.
Meanwhile, it may be faster for you to confirm and share if the rbd import command can be executed on any KVM host, even when they're not Ceph hosts (osd or monitors)? If this works on KVM hosts irrespective of them not being a ceph host (mon, osd etc) and if you can share if this requires installation of generally available/required dependency (such as ceph-common etc) then we can explore into the implementation of the proposed solution.

[poonam-agarwal28]

@rhtyd
a) rbd import ISETest12.raw --dest-pool cloudstack

This is the rbd import command you can use to directly import the converted image into CEPH

b ) The package ceph-common should provide the rbd binary used for executing the command.

ceph-common-10.2.11-0.el7.x86_64 : Ceph Common
Repo : @ceph-x86-64
Matched from:
Filename : /usr/bin/rbd

[rohityadavcloud]

Thanks for confirming the dependency packages @poonam-agarwal28, but my question was to understand if this requires any dependency on the host where it can be performed. For example, can you run the rbd import command on any KVM host (irrespective whether it's an OSD/mon host or not), for example if you run this on a non-osd/non-mon KVM host do you need to pass any other parameters (such as credentials, path/port/ip address, any config file etc)?

[nvazquez]

Ping @poonam-agarwal28

[poonam-agarwal28]

@nvazquez @rohityadavcloud For rbd import command to run, ceph packages must be installed and the node running the command should be configured as a ceph client ( it should have the admin key with read/write access to connect to the ceph pool ). In my scenario the command was run from one of the OSD nodes

[rohityadavcloud]

Thanks @poonam-agarwal28 for explaining.
As I understand this is a special case but not a bug. Functionally it would take more time for the import/provisioning and this doesn't affect the ceph support.

[weizhouapache]

@nvazquez @rohityadavcloud For rbd import command to run, ceph packages must be installed and the node running the command should be configured as a ceph client ( it should have the admin key with read/write access to connect to the ceph pool ). In my scenario the command was run from one of the OSD nodes

In most cloudstack setup, the OSD nodes are not used as kvm nodes in CloudStack.
good idea, but may not work for most users.

another idea, is it possible to support templates in RAW format in cloudstack ?