DRBD Volume above 4TB not working

[Ganeti-Issues-Migrator]

Originally reported of Google Code with ID 256.

What software version are you running? Please provide the output of "gnt-
cluster --version" and "gnt-cluster version".

gnt-cluster (ganeti v2.5.2) 2.5.2
Software version: 2.5.2
Internode protocol: 2050000
Configuration format: 2050000
OS api version: 20
Export interface: 0

<b>What distribution are you using?</b>

Ubuntu 12.04, DRBD 8.3.11 (api:88)

<b>What steps will reproduce the problem?</b>
1. Create a VM with disk type DRBD above 4TB

<b>What is the expected output? What do you see instead?</b>

Instance creation fails with "/var/log/ganeti/node-daemon.log:2012-08-08 16:27:44,552: ganeti-noded pid=4148 ERROR Can't assemble device after creation, unusual event: drbd2: can't attach local disk: /dev/drbd2: Failure: (111) Low.dev. smaller than requested DRBD-dev. size."


<b>Please provide any additional information below.</b>

Even if you specify more than 128MB as DRBD meta volume size it fails. This is due DRBD external meta disk handling. When it's created as an index it is always fixed to 128MB and results in a maximum volume size somewhere below 4TB.

Creating a flexible meta disk without an index or using internal meta-data would solve this issue. There is no need for indexed meta-disks as Ganeti is creating one meta-disk lvm volume for each VM anyway.

Here is a Florian Haas' statement from Linbit (the DRBD creators) considering internal and external meta-disks.

http://lists.linbit.com/pipermail/drbd-user/2008-June/009628.html

best regards,

Adrian

Originally added on 2012-08-09 09:18:29 +0000 UTC.

[Ganeti-Issues-Migrator]

Same issue on Ganeti 2.6.x on Gentoo Linux.

This issue could have a solution if we have a way to tell ganeti to use drbd with internal meetadata. That shoud remove this 4TB limitation at disk creation time.

 

Originally added on 2013-02-02 14:18:33 +0000 UTC.

[Ganeti-Issues-Migrator]

The problem with internal metadata is that it makes disk resizes and conversion impossible, since now the drbd volume is not a "raw" image. This is the reason why we did not implement internal metadata so far.

It could be possible to use internal metadata, but then this would have to be tracked, and all the above operation disallowed. Hmm…

Originally added on 2013-02-04 08:42:00 +0000 UTC.

[Ganeti-Issues-Migrator]

Which would be possible if we used a disk parameter and documented that enabling it loses support for resize and conversion operations. (of course code-wise this shouldn't be hardcoded then, but asked by cmdlib to each drbd device, to avoid a layering violation).

I think if someone would contribute this patch we could accept it, what do you think?
Thanks,

Guido

Originally added on 2013-02-15 14:48:46 +0000 UTC.

[Ganeti-Issues-Migrator]

Yes, sure. But I don't think it'll be a trivial patch…

Originally added on 2013-02-15 14:54:46 +0000 UTC.

[Ganeti-Issues-Migrator]

As discussed yesterday on IRC an alternative would be to use flexible external metadata, which allows a metadata size proportional to the data size, instead of exactly 128M, and thus scalable over 4T. What do you think?

Originally added on 2013-03-23 07:29:02 +0000 UTC.

[Ganeti-Issues-Migrator]

-- Empty comment --

Originally added on 2013-05-03 09:27:54 +0000 UTC.

Changed State: Accepted
Added to Milestone: Release2.9

[Ganeti-Issues-Migrator]

Ganeti uses for DRBD 8.4 (supported as of 2.9) flexible external metadata per default.
It does not, however, compute the metadata size based on the actual disk size. So there is still work to do in order to support disks > 4TB.

I'm unassigning myself from this bug, as I'm not actively working on it.

Originally added on 2013-06-28 11:47:48 +0000 UTC.

[Ganeti-Issues-Migrator]

Move non-critical bugs scheduled for 2.8 or 2.9 to 2.11, as in those versions only critical bug fixes will be integrated.

Originally added on 2013-10-30 09:48:16 +0000 UTC.

Added to Milestone: Release2.11
Removed from Milestone: Release2.9

[Ganeti-Issues-Migrator]

is there any current workaround for this? i ran into this issue today. :)

Originally added on 2014-02-06 07:36:12 +0000 UTC.

[Ganeti-Issues-Migrator]

A quick and dirty workaround is to change the value of the DRBD_META_SIZE constant in constants.py (or _constants.py on newer Ganeti versions) to something above 128MB (double it for 8TB, for example). Make sure to change it to the same value on all nodes in your cluster.

While this would allow you to create disks bigger than 4TB, I'm not quite sure which problems you could run into with existing disks. Make sure to test such a workaround before using it in production.

Originally added on 2014-02-06 09:12:44 +0000 UTC.

[Ganeti-Issues-Migrator]

Another dirty work-around we use is to create several virtual disks of 4TB each for the same instance and merge them in the instance with LVM.

It's been working great so far with 12TB+ devices without any modification in the Ganeti code.

We are still waiting for this issue to be handled as >4TB devices are getting more and more common those days.

Originally added on 2014-02-06 09:18:30 +0000 UTC.

[Ganeti-Issues-Migrator]

I'm changing the DRBD_META_SIZE value of /usr/share/ganeti/2.11/ganeti/constants.py each time I need a volume > 4TB and reverse the change after.

It's really annoying. I think the simple way is to calculate DRBD_META_SIZE based on the disk size to be created with a minimum of 128 MB like actually fixed.

if DRBD_DISK_SIZE > 4TB then DRBD_META_SIZE = DRBD_DISK_SIZE/32

This way we could have any size of volume.

I wonder even if we can use DRBD_DISK_SIZE/32 for any size of drbd disk to get a low space overhead in case of little volume usage.

Originally added on 2014-12-03 17:48:30 +0000 UTC.