Skip to content
This repository has been archived by the owner on Apr 19, 2019. It is now read-only.

Cinder volume from image can cause a full fs #166

Closed
cloudnull opened this issue Sep 24, 2014 · 7 comments
Closed

Cinder volume from image can cause a full fs #166

cloudnull opened this issue Sep 24, 2014 · 7 comments

Comments

@cloudnull
Copy link
Member

In order to ensure that cinder is available at all times and does not suffer from 100% full disks due to building volumes from images the path /mnt/var/lib/cinder/conversion/ needs to be bind mounted to the host or the cinder_volume container needs to be built with a larger default LV size.

@jameswthorne
Copy link

Some additional info: https://gist.github.com/jameswthorne/78dc7fab6a83598aff85

@b3rn4rd0s
Copy link

We will need some documentation if this does not get fixed.

@b3rn4rd0s b3rn4rd0s modified the milestone: Sprint 13 Nov 13, 2014
@hughsaunders
Copy link
Contributor

From @jameswthorne's logs:

2014-10-24 12:26:22.901 23976 TRACE oslo.messaging.rpc.dispatcher Command: sudo cinder-rootwrap /etc/cinder/rootwrap.conf dd if=/dev/zero of=/dev/mapper/cinder--volumes-volume--d03e1fce--a2f4--466f--aa7b--a408f6c08c9f count=10240 bs=1M conv=fdatasync
2014-10-24 12:26:22.901 23976 TRACE oslo.messaging.rpc.dispatcher Exit code: 1
2014-10-24 12:26:22.901 23976 TRACE oslo.messaging.rpc.dispatcher Stdout: ''
2014-10-24 12:26:22.901 23976 TRACE oslo.messaging.rpc.dispatcher Stderr: "/bin/dd: error writing '/dev/mapper/cinder--volumes-volume--d03e1fce--a2f4--466f--aa7b--a408f6c08c9f': No space left on device\n204
9+0 records in\n2048+0 records out\n2147483648 bytes (2.1 GB) copied, 2.57972 s, 832 MB/s\n"

This failed DDing directly to LV, I'm not sure how adding space to /mnt/var/lib/cinder/conversion will help? Question is, why was DD only able to write just over 2GB to the device while LVS shows that the LV size is 10GB?

@hughsaunders hughsaunders self-assigned this Nov 21, 2014
@hughsaunders
Copy link
Contributor

I am going to discount the DD problem in @jameswthorne's post as I don't have enough information to determine the cause. This issue will now focus on adding space for cinder to convert images. James, please raise another issue if the DD problem persists.

hughsaunders added a commit to hughsaunders/ansible-lxc-rpc that referenced this issue Nov 21, 2014
Cinder volume needs temporary space to convert images. This patch
enlarges the default size of the cinder volume containe to 105GB. It
also exposes the size in the user configuration. This way a user can
increase the size if they anticipate converting larger images, or reduce
it if they don't require image conversion.

Related: rcbops#166
hughsaunders added a commit to hughsaunders/ansible-lxc-rpc that referenced this issue Nov 21, 2014
Cinder volume needs temporary space to convert images. This patch
enlarges the default size of the cinder volume container to 105GB. It
also exposes the size in the user configuration. This way a user can
increase the size if they anticipate converting larger images, or reduce
it if they don't require image conversion.

Related: rcbops#166
@cloudnull
Copy link
Member Author

@hughsaunders - the changes you have in your private repo seem sensible, at commit "22845a6" do you think you could PR them into the master for this issue or do you think we should bind mount back to the host?

@hughsaunders
Copy link
Contributor

@cloudnull my commit doesn't work due to ansible/ansible#7844 so I need to refactor to avoid that issue :(

@hughsaunders
Copy link
Contributor

Still working on this, but bumped into issue ansible/ansible#8705 (see: https://gist.github.com/hughsaunders/4082a3732b3bf807cec3#file-gistfile2-txt-L8). This was mostly due to me running ansible 1.7.2

hughsaunders added a commit to hughsaunders/ansible-lxc-rpc that referenced this issue Nov 28, 2014
Cinder volume needs temporary space to convert images. This patch
enlarges the default size of the cinder volume container to 105GB. It
also exposes the size in the user configuration. This way a user can
increase the size if they anticipate converting larger images, or reduce
it if they don't require image conversion.

Related: rcbops#166
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Projects
None yet
Development

No branches or pull requests

5 participants