New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add quota support for storage backends #3804

Closed
fjakobs opened this Issue Jan 28, 2014 · 50 comments

Comments

Projects
None yet
@fjakobs

fjakobs commented Jan 28, 2014

I understand that the aufs does not support quotas but since the device mapper backend mounts an ext4 partition it should be possible to enable quota support there. If the loopback device with the ext4 partition is mounted with the proper quota options I think it would be possible to use regular linux quota tools inside of a docker container.

@unclejack

This comment has been minimized.

Show comment
Hide comment
@unclejack

unclejack Jan 28, 2014

Contributor

This feature is going to be added at some point, but not right away. It's a bit more difficult to add this functionality right now because a lot of chunks of code are moving from one place to another. After this work is done, it should be much easier to implement this functionality.

Please keep in mind that quota support can't be added as a hack to devicemapper, it has to be implemented for as many storage backends as possible, so it has to be implemented in a way which makes it easy to add quota support for other storage backends.

While I do understand that you need quota support for the container itself, wouldn't using bind mounts with quota help until quota support gets added to Docker?

Contributor

unclejack commented Jan 28, 2014

This feature is going to be added at some point, but not right away. It's a bit more difficult to add this functionality right now because a lot of chunks of code are moving from one place to another. After this work is done, it should be much easier to implement this functionality.

Please keep in mind that quota support can't be added as a hack to devicemapper, it has to be implemented for as many storage backends as possible, so it has to be implemented in a way which makes it easy to add quota support for other storage backends.

While I do understand that you need quota support for the container itself, wouldn't using bind mounts with quota help until quota support gets added to Docker?

@fjakobs

This comment has been minimized.

Show comment
Hide comment
@fjakobs

fjakobs Jan 28, 2014

Right now I'm using the fixed ext4 image size as a quota replacement but for this to work well I depend on #3548 being implemented.

If I wanted to hack quota support into my docker clone, where could I change the mount arguments?

fjakobs commented Jan 28, 2014

Right now I'm using the fixed ext4 image size as a quota replacement but for this to work well I depend on #3548 being implemented.

If I wanted to hack quota support into my docker clone, where could I change the mount arguments?

@unclejack

This comment has been minimized.

Show comment
Hide comment
@unclejack

unclejack Feb 26, 2014

Contributor

I've updated the title to make it more generic. We have more backends now and it wouldn't make sense to "hack" quota support into a specific backend.

Contributor

unclejack commented Feb 26, 2014

I've updated the title to make it more generic. We have more backends now and it wouldn't make sense to "hack" quota support into a specific backend.

@itsnotvalid

This comment has been minimized.

Show comment
Hide comment
@itsnotvalid

itsnotvalid May 24, 2014

Is there any updates to this problem?

itsnotvalid commented May 24, 2014

Is there any updates to this problem?

@mtasic85

This comment has been minimized.

Show comment
Hide comment
@mtasic85

mtasic85 Jun 7, 2014

Is there any way for docker to use BTRFS instead of AUFS?

mtasic85 commented Jun 7, 2014

Is there any way for docker to use BTRFS instead of AUFS?

@crosbymichael

This comment has been minimized.

Show comment
Hide comment
@crosbymichael

crosbymichael Jun 7, 2014

Contributor

@mtasic85 yes, format your dive on btrfs and docker will use it automatically. or use the flag when starting the daemon docker -d -s btrfs

Contributor

crosbymichael commented Jun 7, 2014

@mtasic85 yes, format your dive on btrfs and docker will use it automatically. or use the flag when starting the daemon docker -d -s btrfs

@mtasic85

This comment has been minimized.

Show comment
Hide comment
@mtasic85

mtasic85 Jun 7, 2014

@crosbymichael Thanks. Is there way to know or specify CONTAINER ID before it is created? I'm asking because I want to create btrfs subvolume with quota and mount it to containers working dir.

mtasic85 commented Jun 7, 2014

@crosbymichael Thanks. Is there way to know or specify CONTAINER ID before it is created? I'm asking because I want to create btrfs subvolume with quota and mount it to containers working dir.

@cscetbon

This comment has been minimized.

Show comment
Hide comment
@cscetbon

cscetbon Aug 11, 2014

Is there any progress concerning quota support ? I don't see any news since @itsnotvalid question

cscetbon commented Aug 11, 2014

Is there any progress concerning quota support ? I don't see any news since @itsnotvalid question

@sherter

This comment has been minimized.

Show comment
Hide comment
@sherter

sherter Aug 12, 2014

Since BTRFS support was added, you should be able to manually add quotas (see btrfs wiki page). However, I didn't try it myself.

sherter commented Aug 12, 2014

Since BTRFS support was added, you should be able to manually add quotas (see btrfs wiki page). However, I didn't try it myself.

@cscetbon

This comment has been minimized.

Show comment
Hide comment
@cscetbon

cscetbon Aug 12, 2014

yeah but I guess even device-mapper should support as it uses ext4/xfs as a filesystem. What I was looking for was a way to tell docker to apply it on the fly without having to find sub volumes (for Btrfs) after the creation of a new container

cscetbon commented Aug 12, 2014

yeah but I guess even device-mapper should support as it uses ext4/xfs as a filesystem. What I was looking for was a way to tell docker to apply it on the fly without having to find sub volumes (for Btrfs) after the creation of a new container

@majst01

This comment has been minimized.

Show comment
Hide comment
@majst01

majst01 Nov 23, 2014

I implemented btrfs quota support here:

majst01/docker@docker:master...master
this is my first golang experience so there might be some stupidities.
This is not ready for a pull request because i am currently talking to original author of the btrfs code for a review. This code is kinda working so far but with limited testing.

So review and comments welcome.

For the long term i would like to have per container quotas. It should be possible to limit disk space the same way cpushares and memory is limited. But this would be a start.

majst01 commented Nov 23, 2014

I implemented btrfs quota support here:

majst01/docker@docker:master...master
this is my first golang experience so there might be some stupidities.
This is not ready for a pull request because i am currently talking to original author of the btrfs code for a review. This code is kinda working so far but with limited testing.

So review and comments welcome.

For the long term i would like to have per container quotas. It should be possible to limit disk space the same way cpushares and memory is limited. But this would be a start.

@fza

This comment has been minimized.

Show comment
Hide comment
@fza

fza Nov 23, 2014

@majst01 .. +1, so much!!

fza commented Nov 23, 2014

@majst01 .. +1, so much!!

@nfnty

This comment has been minimized.

Show comment
Hide comment
@nfnty

nfnty May 17, 2015

@majst01 Any updates on the btrfs quota support? Very interested in this feature.

nfnty commented May 17, 2015

@majst01 Any updates on the btrfs quota support? Very interested in this feature.

@blackfader

This comment has been minimized.

Show comment
Hide comment
@blackfader

blackfader May 28, 2015

@majst01 Any updates on the btrfs quota support? Very interested in this feature. +1

blackfader commented May 28, 2015

@majst01 Any updates on the btrfs quota support? Very interested in this feature. +1

@denghongcai

This comment has been minimized.

Show comment
Hide comment
@denghongcai

denghongcai Jul 23, 2015

Any updates? Seems nothing changed at all?

denghongcai commented Jul 23, 2015

Any updates? Seems nothing changed at all?

@SunWeicheng0001

This comment has been minimized.

Show comment
Hide comment
@SunWeicheng0001

SunWeicheng0001 commented Aug 3, 2015

Any updates?

1 similar comment
@xiaods

This comment has been minimized.

Show comment
Hide comment
@xiaods

xiaods Aug 3, 2015

Contributor

Any updates?

Contributor

xiaods commented Aug 3, 2015

Any updates?

@yijia2413

This comment has been minimized.

Show comment
Hide comment
@yijia2413

yijia2413 commented Aug 24, 2015

ok now ?

@GordonTheTurtle

This comment has been minimized.

Show comment
Hide comment
@GordonTheTurtle

GordonTheTurtle Aug 28, 2015

USER POLL

The best way to get notified when there are changes in this discussion is by clicking the Subscribe button in the top right.

The people listed below have appreciated your meaningfull discussion with a random +1:

@rbugajewski

GordonTheTurtle commented Aug 28, 2015

USER POLL

The best way to get notified when there are changes in this discussion is by clicking the Subscribe button in the top right.

The people listed below have appreciated your meaningfull discussion with a random +1:

@rbugajewski

@cpuguy83

This comment has been minimized.

Show comment
Hide comment
@cpuguy83

cpuguy83 Aug 28, 2015

Contributor

Docker 1.8 supports blkio-weight.
#15879 adds blkio iops per device
#14466 adds read/write bps per device
#13959 adds blkio weight per device

Contributor

cpuguy83 commented Aug 28, 2015

Docker 1.8 supports blkio-weight.
#15879 adds blkio iops per device
#14466 adds read/write bps per device
#13959 adds blkio weight per device

@denghongcai

This comment has been minimized.

Show comment
Hide comment
@denghongcai

denghongcai Aug 29, 2015

@cpuguy83 features you listed here is not related with disk space quota...

denghongcai commented Aug 29, 2015

@cpuguy83 features you listed here is not related with disk space quota...

@cpuguy83

This comment has been minimized.

Show comment
Hide comment
@cpuguy83

cpuguy83 Aug 29, 2015

Contributor

They are not "disk quota support", but they are related to the topic.

Contributor

cpuguy83 commented Aug 29, 2015

They are not "disk quota support", but they are related to the topic.

@jessfraz jessfraz removed the kind/proposal label Sep 8, 2015

@fahimeh2010

This comment has been minimized.

Show comment
Hide comment
@fahimeh2010

fahimeh2010 Feb 3, 2016

any update in disk quota (not daemon-wide limitation)?

fahimeh2010 commented Feb 3, 2016

any update in disk quota (not daemon-wide limitation)?

@zhuguihua

This comment has been minimized.

Show comment
Hide comment
@zhuguihua

zhuguihua Feb 4, 2016

Contributor

@fahimeh2010 I sent a PR (#19651)about btrfs disk quota, but it must have more discussion on ux. You can go to see it, and welcome more suggestions.

Contributor

zhuguihua commented Feb 4, 2016

@fahimeh2010 I sent a PR (#19651)about btrfs disk quota, but it must have more discussion on ux. You can go to see it, and welcome more suggestions.

@denghongcai

This comment has been minimized.

Show comment
Hide comment
@denghongcai

denghongcai Feb 4, 2016

@zhuguihua well done! hope it will merge into mainline soon!

denghongcai commented Feb 4, 2016

@zhuguihua well done! hope it will merge into mainline soon!

@arkkanoid

This comment has been minimized.

Show comment
Hide comment
@arkkanoid

arkkanoid Mar 16, 2016

So, it's not possible to use Disk Quotas inside a contanier for now?

arkkanoid commented Mar 16, 2016

So, it's not possible to use Disk Quotas inside a contanier for now?

@cpuguy83

This comment has been minimized.

Show comment
Hide comment
@cpuguy83

cpuguy83 Mar 16, 2016

Contributor

@arkkanoid correct

Contributor

cpuguy83 commented Mar 16, 2016

@arkkanoid correct

@arkkanoid

This comment has been minimized.

Show comment
Hide comment
@arkkanoid

arkkanoid Mar 16, 2016

And is there another way to limit the disk size for an specific user inside docker container?

arkkanoid commented Mar 16, 2016

And is there another way to limit the disk size for an specific user inside docker container?

@vote539

This comment has been minimized.

Show comment
Hide comment
@vote539

vote539 Mar 16, 2016

The solution I ended up using, which is far from perfect but gets the job done, is to make a virtual, limited-size file system loop-mounted on the host, and then mount it as a volume on the container. This page has a good tutorial on setting up limited-size filesystems (the quota reporting mechanisms are optional):

http://souptonuts.sourceforge.net/quota_tutorial.html

One downside of this approach is that you need to have root permissions in order to mount the loop filesystem. (That's not really a problem though because you also need root permissions to control Docker. 😉 ) The other downside is that only the one directory is going to be quota'd. If the user inside the container tries to write a file outside of their designated directory, they can still take up space on the host filesystem. Depending on your setup, read/write permissions inside the container could help deal with that potential problem.

This obviously works only if you have control of the host machine.

vote539 commented Mar 16, 2016

The solution I ended up using, which is far from perfect but gets the job done, is to make a virtual, limited-size file system loop-mounted on the host, and then mount it as a volume on the container. This page has a good tutorial on setting up limited-size filesystems (the quota reporting mechanisms are optional):

http://souptonuts.sourceforge.net/quota_tutorial.html

One downside of this approach is that you need to have root permissions in order to mount the loop filesystem. (That's not really a problem though because you also need root permissions to control Docker. 😉 ) The other downside is that only the one directory is going to be quota'd. If the user inside the container tries to write a file outside of their designated directory, they can still take up space on the host filesystem. Depending on your setup, read/write permissions inside the container could help deal with that potential problem.

This obviously works only if you have control of the host machine.

@tjdett

This comment has been minimized.

Show comment
Hide comment
@tjdett

tjdett Mar 16, 2016

Contributor

@vote539 This is essentially what the devicemapper storage backend does when used in loopback mode. To quote @jasonbrooks from Project Atomic:

Friends Don't Let Friends Run Docker on Loopback in Production

If this is just for dev, then stop reading now. You have a solution that works, just go with it.

If it's for prod, then you will probably get better performance from using:

  • LVM logical volumes, or
  • LVM thin pool volumes, or
  • normal disk partitions, or
  • a single BTRFS/ZFS filesystem based on one of the above, using subvolumes/datasets to limit directory space.

You should also consider if devicemapper would be an easier solution depending on the environment you're running Docker in.

Contributor

tjdett commented Mar 16, 2016

@vote539 This is essentially what the devicemapper storage backend does when used in loopback mode. To quote @jasonbrooks from Project Atomic:

Friends Don't Let Friends Run Docker on Loopback in Production

If this is just for dev, then stop reading now. You have a solution that works, just go with it.

If it's for prod, then you will probably get better performance from using:

  • LVM logical volumes, or
  • LVM thin pool volumes, or
  • normal disk partitions, or
  • a single BTRFS/ZFS filesystem based on one of the above, using subvolumes/datasets to limit directory space.

You should also consider if devicemapper would be an easier solution depending on the environment you're running Docker in.

@gegere

This comment has been minimized.

Show comment
Hide comment
@gegere

gegere Apr 2, 2016

Basically the solution for the Docker disk limit stuck at 20gb would be to share the growing file directory to the host system?

https://docs.docker.com/compose/compose-file/#volumes-volume-driver

    volumes:
   - "~/SITES/docker/mysql:/var/lib/mysql"

gegere commented Apr 2, 2016

Basically the solution for the Docker disk limit stuck at 20gb would be to share the growing file directory to the host system?

https://docs.docker.com/compose/compose-file/#volumes-volume-driver

    volumes:
   - "~/SITES/docker/mysql:/var/lib/mysql"
@tjdett

This comment has been minimized.

Show comment
Hide comment
@tjdett

tjdett Apr 4, 2016

Contributor

@gegre You should store your data in a volume. That part you've definitely got right. The rest isn't good practice, and seems completely unrelated to the topic of this issue.

AFAIK, the only Docker storage backend which limits container disk by default is devicemapper. You can configure the daemon to adjust that size, so it's not "stuck" in any real sense.

Volumes use flat storage rather than layered storage, and use a completely different backend which can be extended via plugins. Simply making /var/lib/mysql a volume would ordinarily remove the 20GB limit from devicemapper, and also improve performance.

You do not need to map it to the host filesystem as you're suggesting - this is a pre-1.9 way of doing things, and was discouraged even then in favour of data only containers. Now that named volumes exist, you should use them for your data instead.

Contributor

tjdett commented Apr 4, 2016

@gegre You should store your data in a volume. That part you've definitely got right. The rest isn't good practice, and seems completely unrelated to the topic of this issue.

AFAIK, the only Docker storage backend which limits container disk by default is devicemapper. You can configure the daemon to adjust that size, so it's not "stuck" in any real sense.

Volumes use flat storage rather than layered storage, and use a completely different backend which can be extended via plugins. Simply making /var/lib/mysql a volume would ordinarily remove the 20GB limit from devicemapper, and also improve performance.

You do not need to map it to the host filesystem as you're suggesting - this is a pre-1.9 way of doing things, and was discouraged even then in favour of data only containers. Now that named volumes exist, you should use them for your data instead.

@cpuguy83

This comment has been minimized.

Show comment
Hide comment
@cpuguy83

cpuguy83 Jun 27, 2016

Contributor

FYI, quotas are now supported in devicemapper, zfs, and btrfs.
Probably won't ever be supported in aufs or overlay.

Contributor

cpuguy83 commented Jun 27, 2016

FYI, quotas are now supported in devicemapper, zfs, and btrfs.
Probably won't ever be supported in aufs or overlay.

@mYmNeo

This comment has been minimized.

Show comment
Hide comment
@mYmNeo

mYmNeo Jun 28, 2016

Contributor

Although disk quota is supported by ext4, but at most of time the user running in the container is root, it is not limited by disk quota

Contributor

mYmNeo commented Jun 28, 2016

Although disk quota is supported by ext4, but at most of time the user running in the container is root, it is not limited by disk quota

@treeder

This comment has been minimized.

Show comment
Hide comment
@treeder

treeder Jun 28, 2016

@cpuguy83 do you have links to any information on this?

treeder commented Jun 28, 2016

@cpuguy83 do you have links to any information on this?

@cpuguy83

This comment has been minimized.

Show comment
Hide comment
@cpuguy83

cpuguy83 Jun 28, 2016

Contributor

@treeder Looks like docs are not ready for that yet, though there is some mention in docs/reference/commandline/dockerd.md

Basically, can set --storage-opt size=1G at either the daemon level or at runtime.

Contributor

cpuguy83 commented Jun 28, 2016

@treeder Looks like docs are not ready for that yet, though there is some mention in docs/reference/commandline/dockerd.md

Basically, can set --storage-opt size=1G at either the daemon level or at runtime.

@tobegit3hub

This comment has been minimized.

Show comment
Hide comment
@tobegit3hub

tobegit3hub Aug 25, 2016

This is closed but I'm not sure if it works for our requirements.

Are there issues about supporting ext4, aufs or overlayfs? Which version of docker daemon has merged the patch and we can start to use?

tobegit3hub commented Aug 25, 2016

This is closed but I'm not sure if it works for our requirements.

Are there issues about supporting ext4, aufs or overlayfs? Which version of docker daemon has merged the patch and we can start to use?

@ghost

This comment has been minimized.

Show comment
Hide comment
@ghost

ghost Aug 25, 2016

I think it isn’t working yet. I haven’t read that this feature already
implemented.

tobe notifications@github.com ezt írta (időpont: 2016. aug. 25., Cs 5:50):

This is closed but I'm not sure if it works for our requirements.

Does this work for btrfs, devicemapper or any storage backend? Which
version of docker daemon has merged the patch and we can start to use?


You are receiving this because you commented.
Reply to this email directly, view it on GitHub
#3804 (comment), or mute
the thread
https://github.com/notifications/unsubscribe-auth/ANfEFcrBx5A56cxAMWcl5mI9ujFZxjjTks5qjRETgaJpZM4BdHcS
.

ghost commented Aug 25, 2016

I think it isn’t working yet. I haven’t read that this feature already
implemented.

tobe notifications@github.com ezt írta (időpont: 2016. aug. 25., Cs 5:50):

This is closed but I'm not sure if it works for our requirements.

Does this work for btrfs, devicemapper or any storage backend? Which
version of docker daemon has merged the patch and we can start to use?


You are receiving this because you commented.
Reply to this email directly, view it on GitHub
#3804 (comment), or mute
the thread
https://github.com/notifications/unsubscribe-auth/ANfEFcrBx5A56cxAMWcl5mI9ujFZxjjTks5qjRETgaJpZM4BdHcS
.

@AkihiroSuda

This comment has been minimized.

Show comment
Hide comment
@AkihiroSuda

AkihiroSuda Aug 25, 2016

Member

@tobegit3hub @kovacsa91
PRs are available:
XFS+overlay2: #24771
XFS+overlay: #24807

Member

AkihiroSuda commented Aug 25, 2016

@tobegit3hub @kovacsa91
PRs are available:
XFS+overlay2: #24771
XFS+overlay: #24807

@tobegit3hub

This comment has been minimized.

Show comment
Hide comment
@tobegit3hub

tobegit3hub Aug 25, 2016

Thanks @kovacsa91 and @AkihiroSuda . That's clear to me now 😃

--- Update ---

Refer to docs/reference/commandline/run.md, we can use docker create -it --storage-opt size=120G fedora /bin/bash now.

And after diving into the code, the devicemapper has been implemented in b16decf . And maybe btrfs, windowsfilter, and zfs have been added according to the doc. Please correct me if I'm wrong.

tobegit3hub commented Aug 25, 2016

Thanks @kovacsa91 and @AkihiroSuda . That's clear to me now 😃

--- Update ---

Refer to docs/reference/commandline/run.md, we can use docker create -it --storage-opt size=120G fedora /bin/bash now.

And after diving into the code, the devicemapper has been implemented in b16decf . And maybe btrfs, windowsfilter, and zfs have been added according to the doc. Please correct me if I'm wrong.

@bklau

This comment has been minimized.

Show comment
Hide comment
@bklau

bklau Jun 14, 2017

@AkihiroSuda What is --storage-opt size=xxx means for the various storage drivers?. Is 'xxx' means just the max size of the writable(CoW) layer in the container or the sum total of the container base image layers plus the writable(CoW) layer?. It's not clear to me. Also, what this "disk quota" field in the container.HostConfig?. How it is related to --storage-opt size=XXX ?

bklau commented Jun 14, 2017

@AkihiroSuda What is --storage-opt size=xxx means for the various storage drivers?. Is 'xxx' means just the max size of the writable(CoW) layer in the container or the sum total of the container base image layers plus the writable(CoW) layer?. It's not clear to me. Also, what this "disk quota" field in the container.HostConfig?. How it is related to --storage-opt size=XXX ?

@timchenxiaoyu

This comment has been minimized.

Show comment
Hide comment
@timchenxiaoyu

timchenxiaoyu Sep 25, 2017

if I use devicemapper /aufs /overlay ...,all support storage-opt size?

timchenxiaoyu commented Sep 25, 2017

if I use devicemapper /aufs /overlay ...,all support storage-opt size?

@Soullivaneuh

This comment has been minimized.

Show comment
Hide comment
@Soullivaneuh

Soullivaneuh Oct 14, 2017

This is not supported for aufs either vfs:

$ docker create -it --storage-opt size=5G debian /bin/bash
Error response from daemon: --storage-opt is not supported for aufs
(reverse-i-search)` sh': git^Chow HEAD
/ # docker create -it --storage-opt size=5G debian /bin/bash
Error response from daemon: --storage-opt is not supported for vfs

The issue should be re-opened.

Soullivaneuh commented Oct 14, 2017

This is not supported for aufs either vfs:

$ docker create -it --storage-opt size=5G debian /bin/bash
Error response from daemon: --storage-opt is not supported for aufs
(reverse-i-search)` sh': git^Chow HEAD
/ # docker create -it --storage-opt size=5G debian /bin/bash
Error response from daemon: --storage-opt is not supported for vfs

The issue should be re-opened.

@katsar0v

This comment has been minimized.

Show comment
Hide comment
@katsar0v

katsar0v Oct 17, 2018

Any updates on this issue?

katsar0v commented Oct 17, 2018

Any updates on this issue?

@AkihiroSuda

This comment has been minimized.

Show comment
Hide comment
@AkihiroSuda
Member

AkihiroSuda commented Oct 17, 2018

@katsar0v #24771

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment