Docker Run Error with –storage-opt and Overlay2 on XFS with pquota

[rajat709]

Description

According to the documentation at docker run | Docker Docs, --storage-opt should work with overlay2, and the backing filesystem is XFS mounted with pquota. I’ve been trying for a while, but I keep encountering this error.

docker: Error response from daemon: --storage-opt is supported only for overlay over xfs with 'pquota' mount option.
See 'docker run --help'.

Reproduce

docker run -d it --storage-opt size=10G ubuntu

Expected behavior

No response

docker version

ubuntu@ip-172-31-15-152:~$ docker version
Client:
 Version:           24.0.5
 API version:       1.43
 Go version:        go1.20.3
 Git commit:        24.0.5-0ubuntu1~22.04.1
 Built:             Mon Aug 21 19:50:14 2023
 OS/Arch:           linux/amd64
 Context:           default

Server:
 Engine:
  Version:          24.0.5
  API version:      1.43 (minimum version 1.12)
  Go version:       go1.20.3
  Git commit:       24.0.5-0ubuntu1~22.04.1
  Built:            Mon Aug 21 19:50:14 2023
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          1.7.2
  GitCommit:
 runc:
  Version:          1.1.7-0ubuntu1~22.04.1
  GitCommit:
 docker-init:
  Version:          0.19.0
  GitCommit:

docker info

ubuntu@ip-172-31-15-152:~$ docker info
Client:
 Version:    24.0.5
 Context:    default
 Debug Mode: false

Server:
 Containers: 1
  Running: 1
  Paused: 0
  Stopped: 0
 Images: 2
 Server Version: 24.0.5
 Storage Driver: overlay2
  Backing Filesystem: extfs
  Supports d_type: true
  Using metacopy: false
  Native Overlay Diff: true
  userxattr: false
 Logging Driver: json-file
 Cgroup Driver: systemd
 Cgroup Version: 2
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
 Swarm: inactive
 Runtimes: runc io.containerd.runc.v2
 Default Runtime: runc
 Init Binary: docker-init
 containerd version:
 runc version:
 init version:
 Security Options:
  apparmor
  seccomp
   Profile: builtin
  cgroupns
 Kernel Version: 6.2.0-1012-aws
 Operating System: Ubuntu 22.04.3 LTS
 OSType: linux
 Architecture: x86_64
 CPUs: 4
 Total Memory: 15.32GiB
 Name: ip-172-31-15-152
 ID: 6b042b0f-e260-4b24-a9d3-1ed982cc6c53
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 Experimental: false
 Insecure Registries:
  127.0.0.0/8
 Live Restore Enabled: false

Additional Info

No response

[thaJeztah]

Thanks for reporting; from your docker info output, it looks like your system is running extfs, not xfs

Backing Filesystem: extfs

Project quotas are currently only supported on xfs (as the error-message mentions), so this should currently be the expected behavior. There's a feature request for this, and there was a pull request, which got abandoned, so it was not implemented so far;

• Feature request: add extfs quota support for overlay(2) #29364
• Implement filesystem quota for overlay2 over ext4 if it support #34364

We should also look if we want to implement this as part of the graph-drivers, because we're transitioning to he containerd image-store and snapshotters, so it would be better to implement there (if not implemented yet)

I'll close this ticket because of the above, but feel free to continue the conversation

[rajat709]

Hi @thaJeztah

I have tried with both file backing systems but did'nt work

you check the conversion here https://forums.docker.com/t/docker-run-error-with-storage-opt-and-overlay2-on-xfs-with-pquota/138489

[thaJeztah]

I know there's been some corner-cases around feature-detection for the overlay driver, and there's some work (but unfinished) to improve (see #45890), although I don't know if feature-detection for quotas is part of that /cc @neersighted

Feature detection for the overlay driver is initiated in

moby/daemon/graphdriver/overlay2/overlay.go

Lines 189 to 196 in 75324c4

	if backingFs == "xfs" {
	// Try to enable project quota support over xfs.
	if d.quotaCtl, err = quota.NewControl(home); err == nil {
	projectQuotaSupported = true
	} else if opts.quota.Size > 0 {
	return nil, fmt.Errorf("Storage option overlay2.size not supported. Filesystem does not support Project Quota: %v", err)
	}
	} else if opts.quota.Size > 0 {

Which uses code from the quota package to perform the actual detection;

moby/quota/projectquota.go

Lines 107 to 132 in 75324c4

	// xfs_quota tool can be used to assign a project id to the driver home directory, e.g.:
	//
	// echo 999:/var/lib/docker/overlay2 >> /etc/projects
	// echo docker:999 >> /etc/projid
	// xfs_quota -x -c 'project -s docker' /<xfs mount point>
	//
	// In that case, the home directory project id will be used as a "start offset"
	// and all containers will be assigned larger project ids (e.g. >= 1000).
	// This is a way to prevent xfs_quota management from conflicting with docker.
	//
	// Then try to create a test directory with the next project id and set a quota
	// on it. If that works, continue to scan existing containers to map allocated
	// project ids.
	func NewControl(basePath string) (*Control, error) {
	//
	// If we are running in a user namespace quota won't be supported for
	// now since makeBackingFsDev() will try to mknod().
	//
	if userns.RunningInUserNS() {
	return nil, ErrQuotaNotSupported
	}
	//
	// create backing filesystem device node
	//
	backingFsBlockDev, err := makeBackingFsDev(basePath)

That code involves CGO though, and depends on some headers;

moby/quota/projectquota.go

Lines 1 to 19 in 75324c4

	//go:build linux && !exclude_disk_quota && cgo
	//
	// projectquota.go - implements XFS project quota controls
	// for setting quota limits on a newly created directory.
	// It currently supports the legacy XFS specific ioctls.
	//
	// TODO: use generic quota control ioctl FS_IOC_FS{GET,SET}XATTR
	// for both xfs/ext4 for kernel version >= v4.5
	//
	package quota // import "github.com/docker/docker/quota"
	/*
	#include <stdlib.h>
	#include <dirent.h>
	#include <linux/fs.h>
	#include <linux/quota.h>
	#include <linux/dqblk_xfs.h>

I also seem to recall that feature detection may happen on /var/lib/docker; in your case, is the whole filesystem xfs, or are you using a separate mount for it? Are you using a custom configuration to use a non-default path for docker's storage?

From the output of your docker version and docker info, I suspect you're running a version of docker that was built and packaged by your distro (Ubuntu);

Git commit:        24.0.5-0ubuntu1~22.04.1

That may not be relevant, but we know that distro-maintainers tend to make modifications in the build-process (including different versions of the Go toolchain and build-time dependencies), which has resulted in (sometimes subtly) broken packages; if you have a system to test on, do you also see the same issue when installing packages from download.docker.com? (instructions for installing in https://docs.docker.com/engine/install/ubuntu/).

[MohamedKarrab]

I need this on extfs I also have this problem

[tung06021992]

Dear @thaJeztah
Do you have any update for this issue ?
My system is running extfs so I need resolve it.
Thank you so much.

[thaJeztah]

@tung06021992 it's not supported on extfs currently; see #29364