New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

buildah inspect errors with "vfs driver does not support mount options" but podman inspect works fine #1251

Closed
TomasTomecek opened this Issue Dec 27, 2018 · 9 comments

Comments

Projects
None yet
2 participants
@TomasTomecek
Copy link
Contributor

TomasTomecek commented Dec 27, 2018

Description

buildah inspect errors out while podman inspect works just fine

Steps to reproduce the issue:
#1251 (comment)

Describe the results you received:

vfs driver does not support mount options
vfs driver does not support mount options

Output of buildah version:

Version:         1.5
Go Version:      go1.11.2
Image Spec:      1.0.0
Runtime Spec:    1.0.0
CNI Spec:        0.4.0
libcni Version:  
Git Commit:      
Built:           Thu Jan  1 00:00:00 1970
OS/Arch:         linux/amd64

Host: F29
Environment: F29, openshift pod running in privileged mode

More info

[root@ab-pod /]# uname -a
Linux ab-pod 4.19.7-300.fc29.x86_64 #1 SMP Wed Dec 5 22:21:07 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux

[root@ab-pod /]# cat /etc/containers/storage.conf
[storage]

# Default Storage Driver
driver = "vfs"

# Primary Read/Write location of container storage
graphroot = "/tmp/containers"

[root@ab-pod /]# id
uid=0(root) gid=0(root) groups=0(root)

I only changed those two storage opts.

@TomasTomecek

This comment has been minimized.

Copy link
Contributor

TomasTomecek commented Dec 27, 2018

same thing happens with buildah from, while podman run works fine

@TomasTomecek

This comment has been minimized.

Copy link
Contributor

TomasTomecek commented Dec 27, 2018

Tried buildah 1.6, the same result:

vfs driver does not support mount options                                                                                                                            
vfs driver does not support mount options
@rhatdan

This comment has been minimized.

Copy link
Member

rhatdan commented Dec 28, 2018

Does podman info or buildah info show any mount options?

@TomasTomecek

This comment has been minimized.

Copy link
Contributor

TomasTomecek commented Dec 30, 2018

Here are the complete logs: https://paste.fedoraproject.org/paste/7zMBAVc0SxFgQfwNpsE6cQ

podman info output:

store:
  ContainerStore:
    number: 0
  GraphDriverName: vfs
  GraphOptions: null
  GraphRoot: /tmp/containers
  GraphStatus: {}
  ImageStore:
    number: 0
  RunRoot: /var/run/containers/storage
@rhatdan

This comment has been minimized.

Copy link
Member

rhatdan commented Dec 30, 2018

I don't get these failures.

buildah from fedora
Getting image source signatures
Copying blob sha256:cd6c8343b59020c6bef022d3f81902a428b0552f1b3c0938fea36383eb68ff6c
 85.64 MiB / 85.64 MiB [===================================================] 18s
Copying config sha256:25e6809f6fabf364c645430961bb3a02adc5b93f7bcdc8a41b692100cd0aa8bd
 2.20 KiB / 2.20 KiB [======================================================] 0s
Writing manifest to image destination
Storing signatures
fedora-working-container
# ./buildah info
{
    "host": {
        "Distribution": {
            "distribution": "fedora",
            "version": "29"
        },
        "MemTotal": 16450400256,
        "MenFree": 197111808,
        "SwapFree": 8264347648,
        "SwapTotal": 8296329216,
        "arch": "amd64",
        "cpus": 8,
        "hostname": "localhost.localdomain",
        "kernel": "4.19.7-300.fc29.x86_64",
        "os": "linux",
        "rootless": false,
        "uptime": "388h 35m 0.8s (Approximately 16.17 days)"
    },
    "store": {
        "ContainerStore": {
            "number": 1
        },
        "GraphDriverName": "vfs",
        "GraphOptions": null,
        "GraphRoot": "/tmp/containers",
        "GraphStatus": {},
        "ImageStore": {
            "number": 1
        },
        "RunRoot": "/var/run/containers/storage"
    }
}
@rhatdan

This comment has been minimized.

Copy link
Member

rhatdan commented Dec 30, 2018

# cat /etc/containers/storage.conf
[storage]

# Default Storage Driver
driver = "vfs"

# Primary Read/Write location of container storage
graphroot = "/tmp/containers"
@rhatdan

This comment has been minimized.

Copy link
Member

rhatdan commented Jan 3, 2019

Could you do the debug output
buildah --debug pull

@TomasTomecek

This comment has been minimized.

Copy link
Contributor

TomasTomecek commented Jan 4, 2019

I probably haven't made it clear in the original post -- I'm sorry about that. I am running this inside an openshift pod, not on my workstation. Hence I'm going to post the "minimal reproducer":

$ oc cluster up

$ # I want to run buildah in a privileged pod
$ oc login -u system:admin
$ oc adm policy add-scc-to-user privileged developer
$ oc adm policy remove-scc-from-user restricted developer

$ oc login -u developer -p developer

$ cat pod.yml
apiVersion: v1
kind: Pod
metadata:
  name: b
spec:
  containers:
  - command:
    - bash
    stdin: true
    tty: true
    image: registry.fedoraproject.org/fedora:29
    imagePullPolicy: IfNotPresent
    name: b
    securityContext:
      privileged: true
  restartPolicy: Never

$ oc create -f pod.yml
pod/b created
$ oc attach pod/b -t -i
Defaulting container name to b.                                                                                                                                             
Use 'oc describe pod/ -n myproject' to see all of the containers in this pod.                                                                                               
If you don't see a command prompt, try pressing enter.                                                                                                                      

[root@b /]# id                                                                                                                                                              
uid=0(root) gid=0(root) groups=0(root) 

[root@b /]# dnf install -y buildah
Installed:
  buildah-1.5-1.gite94b4f9.fc29.x86_64

[root@b /]# vi /etc/containers/storage.conf

[root@b /]# buildah --debug pull busybox
DEBU[0000] [graphdriver] trying provided driver "vfs"   
DEBU[0000] [graphdriver] trying provided driver "vfs"   
ERRO[0000] vfs driver does not support mount options
vfs driver does not support mount options 

I also tried to strace -f the bash process to see the syscalls:

[pid 16585] newfstatat(AT_FDCWD, "/tmp/containers", {st_mode=S_IFDIR|0700, st_size=62, ...}, 0) = 0                                                                          
[pid 16585] newfstatat(AT_FDCWD, "/tmp/containers/mounts", {st_mode=S_IFDIR|0700, st_size=6, ...}, 0) = 0                                                                    
[pid 16585] newfstatat(AT_FDCWD, "/tmp/containers/tmp", {st_mode=S_IFDIR|0700, st_size=6, ...}, 0) = 0                                                                       
[pid 16585] newfstatat(AT_FDCWD, "/tmp/containers/vfs", {st_mode=S_IFDIR|0700, st_size=6, ...}, 0) = 0                                                                      
[pid 16585] getpid()                    = 133                                                
[pid 16585] fcntl(5, F_SETLKW, {l_type=F_WRLCK, l_whence=SEEK_SET, l_start=0, l_len=0}) = 0                                                                                 
[pid 16585] fstat(5, {st_mode=S_IFREG|0600, st_size=0, ...}) = 0                                                                                                            
[pid 16585] write(2, "\33[37mDEBU\33[0m[0000] [graphdriver] trying provided driver \"vfs\"   \n", 66) = 66                                                                  
[pid 16585] newfstatat(AT_FDCWD, "/tmp/containers/vfs", {st_mode=S_IFDIR|0700, st_size=6, ...}, 0) = 0                                                                       
[pid 16585] fchownat(AT_FDCWD, "/tmp/containers/vfs", 0, 0, 0) = 0                           
[pid 16585] getpid()                    = 133                                                                                                                               
[pid 16585] fcntl(5, F_SETLKW, {l_type=F_UNLCK, l_whence=SEEK_SET, l_start=0, l_len=0}) = 0  
[pid 16585] write(2, "\33[31mERRO\33[0m[0000] vfs driver does not support mount options\nvfs driver does not support mount options \n", 105) = 105

Honestly I'm not sure what went wrong.

[root@b /]# ls -lha /tmp/containers/
total 0
drwx------. 5 root root 62 Jan  4 10:58 .
drwxrwxrwt. 1 root root 24 Jan  4 10:58 ..
drwx------. 2 root root  6 Jan  4 10:58 mounts
-rw-------. 1 root root  0 Jan  4 10:58 storage.lock
drwx------. 2 root root  6 Jan  4 10:58 tmp
drwx------. 2 root root  6 Jan  4 10:58 vfs
[root@b /]# ls -lha /tmp/containers/vfs
total 0
drwx------. 2 root root  6 Jan  4 10:58 .
drwx------. 5 root root 62 Jan  4 10:58 ..
[root@b /]# ls -lha /tmp/containers/mounts/
total 0
drwx------. 2 root root  6 Jan  4 10:58 .
drwx------. 5 root root 62 Jan  4 10:58 ..

Since OKD is already integrating buildah, I'll do the same thing as them -- create a volume in the pod and use overlay backend. Therefore we can close this since I have no idea what's wrong. Feel free to leave it open if you want to investigate more.

@TomasTomecek

This comment has been minimized.

Copy link
Contributor

TomasTomecek commented Jan 5, 2019

After I did what I hinted: created an openshift volume at /var/lib/containers and switched graph backend to overaly, everything works just fine (well except that the pod is still privileged): I can pull images, do buildah from, buildah run and buildah commit. Hence I'm closing since I definitely want to use overlay over vfs.

Dan, thanks for your help.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment