New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

buildah connection doesn't work with rootless buildah & fuse-overlay #50583

Open
TomasTomecek opened this Issue Jan 6, 2019 · 2 comments

Comments

Projects
None yet
3 participants
@TomasTomecek
Copy link
Contributor

TomasTomecek commented Jan 6, 2019

SUMMARY

When using rootless buildah and fuse-overlay together, buildah's mount command doesn't function due to a kernel limitation. This completely breaks the connection plugin since mount is the first thing that happens.

We have tracked the original issue at buildah's side: containers/buildah#1225, check it out for the technical details.

Sadly, buildah devs decided to disable the mount command for such setup which means that buildah conn can't work in that environment. @giuseppe posted a way around the issue by utilizing buildah unshare. I haven't checked whether it would be possible to fix the connection plugin with such approach.

I am creating this issue for sake of tracking the limitation.

ISSUE TYPE
  • Bug Report
COMPONENT NAME

plugin/connection/buildah

ANSIBLE VERSION
ansible 2.7.5
  config file = /etc/ansible/ansible.cfg
  configured module search path = ['/home/tt/.ansible/plugins/modules', '/usr/share/ansible/plugins/modules']
  ansible python module location = /usr/lib/python3.7/site-packages/ansible
  executable location = /usr/bin/ansible
  python version = 3.7.2 (default, Jan  3 2019, 09:14:01) [GCC 8.2.1 20181215 (Red Hat 8.2.1-6)]
OS / ENVIRONMENT

Fedora 29

ACTUAL RESULTS
<this-is-my-image-20190106-135255878492-cont> RUN [b'buildah', b'mount', b'--', b'this-is-my-image-20190106-135255878492-cont']
MOUNTPOINT b'/home/tt/.local/share/containers/storage/overlay/1309b1651de8f37500787b32feceac19a82d40b9c650d31d73d67f473f45253b/merged' RC 0 STDERR b''
<this-is-my-image-20190106-135255878492-cont> RUN [b'buildah', b'run', b'--', b'this-is-my-image-20190106-135255878492-cont', b'/bin/sh', b'-c', b'( umask 77 && mkdir -p "` echo /tmp/ansible-tmp-1546779179.0218472-76849117941910 `" && echo ansible-tmp-1546779179.0218472-76849117941910="` echo /tmp/ansible-tmp-1546779179.0218472-76849117941910 `" ) && sleep 0']
STDOUT b'error running container: error creating container for [/bin/sh -c ( umask 77 && mkdir -p "` echo /tmp/ansible-tmp-1546779179.0218472-76849117941910 `" && echo ansible-tmp-1546779179.0218472-76849117941910="` echo /tmp/ansible-tmp-1546779179.0218472-76849117941910 `" ) && sleep 0]: container_linux.go:344: starting container process caused "exec: \\"/bin/sh\\": stat /bin/sh: no such file or directory"\n: exit status 1\nerror while running runtime: exit status 1\ntime="2019-01-06T13:52:59+01:00" level=error msg="exit status 1"\n' STDERR b'error running container: error creating container for [/bin/sh -c ( umask 77 && mkdir -p "` echo /tmp/ansible-tmp-1546779179.0218472-76849117941910 `" && echo ansible-tmp-1546779179.0218472-76849117941910="` echo /tmp/ansible-tmp-1546779179.0218472-76849117941910 `" ) && sleep 0]: container_linux.go:344: starting container process caused "exec: \\"/bin/sh\\": stat /bin/sh: no such file or directory"\n: exit status 1\nerror while running runtime: exit status 1\ntime="2019-01-06T13:52:59+01:00" level=error msg="exit status 1"\n'
<this-is-my-image-20190106-135255878492-cont> RUN [b'buildah', b'umount', b'--', b'this-is-my-image-20190106-135255878492-cont']
RC 0 STDOUT b'' STDERR b''
fatal: [this-is-my-image-20190106-135255878492-cont]: UNREACHABLE! => {
    "changed": false,
    "msg": "Authentication or permission failure. In some cases, you may have been able to authenticate and did not have permissions on the target directory. Consider changing the remote tmp path in ansible.cfg to a path rooted in \"/tmp\". Failed command was: ( umask 77 && mkdir -p \"` echo /tmp/ansible-tmp-1546779179.0218472-76849117941910 `\" && echo ansible-tmp-1546779179.0218472-76849117941910=\"` echo /tmp/ansible-tmp-1546779179.0218472-76849117941910 `\" ), exited with result 1, stderr output: error running container: error creating container for [/bin/sh -c ( umask 77 && mkdir -p \"` echo /tmp/ansible-tmp-1546779179.0218472-76849117941910 `\" && echo ansible-tmp-1546779179.0218472-76849117941910=\"` echo /tmp/ansible-tmp-1546779179.0218472-76849117941910 `\" ) && sleep 0]: container_linux.go:344: starting container process caused \"exec: \\\"/bin/sh\\\": stat /bin/sh: no such file or directory\"\n: exit status 1\nerror while running runtime: exit status 1\ntime=\"2019-01-06T13:52:59+01:00\" level=error msg=\"exit status 1\"\n",
    "unreachable": true
}
@ansibot

This comment has been minimized.

Copy link
Contributor

ansibot commented Jan 6, 2019

Files identified in the description:

If these files are inaccurate, please update the component name section of the description or use the !component bot command.

click here for bot help

@TomasTomecek

This comment has been minimized.

Copy link
Contributor

TomasTomecek commented Jan 9, 2019

Bad news, the proposed solution with buildah unshare doesn't work, filed this buildah issue to track it: containers/buildah#1271

Some ugly debug output:

<python-working-container> RUN [b'buildah', b'unshare', b'buildah', b'mount', b'--', b'python-working-container']
MOUNTPOINT b'/home/tt/.local/share/containers/storage/overlay/eaba594414c72a60a899bcc3e3b09b7004742f1d947ecc146d502f11a67f0c05/merged' RC 0 STDERR b''
<python-working-container> RUN [b'buildah', b'run', b'--', b'python-working-container', b'/bin/sh', b'-c', b'echo ~ && sleep 0']                                                                                  
STDOUT b'error running container: error creating container for [/bin/sh -c echo ~ && sleep 0]: container_linux.go:344: starting container process caused "exec: \\"/bin/sh\\": stat /bin/sh: no such file or direct
ory"\n: exit status 1\nerror while running runtime: exit status 1\ntime="2019-01-09T21:43:36+01:00" level=error msg="exit status 1"\n' STDERR b'error running container: error creating container for [/bin/sh -c e
cho ~ && sleep 0]: container_linux.go:344: starting container process caused "exec: \\"/bin/sh\\": stat /bin/sh: no such file or directory"\n: exit status 1\nerror while running runtime: exit status 1\ntime="201
9-01-09T21:43:36+01:00" level=error msg="exit status 1"\n'                                                                                                                                                        
<python-working-container> RUN [b'buildah', b'run', b'--', b'python-working-container', b'/bin/sh', b'-c', b'echo "`pwd`" && sleep 0']                                                                            
STDOUT b'error running container: error creating container for [/bin/sh -c echo "`pwd`" && sleep 0]: container_linux.go:344: starting container process caused "exec: \\"/bin/sh\\": stat /bin/sh: no such file or 
directory"\n: exit status 1\nerror while running runtime: exit status 1\ntime="2019-01-09T21:43:36+01:00" level=error msg="exit status 1"\n' STDERR b'error running container: error creating container for [/bin/s
h -c echo "`pwd`" && sleep 0]: container_linux.go:344: starting container process caused "exec: \\"/bin/sh\\": stat /bin/sh: no such file or directory"\n: exit status 1\nerror while running runtime: exit status 
1\ntime="2019-01-09T21:43:36+01:00" level=error msg="exit status 1"\n'                                                                                                                                            
<python-working-container> RUN [b'buildah', b'run', b'--', b'python-working-container', b'/bin/sh', b'-c', b'( umask 77 && mkdir -p "` echo ~/.ansible/tmp/ansible-tmp-1547066615.8078423-108154880266590 `" && ech
o ansible-tmp-1547066615.8078423-108154880266590="` echo ~/.ansible/tmp/ansible-tmp-1547066615.8078423-108154880266590 `" ) && sleep 0']
STDOUT b'error running container: error creating container for [/bin/sh -c ( umask 77 && mkdir -p "` echo ~/.ansible/tmp/ansible-tmp-1547066615.8078423-108154880266590 `" && echo ansible-tmp-1547066615.8078423-1
08154880266590="` echo ~/.ansible/tmp/ansible-tmp-1547066615.8078423-108154880266590 `" ) && sleep 0]: container_linux.go:344: starting container process caused "exec: \\"/bin/sh\\": stat /bin/sh: no such file o
r directory"\n: exit status 1\nerror while running runtime: exit status 1\ntime="2019-01-09T21:43:37+01:00" level=error msg="exit status 1"\n' STDERR b'error running container: error creating container for [/bin
/sh -c ( umask 77 && mkdir -p "` echo ~/.ansible/tmp/ansible-tmp-1547066615.8078423-108154880266590 `" && echo ansible-tmp-1547066615.8078423-108154880266590="` echo ~/.ansible/tmp/ansible-tmp-1547066615.8078423
-108154880266590 `" ) && sleep 0]: container_linux.go:344: starting container process caused "exec: \\"/bin/sh\\": stat /bin/sh: no such file or directory"\n: exit status 1\nerror while running runtime: exit sta
tus 1\ntime="2019-01-09T21:43:37+01:00" level=error msg="exit status 1"\n'                                                                                                                                        
<python-working-container> RUN [b'buildah', b'umount', b'--', b'python-working-container']
RC 0 STDOUT b'' STDERR b''
fatal: [python-working-container]: UNREACHABLE! => {
    "changed": false,
    "msg": "Authentication or permission failure. In some cases, you may have been able to authenticate and did not have permissions on the target directory. Consider changing the remote tmp path in ansible.cfg 
to a path rooted in \"/tmp\". Failed command was: ( umask 77 && mkdir -p \"` echo ~/.ansible/tmp/ansible-tmp-1547066615.8078423-108154880266590 `\" && echo ansible-tmp-1547066615.8078423-108154880266590=\"` echo
 ~/.ansible/tmp/ansible-tmp-1547066615.8078423-108154880266590 `\" ), exited with result 1, stderr output: error running container: error creating container for [/bin/sh -c ( umask 77 && mkdir -p \"` echo ~/.ans
ible/tmp/ansible-tmp-1547066615.8078423-108154880266590 `\" && echo ansible-tmp-1547066615.8078423-108154880266590=\"` echo ~/.ansible/tmp/ansible-tmp-1547066615.8078423-108154880266590 `\" ) && sleep 0]: contai
ner_linux.go:344: starting container process caused \"exec: \\\"/bin/sh\\\": stat /bin/sh: no such file or directory\"\n: exit status 1\nerror while running runtime: exit status 1\ntime=\"2019-01-09T21:43:37+01:
00\" level=error msg=\"exit status 1\"\n",
    "unreachable": true
}
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment