Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Cannot run rootless mongodb on fedora #3816

Closed
aaronhagopian opened this issue Aug 14, 2019 · 8 comments
Closed

Cannot run rootless mongodb on fedora #3816

aaronhagopian opened this issue Aug 14, 2019 · 8 comments
Labels
kind/bug Categorizes issue or PR as related to a bug. locked - please file new issue/PR Assist humans wanting to comment on an old issue or PR with locked comments.

Comments

@aaronhagopian
Copy link

Is this a BUG REPORT or FEATURE REQUEST? (leave only one on its own line)

/kind bug

Description
Running mongodb rootless on fedora systems errors out while running on Ubuntu seems fine.

Steps to reproduce the issue:
Running on Fedora 30

  1. podman run --rm mongo:4

Describe the results you received:

$ podman run --rm mongo:4
chown: changing ownership of '/proc/1/fd/1': Permission denied
chown: changing ownership of '/proc/1/fd/2': Permission denied
2019-08-14T18:28:58.570+0000 I CONTROL [main] Automatically disabling TLS 1.0, to force-enable TLS 1.0 specify --sslDisabledProtocols 'none'
2019-08-14T18:28:58.575+0000 I CONTROL [initandlisten] MongoDB starting : pid=1 port=27017 dbpath=/data/db 64-bit host=69401cbc276b
2019-08-14T18:28:58.576+0000 I CONTROL [initandlisten] db version v4.2.0
2019-08-14T18:28:58.576+0000 I CONTROL [initandlisten] git version: a4b751dcf51dd249c5865812b390cfd1c0129c30
2019-08-14T18:28:58.576+0000 I CONTROL [initandlisten] OpenSSL version: OpenSSL 1.1.1 11 Sep 2018
2019-08-14T18:28:58.576+0000 I CONTROL [initandlisten] allocator: tcmalloc
2019-08-14T18:28:58.576+0000 I CONTROL [initandlisten] modules: none
2019-08-14T18:28:58.577+0000 I CONTROL [initandlisten] build environment:
2019-08-14T18:28:58.577+0000 I CONTROL [initandlisten] distmod: ubuntu1804
2019-08-14T18:28:58.577+0000 I CONTROL [initandlisten] distarch: x86_64
2019-08-14T18:28:58.577+0000 I CONTROL [initandlisten] target_arch: x86_64
2019-08-14T18:28:58.577+0000 I CONTROL [initandlisten] options: { net: { bindIp: "*" } }
2019-08-14T18:28:58.578+0000 E NETWORK [initandlisten] Failed to chmod socket file /tmp/mongodb-27017.sock No such device or address
2019-08-14T18:28:58.578+0000 F - [initandlisten] Fatal Assertion 40487 at src/mongo/transport/transport_layer_asio.cpp 725
2019-08-14T18:28:58.578+0000 F - [initandlisten]

***aborting after fassert() failure

Describe the results you expected:

Additional information you deem important (e.g. issue happens only occasionally):

Output of podman version:

Problem on fedora 30, built from source to try 1.5.0 since that worked fine on Ubuntu but had same issue w/ 1.5.0 from source.

Version:            1.4.4
RemoteAPI Version:  1
Go Version:         go1.12.7
OS/Arch:            linux/amd64

Output of podman info --debug:

debug:
  compiler: gc
  git commit: ""
  go version: go1.12.7
  podman version: 1.4.4
host:
  BuildahVersion: 1.9.0
  Conmon:
    package: podman-1.4.4-4.fc30.x86_64
    path: /usr/libexec/podman/conmon
    version: 'conmon version 1.0.0-dev, commit: 164df8af4e62dc759c312eab4b97ea9fb6b5f1fc'
  Distribution:
    distribution: fedora
    version: "30"
  MemFree: 547430400
  MemTotal: 8260423680
  OCIRuntime:
    package: runc-1.0.0-93.dev.gitb9b6cc6.fc30.x86_64
    path: /usr/bin/runc
    version: |-
      runc version 1.0.0-rc8+dev
      commit: e3b4c1108f7d1bf0d09ab612ea09927d9b59b4e3
      spec: 1.0.1-dev
  SwapFree: 8396992512
  SwapTotal: 8396992512
  arch: amd64
  cpus: 1
  hostname: lordhelmet.hra.local
  kernel: 5.2.7-200.fc30.x86_64
  os: linux
  rootless: true
  uptime: 29m 22.41s
registries:
  blocked: null
  insecure: null
  search:
  - docker.io
  - registry.fedoraproject.org
  - quay.io
  - registry.access.redhat.com
  - registry.centos.org
store:
  ConfigFile: /home/airhead/.config/containers/storage.conf
  ContainerStore:
    number: 1
  GraphDriverName: overlay
  GraphOptions:
  - overlay.mount_program=/usr/bin/fuse-overlayfs
  GraphRoot: /var/home/airhead/.local/share/containers/storage
  GraphStatus:
    Backing Filesystem: extfs
    Native Overlay Diff: "false"
    Supports d_type: "true"
    Using metacopy: "false"
  ImageStore:
    number: 2
  RunRoot: /tmp/1000
  VolumePath: /var/home/airhead/.local/share/containers/storage/volumes

Additional environment details (AWS, VirtualBox, physical, etc.):
Using HyperV for all of these tests

@openshift-ci-robot openshift-ci-robot added the kind/bug Categorizes issue or PR as related to a bug. label Aug 14, 2019
@mheon
Copy link
Member

mheon commented Aug 14, 2019

That looks like an ENXIO on chown of a unix socket?

Can you try again with --log-level=debug and provide the logs? Might have some warnings popping up...

Otherwise, my suspicion might be an attempt to use a UID/GID that aren't available in the container. I can see a useradd in the Dockerfile for that image, but it looks like it ought to work.

@giuseppe Might want to take a look here, too

@mheon
Copy link
Member

mheon commented Aug 14, 2019

Hmmm. Would a nodev tmpfs prevent unix sockets on /tmp?

@aaronhagopian
Copy link
Author

Here is the output with debug:

[airhead@lordhelmet ~]$ podman run --rm --log-level=debug mongo:4
INFO[0000] running as rootless
DEBU[0000] Initializing boltdb state at /var/home/airhead/.local/share/containers/storage/libpod/bolt_state.db
DEBU[0000] Using graph driver overlay
DEBU[0000] Using graph root /var/home/airhead/.local/share/containers/storage
DEBU[0000] Using run root /tmp/1000
DEBU[0000] Using static dir /var/home/airhead/.local/share/containers/storage/libpod
DEBU[0000] Using tmp dir /run/user/1000/libpod/tmp
DEBU[0000] Using volume path /var/home/airhead/.local/share/containers/storage/volumes
DEBU[0000] Set libpod namespace to ""
DEBU[0000] [graphdriver] trying provided driver "overlay"
DEBU[0000] overlay: mount_program=/usr/bin/fuse-overlayfs
DEBU[0000] backingFs=extfs, projectQuotaSupported=false, useNativeDiff=false, usingMetacopy=false
DEBU[0000] Initializing event backend journald
DEBU[0000] parsed reference into "[overlay@/var/home/airhead/.local/share/containers/storage+/tmp/1000:overlay.mount_program=/usr/bin/fuse-overlayfs]docker.io/library/mongo:4"
DEBU[0000] parsed reference into "[overlay@/var/home/airhead/.local/share/containers/storage+/tmp/1000:overlay.mount_program=/usr/bin/fuse-overlayfs]@1c7e513b19f3df326bb62c0f2ab29caefa55f3a203d28c37ba44002bb74f9a70"
DEBU[0000] exporting opaque data as blob "sha256:1c7e513b19f3df326bb62c0f2ab29caefa55f3a203d28c37ba44002bb74f9a70"
DEBU[0000] parsed reference into "[overlay@/var/home/airhead/.local/share/containers/storage+/tmp/1000:overlay.mount_program=/usr/bin/fuse-overlayfs]@1c7e513b19f3df326bb62c0f2ab29caefa55f3a203d28c37ba44002bb74f9a70"
DEBU[0000] exporting opaque data as blob "sha256:1c7e513b19f3df326bb62c0f2ab29caefa55f3a203d28c37ba44002bb74f9a70"
DEBU[0000] parsed reference into "[overlay@/var/home/airhead/.local/share/containers/storage+/tmp/1000:overlay.mount_program=/usr/bin/fuse-overlayfs]@1c7e513b19f3df326bb62c0f2ab29caefa55f3a203d28c37ba44002bb74f9a70"
DEBU[0000] Got mounts: []
DEBU[0000] Got volumes: [0xc00020a800 0xc00020a840]
DEBU[0000] Using slirp4netns netmode
DEBU[0000] created OCI spec and options for new container
DEBU[0000] Allocated lock 0 for container 0adc55987427eaa22b614ce588eec3bd6b8bd9b973718c9008fced29ef8271fc
DEBU[0000] parsed reference into "[overlay@/var/home/airhead/.local/share/containers/storage+/tmp/1000:overlay.mount_program=/usr/bin/fuse-overlayfs]@1c7e513b19f3df326bb62c0f2ab29caefa55f3a203d28c37ba44002bb74f9a70"
DEBU[0000] exporting opaque data as blob "sha256:1c7e513b19f3df326bb62c0f2ab29caefa55f3a203d28c37ba44002bb74f9a70"
DEBU[0000] created container "0adc55987427eaa22b614ce588eec3bd6b8bd9b973718c9008fced29ef8271fc"
DEBU[0000] container "0adc55987427eaa22b614ce588eec3bd6b8bd9b973718c9008fced29ef8271fc" has work directory "/var/home/airhead/.local/share/containers/storage/overlay-containers/0adc55987427eaa22b614ce588eec3bd6b8bd9b973718c9008fced29ef8271fc/userdata"
DEBU[0000] container "0adc55987427eaa22b614ce588eec3bd6b8bd9b973718c9008fced29ef8271fc" has run directory "/tmp/1000/overlay-containers/0adc55987427eaa22b614ce588eec3bd6b8bd9b973718c9008fced29ef8271fc/userdata"
DEBU[0000] Creating new volume eaac8d19cc8348e0970d5d5d8ceeb3fc241553d50ada9e32af40699d3673f990 for container
DEBU[0000] overlay: mount_data=lowerdir=/var/home/airhead/.local/share/containers/storage/overlay/l/U33BP52XFT3CTHO5BUVX75EKSS:/var/home/airhead/.local/share/containers/storage/overlay/l/33DW44NOXERUX6KKSX4JM5YTZS:/var/home/airhead/.local/share/containers/storage/overlay/l/3MJQBCTNHURRIYG6EGHAD3Y3FP:/var/home/airhead/.local/share/containers/storage/overlay/l/YQUZCYG22I3AONN5XW5H6JS4VM:/var/home/airhead/.local/share/containers/storage/overlay/l/J2U6JWD3YD22PVFYTGCMGNGCOJ:/var/home/airhead/.local/share/containers/storage/overlay/l/DZ6J4YRDKYHAO63KC7LMG63YN6:/var/home/airhead/.local/share/containers/storage/overlay/l/ZYVYK35XDSBZPDH433CANHG3JE:/var/home/airhead/.local/share/containers/storage/overlay/l/3BWT6IURVMREPPEE7K2DXKLZRG:/var/home/airhead/.local/share/containers/storage/overlay/l/RBPTISJMOI7STQTGSHEYL7KCGV:/var/home/airhead/.local/share/containers/storage/overlay/l/SREHUTBKZDQDPI33CZ7J4BO2QW:/var/home/airhead/.local/share/containers/storage/overlay/l/PS5R45A7L5HK5FG5TRXUK4M5NR:/var/home/airhead/.local/share/containers/storage/overlay/l/PXYHJ42TOGJZPNADYL2GVOJORQ:/var/home/airhead/.local/share/containers/storage/overlay/l/TEMIOYKOCXRD4S42IWRQ73FN3I,upperdir=/var/home/airhead/.local/share/containers/storage/overlay/727fd3d7a7a498745ddc793767500de26cc364b9443f35e6bfdec30ce9f06fce/diff,workdir=/var/home/airhead/.local/share/containers/storage/overlay/727fd3d7a7a498745ddc793767500de26cc364b9443f35e6bfdec30ce9f06fce/work,context="system_u:object_r:container_file_t:s0:c389,c892"
DEBU[0000] mounted container "0adc55987427eaa22b614ce588eec3bd6b8bd9b973718c9008fced29ef8271fc" at "/var/home/airhead/.local/share/containers/storage/overlay/727fd3d7a7a498745ddc793767500de26cc364b9443f35e6bfdec30ce9f06fce/merged"
DEBU[0000] Creating dest directory: /var/home/airhead/.local/share/containers/storage/volumes/eaac8d19cc8348e0970d5d5d8ceeb3fc241553d50ada9e32af40699d3673f990/_data
DEBU[0000] Calling TarUntar(/var/home/airhead/.local/share/containers/storage/overlay/727fd3d7a7a498745ddc793767500de26cc364b9443f35e6bfdec30ce9f06fce/merged/data/configdb, /var/home/airhead/.local/share/containers/storage/volumes/eaac8d19cc8348e0970d5d5d8ceeb3fc241553d50ada9e32af40699d3673f990/_data)
DEBU[0000] TarUntar(/var/home/airhead/.local/share/containers/storage/overlay/727fd3d7a7a498745ddc793767500de26cc364b9443f35e6bfdec30ce9f06fce/merged/data/configdb /var/home/airhead/.local/share/containers/storage/volumes/eaac8d19cc8348e0970d5d5d8ceeb3fc241553d50ada9e32af40699d3673f990/_data)
DEBU[0000] Creating new volume cb8c8d6dfefe3826e3720513b588cfd360eff28bdc7374599154a5b80cf16c26 for container
DEBU[0000] mounted container "0adc55987427eaa22b614ce588eec3bd6b8bd9b973718c9008fced29ef8271fc" at "/var/home/airhead/.local/share/containers/storage/overlay/727fd3d7a7a498745ddc793767500de26cc364b9443f35e6bfdec30ce9f06fce/merged"
DEBU[0000] Creating dest directory: /var/home/airhead/.local/share/containers/storage/volumes/cb8c8d6dfefe3826e3720513b588cfd360eff28bdc7374599154a5b80cf16c26/_data
DEBU[0000] Calling TarUntar(/var/home/airhead/.local/share/containers/storage/overlay/727fd3d7a7a498745ddc793767500de26cc364b9443f35e6bfdec30ce9f06fce/merged/data/db, /var/home/airhead/.local/share/containers/storage/volumes/cb8c8d6dfefe3826e3720513b588cfd360eff28bdc7374599154a5b80cf16c26/_data)
DEBU[0000] TarUntar(/var/home/airhead/.local/share/containers/storage/overlay/727fd3d7a7a498745ddc793767500de26cc364b9443f35e6bfdec30ce9f06fce/merged/data/db /var/home/airhead/.local/share/containers/storage/volumes/cb8c8d6dfefe3826e3720513b588cfd360eff28bdc7374599154a5b80cf16c26/_data)
DEBU[0000] New container created "0adc55987427eaa22b614ce588eec3bd6b8bd9b973718c9008fced29ef8271fc"
DEBU[0000] container "0adc55987427eaa22b614ce588eec3bd6b8bd9b973718c9008fced29ef8271fc" has CgroupParent "/libpod_parent/libpod-0adc55987427eaa22b614ce588eec3bd6b8bd9b973718c9008fced29ef8271fc"
DEBU[0000] Not attaching to stdin
DEBU[0000] mounted container "0adc55987427eaa22b614ce588eec3bd6b8bd9b973718c9008fced29ef8271fc" at "/var/home/airhead/.local/share/containers/storage/overlay/727fd3d7a7a498745ddc793767500de26cc364b9443f35e6bfdec30ce9f06fce/merged"
DEBU[0000] Created root filesystem for container 0adc55987427eaa22b614ce588eec3bd6b8bd9b973718c9008fced29ef8271fc at /var/home/airhead/.local/share/containers/storage/overlay/727fd3d7a7a498745ddc793767500de26cc364b9443f35e6bfdec30ce9f06fce/merged
DEBU[0000] /etc/system-fips does not exist on host, not mounting FIPS mode secret
DEBU[0000] Created OCI spec for container 0adc55987427eaa22b614ce588eec3bd6b8bd9b973718c9008fced29ef8271fc at /var/home/airhead/.local/share/containers/storage/overlay-containers/0adc55987427eaa22b614ce588eec3bd6b8bd9b973718c9008fced29ef8271fc/userdata/config.json
DEBU[0000] /usr/libexec/podman/conmon messages will be logged to syslog
DEBU[0000] running conmon: /usr/libexec/podman/conmon    args="[-c 0adc55987427eaa22b614ce588eec3bd6b8bd9b973718c9008fced29ef8271fc -u 0adc55987427eaa22b614ce588eec3bd6b8bd9b973718c9008fced29ef8271fc -n cocky_liskov -r /usr/bin/runc -b /var/home/airhead/.local/share/containers/storage/overlay-containers/0adc55987427eaa22b614ce588eec3bd6b8bd9b973718c9008fced29ef8271fc/userdata -p /tmp/1000/overlay-containers/0adc55987427eaa22b614ce588eec3bd6b8bd9b973718c9008fced29ef8271fc/userdata/pidfile --exit-dir /run/user/1000/libpod/tmp/exits --conmon-pidfile /tmp/1000/overlay-containers/0adc55987427eaa22b614ce588eec3bd6b8bd9b973718c9008fced29ef8271fc/userdata/conmon.pid --exit-command /usr/bin/podman --exit-command-arg --root --exit-command-arg /var/home/airhead/.local/share/containers/storage --exit-command-arg --runroot --exit-command-arg /tmp/1000 --exit-command-arg --log-level --exit-command-arg debug --exit-command-arg --cgroup-manager --exit-command-arg cgroupfs --exit-command-arg --tmpdir --exit-command-arg /run/user/1000/libpod/tmp --exit-command-arg --runtime --exit-command-arg runc --exit-command-arg --storage-driver --exit-command-arg overlay --exit-command-arg --storage-opt --exit-command-arg overlay.mount_program=/usr/bin/fuse-overlayfs --exit-command-arg container --exit-command-arg cleanup --exit-command-arg --rm --exit-command-arg 0adc55987427eaa22b614ce588eec3bd6b8bd9b973718c9008fced29ef8271fc --socket-dir-path /run/user/1000/libpod/tmp/socket -l k8s-file:/var/home/airhead/.local/share/containers/storage/overlay-containers/0adc55987427eaa22b614ce588eec3bd6b8bd9b973718c9008fced29ef8271fc/userdata/ctr.log --log-level debug --syslog]"
WARN[0000] Failed to add conmon to cgroupfs sandbox cgroup: error creating cgroup for cpu: mkdir /sys/fs/cgroup/cpu/libpod_parent: permission denied
[conmon:d]: failed to write to /proc/self/oom_score_adj: Permission denied

DEBU[0000] Received container pid: 1262
DEBU[0000] Created container 0adc55987427eaa22b614ce588eec3bd6b8bd9b973718c9008fced29ef8271fc in OCI runtime
DEBU[0000] Attaching to container 0adc55987427eaa22b614ce588eec3bd6b8bd9b973718c9008fced29ef8271fc
DEBU[0000] connecting to socket /run/user/1000/libpod/tmp/socket/0adc55987427eaa22b614ce588eec3bd6b8bd9b973718c9008fced29ef8271fc/attach
DEBU[0000] Starting container 0adc55987427eaa22b614ce588eec3bd6b8bd9b973718c9008fced29ef8271fc with command [docker-entrypoint.sh mongod]
DEBU[0000] Started container 0adc55987427eaa22b614ce588eec3bd6b8bd9b973718c9008fced29ef8271fc
DEBU[0000] Enabling signal proxying
chown: changing ownership of '/proc/1/fd/1': Permission denied
chown: changing ownership of '/proc/1/fd/2': Permission denied
2019-08-14T20:14:26.875+0000 I  CONTROL  [main] Automatically disabling TLS 1.0, to force-enable TLS 1.0 specify --sslDisabledProtocols 'none'
2019-08-14T20:14:26.879+0000 I  CONTROL  [initandlisten] MongoDB starting : pid=1 port=27017 dbpath=/data/db 64-bit host=0adc55987427
2019-08-14T20:14:26.879+0000 I  CONTROL  [initandlisten] db version v4.2.0
2019-08-14T20:14:26.879+0000 I  CONTROL  [initandlisten] git version: a4b751dcf51dd249c5865812b390cfd1c0129c30
2019-08-14T20:14:26.880+0000 I  CONTROL  [initandlisten] OpenSSL version: OpenSSL 1.1.1  11 Sep 2018
2019-08-14T20:14:26.880+0000 I  CONTROL  [initandlisten] allocator: tcmalloc
2019-08-14T20:14:26.880+0000 I  CONTROL  [initandlisten] modules: none
2019-08-14T20:14:26.880+0000 I  CONTROL  [initandlisten] build environment:
2019-08-14T20:14:26.880+0000 I  CONTROL  [initandlisten]     distmod: ubuntu1804
2019-08-14T20:14:26.880+0000 I  CONTROL  [initandlisten]     distarch: x86_64
2019-08-14T20:14:26.880+0000 I  CONTROL  [initandlisten]     target_arch: x86_64
2019-08-14T20:14:26.880+0000 I  CONTROL  [initandlisten] options: { net: { bindIp: "*" } }
2019-08-14T20:14:26.882+0000 E  NETWORK  [initandlisten] Failed to chmod socket file /tmp/mongodb-27017.sock No such device or address
2019-08-14T20:14:26.882+0000 F  -        [initandlisten] Fatal Assertion 40487 at src/mongo/transport/transport_layer_asio.cpp 725
2019-08-14T20:14:26.882+0000 F  -        [initandlisten]

***aborting after fassert() failure


DEBU[0000] Checking container 0adc55987427eaa22b614ce588eec3bd6b8bd9b973718c9008fced29ef8271fc status...
DEBU[0000] Attempting to read container 0adc55987427eaa22b614ce588eec3bd6b8bd9b973718c9008fced29ef8271fc exit code from file /run/user/1000/libpod/tmp/exits/0adc55987427eaa22b614ce588eec3bd6b8bd9b973718c9008fced29ef8271fc-old
DEBU[0000] [graphdriver] trying provided driver "overlay"
DEBU[0000] overlay: mount_program=/usr/bin/fuse-overlayfs
DEBU[0000] backingFs=extfs, projectQuotaSupported=false, useNativeDiff=false, usingMetacopy=false

@giuseppe
Copy link
Member

I think it is caused by a regression in fuse-overlayfs.

Can you confirm you are using fuse-overlayfs 0.5?

If that is the case, https://bodhi.fedoraproject.org/updates/FEDORA-2019-33bf337c28 might solve the issue you are seeing

@aaronhagopian
Copy link
Author

Yes I am running fuse-overlayfs 0.5.

Upgraded to 0.5.1-2 from the testing repository and the issue was resolved, hopefully they push that package to stable soon.

Thanks

@baude
Copy link
Member

baude commented Aug 14, 2019

@aaronhagopian it needs karma to get released -> https://bodhi.fedoraproject.org/updates/FEDORA-2019-33bf337c28

I'd encourage you to give it karma

@aaronhagopian
Copy link
Author

I'd encourage you to give it karma

@baude Done, thanks for the heads up.

@rhatdan
Copy link
Member

rhatdan commented Aug 15, 2019

Can we get one more good karma it will come out.

@github-actions github-actions bot added the locked - please file new issue/PR Assist humans wanting to comment on an old issue or PR with locked comments. label Sep 23, 2023
@github-actions github-actions bot locked as resolved and limited conversation to collaborators Sep 23, 2023
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
kind/bug Categorizes issue or PR as related to a bug. locked - please file new issue/PR Assist humans wanting to comment on an old issue or PR with locked comments.
Projects
None yet
Development

No branches or pull requests

6 participants