Uninformative slirp4netns ERROR when portmapping

[r-cheologist]

Is this a BUG REPORT or FEATURE REQUEST? (leave only one on its own line)

/kind bug

Description

Running unprivileged (rootless) container on ArchLinux (using dnsmasq and shorewall) gives uninformative ERRO[0006] slirp4netns failed. Same operation as root works.

Steps to reproduce the issue:

1. Install podman.

2. Pull alpine: podman pull alpine

3. Run container while portmapping docker-equivalently: podman run -d --log-level=debug -p 127.0.0.1:8787:8787 docker.io/library/alpine

Describe the results you received:
See the attached (podman.log) output of podman run -d --log-level=debug -p 127.0.0.1:8787:8787 docker.io/library/alpine 2> /tmp/podman.log: container fails with non-descript slirp4netns error.

Describe the results you expected:
Container starts with port forwarding to localhost.

Additional information you deem important (e.g. issue happens only occasionally):

Output of podman version:

Version:            1.1.2
RemoteAPI Version:  1
Go Version:         go1.12
Built:              Fri Mar  8 10:39:55 2019
OS/Arch:            linux/amd64

Output of podman info --debug:

debug:
  compiler: gc
  git commit: ""
  go version: go1.12
  podman version: 1.1.2
host:
  BuildahVersion: 1.7.1
  Conmon:
    package: Unknown
    path: /usr/bin/conmon
    version: 'conmon version , commit: 8fba206232c249a8fc4e2fac1469fb2fddbf5cf7'
  Distribution:
    distribution: arch
    version: unknown
  MemFree: 56151654400
  MemTotal: 67480981504
  OCIRuntime:
    package: Unknown
    path: /usr/bin/runc
    version: |-
      runc version 1.0.0-rc6
      commit: ccb5efd37fb7c86364786e9137e22948751de7ed-dirty
      spec: 1.0.1-dev
  SwapFree: 64135098368
  SwapTotal: 64135098368
  arch: amd64
  cpus: 8
  hostname: KI-P0695
  kernel: 5.0.4-arch1-1-ARCH
  os: linux
  rootless: true
  uptime: 5h 33m 35.72s (Approximately 0.21 days)
insecure registries:
  registries: []
registries:
  registries:
  - docker.io
  - registry.fedoraproject.org
  - quay.io
  - registry.access.redhat.com
  - registry.centos.org
store:
  ConfigFile: /home/balin/.config/containers/storage.conf
  ContainerStore:
    number: 43
  GraphDriverName: vfs
  GraphOptions: null
  GraphRoot: /home/balin/.local/share/containers/storage
  GraphStatus: {}
  ImageStore:
    number: 3
  RunRoot: /run/user/1000
  VolumePath: /home/balin/.local/share/containers/storage/volumes

Additional environment details (AWS, VirtualBox, physical, etc.):
ArchLinux with dnsmasq local dns caching and shorewall running - eqivalent docker install uses dedicated bridge. How does podman handled this? Can't find info in debugging this networking(?) problem.

[mheon]

You probably have a too-early slirp4netns - can you grab a slirp4netns --version and check if it's under 0.3.0?

[r-cheologist]

Wow, that was fast.
slirp4netns --version

slirp4netns version 0.2.1
commit: 1797e46728440e93f9229d5a34874befe00b4cab

Indeed. Need to check how to get that running in ArchLinux. No other way out?

[mheon]

Yeah, you need a 0.3.0 or later release to have port-forwarding support. The project lives at https://github.com/rootless-containers/slirp4netns and should be easy enough to build/install, though you might want to ping the package maintainers for Arch (assuming that you're using the official packages) and ask them to update.

[r-cheologist]

Ok. Just browsed by the github repo and saw this: Latest stable release: v0.2.X. Likely will make it hard to convince the maintainer(s) of the Arch package to switch. I'll try anyway. Many thanks for the amazingly fast reaction - was banging my head against the wall for days before bringing this triviality here - the error really could be more informative.

[mheon]

That might actually not be accurate anymore - @giuseppe can you confirm that Slirp 0.3.0 is what we'd consider stable?

(I suspect that we forgot to update the readme for the new slirp release)

[r-cheologist]

Oh - that would be wonderful.

[r-cheologist]

... but the tags imply pre-release state ...

[giuseppe]

yes, slirp 0.3 is what we need and consider it stable

[r-cheologist]

Can you please initiate an appropriate git tag/release and edit in the README.md?

[r-cheologist]

See rootless-containers/slirp4netns#81

[r-cheologist]

There's a proper 0.3 release now that has already percolated into the ArchLinux repos and I can now start the container - in principle this could be closed.

[jcberthon]

I am currently trying to write a tutorial on how to deploy certain network services easily using containers. The platform for the basis is Ubuntu 18.04 LTS. I wanted to try first Docker using the Snap packages but it is not mature enough, so it makes for over complicated setup and I miss the goal of "teaching how to deploy network services".

So I went to Podman. Installation was a bliss and the hello-world container run superbly (podman run docker.io/hello-world runs fine "out-of-the-box"), and by default it is root-less which is pretty cool.

However, when trying the next level of "hello world" it failed. I simply tried the following:

$ podman run --detach --name nginx -p 8080:80 docker.io/nginx:stable 
Trying to pull docker.io/nginx:stable...Getting image source signatures
Copying blob 8ca774778e85 done
Copying blob 0f23e58bd0b7 done
Copying blob 27833a3ba0a5 done
Copying config 295c7be079 done
Writing manifest to image destination
Storing signatures
Error: slirp4netns failed

And as I understand the above thread, this is simply because I have the wrong version of slirp4netns. When I installed podman from the Project Atomic PPA (as recommended for Ubuntu on your installation instruction), it also installed the dependency correctly but in version 0.2.1.

Could you upgrade the version available in the PPA for Ubuntu?

PS: of course I could install the package from source, but again like for Snap, it is getting me away from my main goal and make the installation more complex. However, I was impress by the rest of podman, and this is a small fix (uploading the new release of a package) as I understand it. So I hope this could be fixed soon.

PS2: perhaps a tutorial could be written about how networking works with Podman. A simple example like I was trying above to start with and then more advanced configurations. From the man page, I can see that there is also bridge and other networking. But nothing is currently explained how to use those options. I obviously tried to run using bridge mode but got a permission denied which I kind of expected because there is currently no bridge defined. But how to define it would be very interesting.

[r-cheologist]

I cannot help with the slirp4netns situation on Ubuntu/Debian, likely at the root of the problem described, BUT I would heartily like to support the proposal that a tutorial be made available that dissects and explains podman's networking. After having the correct slirp4netns version in my ArchLinux install, I get my containers to build and run, but how to get access to them from the base system via port forwarding is the challenge I'm currently working on ...

[rhatdan]

@giuseppe PTAL

[giuseppe]

I can write down some information on how the networking works with slirp4netns, what would be the best place for it? A blog post?

@AkihiroSuda have you already written down something like that?

[AkihiroSuda]

have you already written down something like that?

slirp4netns has the man page, but probably better to have a doc/blog in Podman's own repo/website for clarifying how Podman works with slirp4netns.

[edencorbin]

I'm a bit new to podman, but working through some basics, I'm running into:
Error: error from slirp4netns while setting up port redirection: map[desc:bad request: add_hostfwd: slirp_add_hostfwd failed]
I'm on KDE Neon (ubuntu 18.04)
I ran this:

podman pod create --name postgresql -p 5432 -p 9187
podman run -d --pod postgresql -e POSTGRES_PASSWORD=password postgres:latest

Then I get the error trying to run

podman pod start postgresql

Complete error:

ERRO[0000] "error starting some containers: container already exists" 
ERRO[0000] "a dependency of container e20523660139134d09bbc0cc95f22f15f8205c53485df6a39a729a74f63660fc failed to start: container state improper" 
Error: error from slirp4netns while setting up port redirection: map[desc:bad request: add_hostfwd: slirp_add_hostfwd failed]

I'm not sure how to troubleshoot this. Maybe the container is not properly setup for ubuntu 18.04 and I should build my own with buildah? That's what I'll try next, but would love any tips if someone has some insight.

[AkihiroSuda]

Seems the port is either less than 1024 or already taken by another process

[edencorbin]

Seems the port is either less than 1024 or already taken by another process

Yes that was it, my system installed postgres was also on default port 5432, my bad, user error.

[HypeFlighter]

i have this error because of wrong subuid/subgid allocated to your current user, make sure there is enough ids, check that first, then do to recreate user namespace.
if still not works, try to kill all podman and /usr/bin/conmon related processes.
this works for me.