--preserve-fds hangs podman

[rohanlean]

Is this a BUG REPORT or FEATURE REQUEST? (leave only one on its own line)

/kind bug

Description

Steps to reproduce the issue:

1. podman run --rm -it --name whatever busybox

2. New terminal: podman exec --preserve-fds=1 -ti whatever sh

Describe the results you received:
The podman exec command hangs for about two minutes, then exits with Error: container creation timeout: internal libpod error. While exec hangs, the initial session is unresponsive as well, but it resumes afterwards.

Describe the results you expected:
podman exec enters the container without any hangs.

Additional information you deem important (e.g. issue happens only occasionally):
I believe this also breaks the container support of Gnome Builder.

Output of podman version:

podman version 1.5.1

Output of podman info --debug:

debug:
  compiler: gc
  git commit: ""
  go version: go1.12.7
  podman version: 1.5.1
host:
  BuildahVersion: 1.10.1
  Conmon:
    package: podman-1.5.1-3.fc30.x86_64
    path: /usr/libexec/podman/conmon
    version: 'conmon version 2.0.0, commit: d728afa06cd2df86a27f32a4692c7099a56acc97-dirty'
  Distribution:
    distribution: fedora
    version: "30"
  MemFree: 11582738432
  MemTotal: 14639816704
  OCIRuntime:
    package: runc-1.0.0-93.dev.gitb9b6cc6.fc30.x86_64
    path: /usr/bin/runc
    version: |-
      runc version 1.0.0-rc8+dev
      commit: e3b4c1108f7d1bf0d09ab612ea09927d9b59b4e3
      spec: 1.0.1-dev
  SwapFree: 34359734272
  SwapTotal: 34359734272
  arch: amd64
  cpus: 8
  eventlogger: journald
  hostname: podstation
  kernel: 5.2.13-200.fc30.x86_64
  os: linux
  rootless: true
  uptime: 16m 17.29s
registries:
  blocked: null
  insecure: null
  search:
  - docker.io
  - registry.fedoraproject.org
  - quay.io
  - registry.access.redhat.com
  - registry.centos.org
store:
  ConfigFile: /var/home/poduser/.config/containers/storage.conf
  ContainerStore:
    number: 1
  GraphDriverName: overlay
  GraphOptions:
  - overlay.mount_program=/usr/bin/fuse-overlayfs
  GraphRoot: /var/home/poduser/.local/share/containers/storage
  GraphStatus:
    Backing Filesystem: xfs
    Native Overlay Diff: "false"
    Supports d_type: "true"
    Using metacopy: "false"
  ImageStore:
    number: 5
  RunRoot: /run/user/1000
  VolumePath: /var/home/poduser/.local/share/containers/storage/volumes

Package info (e.g. output of rpm -q podman or apt list podman):

podman-1.5.1-3.fc30.x86_64

Additional environment details (AWS, VirtualBox, physical, etc.):

Fedora Silverblue 30:

● ostree://fedora:fedora/30/x86_64/silverblue
                   Version: 30.20190912.0 (2019-09-12T20:42:50Z)
                    Commit: 6eab3e7e6afb27f6c3271c85242d744cba11287b39c08484b68d64cf85c01f9f

[mheon]

Damn. Did we miss this one when we implemented conmon exec? @haircommander

[haircommander]

I thought i implemented it but I forgot to verify it worked

[edsantiago]

Regression: this is back. I don't know if it's podman, conmon, runc, or something else:

tool	works	hangs
podman	1.7.0-3.fc30	1.8.0-4.fc30
conmon	2.0.2-1.fc30	2.0.13-1.fc30
runc	1.0.0-95.rc9.gitc1485a1.fc30	1.0.0-102.dev.gitdc9208a.fc30

Bizarrely, I dnf-upgraded the 'works' system to match all the versions in 'hangs' --- and everything still works. I know it's not just my systems because this is causing CI failures

Symptom:

# podman run --name test1 -d docker.io/library/alpine:latest top
39c9fd570fe7634acfd66140de740472ff611f7d6be6f302927fe38bdfc816ca
# podman exec --preserve-fds 1 test1 date
Sat Mar 28 14:55:36 UTC 2020
[hangs]

If I try a ^C - !! loop, sometimes it exits cleanly, sometimes it hangs with no output at all, but usually it just does the same thing: echo one line, hang.

[haircommander]

oddly, the conmon version in that CI run is 2.0.3 (according to the podman info tests), which is really out of date

I ran manually, and wasn't able to hit it with conmon 2.0.13 and podman 1.7.0, so I think this has to do with the exec rework and not conmon

[edsantiago]

Thanks for looking, @haircommander. I've filed new issue #5666 and will close this one.