Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

"podman start" not working #4522

Closed
sdarwin opened this issue Nov 15, 2019 · 21 comments
Closed

"podman start" not working #4522

sdarwin opened this issue Nov 15, 2019 · 21 comments
Labels
kind/bug Categorizes issue or PR as related to a bug. locked - please file new issue/PR Assist humans wanting to comment on an old issue or PR with locked comments.

Comments

@sdarwin
Copy link

sdarwin commented Nov 15, 2019

Is this a BUG REPORT or FEATURE REQUEST? (leave only one on its own line)

/kind bug

Description

On Ubuntu 18.04 in AWS, when stopping and starting a standard container, it seems to always show an error during "start".

Error: unable to start container "<container id>": container create failed (no logs from conmon): EOF

Steps to reproduce the issue:

Launch a small instance in AWS with Ubuntu 18.04.
Connect and switch to the root user.

sudo su -

Create install.sh:

set -x
sudo apt-get install -y \
  btrfs-tools \
  git \
  golang-go \
  go-md2man \
  iptables \
  libassuan-dev \
  libc6-dev \
  libdevmapper-dev \
  libglib2.0-dev \
  libgpgme-dev \
  libgpg-error-dev \
  libprotobuf-dev \
  libprotobuf-c0-dev \
  libseccomp-dev \
  libselinux1-dev \
  libsystemd-dev \
  pkg-config \
  runc \
  uidmap

sudo apt-get update -qq
sudo apt-get install -qq -y software-properties-common uidmap
sudo add-apt-repository -y ppa:projectatomic/ppa
sudo apt-get update -qq
sudo apt-get -qq -y install podman
sudo mkdir -p /etc/containers
echo -e "[registries.search]\nregistries = ['docker.io', 'quay.io']" | sudo tee /etc/containers/registries.conf

Run the installation:

chmod 755 install.sh
./install.sh

Download and run a container:

podman pull docker.io/nginx
podman run -d docker.io/library/nginx

View that it's running:

podman ps

Stop the container:

podman stop 571c0  (or podman stop <container id>)

The 'stop' is successful, and shows the id.

Start the container:

podman start 571c0 (or podman start <container id>)

Describe the results you received:

Error: unable to start container "571c0": container create failed (no logs from conmon): EOF

Describe the results you expected:

Just the id of the started container:

571c0

Additional information you deem important (e.g. issue happens only occasionally):

With those steps, the issue seems to be consistent and reproducible.

Output of podman version:

# podman version
Version:            1.6.2
RemoteAPI Version:  1
Go Version:         go1.10.4
OS/Arch:            linux/amd64

Output of podman info --debug:

# podman info --debug
debug:
  compiler: gc
  git commit: ""
  go version: go1.10.4
  podman version: 1.6.2
host:
  BuildahVersion: 1.11.3
  CgroupVersion: v1
  Conmon:
    package: 'conmon: /usr/bin/conmon'
    path: /usr/bin/conmon
    version: 'conmon version 2.0.3, commit: unknown'
  Distribution:
    distribution: ubuntu
    version: "18.04"
  MemFree: 1668878336
  MemTotal: 4135120896
  OCIRuntime:
    name: runc
    package: 'containerd.io: /usr/bin/runc'
    path: /usr/bin/runc
    version: |-
      runc version 1.0.0-rc8+dev
      commit: 3e425f80a8c931f88e6d94a8c831b9d5aa481657
      spec: 1.0.1-dev
  SwapFree: 0
  SwapTotal: 0
  arch: amd64
  cpus: 2
  eventlogger: journald
  hostname: ip-172-31-13-241
  kernel: 4.15.0-1051-aws
  os: linux
  rootless: false
  uptime: 22m 1.82s
registries:
  blocked: null
  insecure: null
  search:
  - docker.io
  - quay.io
store:
  ConfigFile: /etc/containers/storage.conf
  ContainerStore:
    number: 1
  GraphDriverName: overlay
  GraphOptions: {}
  GraphRoot: /var/lib/containers/storage
  GraphStatus:
    Backing Filesystem: extfs
    Native Overlay Diff: "true"
    Supports d_type: "true"
    Using metacopy: "false"
  ImageStore:
    number: 1
  RunRoot: /var/run/containers/storage
  VolumePath: /var/lib/containers/storage/volumes

Package info (e.g. output of rpm -q podman or apt list podman):

# apt list podman
Listing... Done
podman/bionic,now 1.6.2-1~ubuntu18.04~ppa1 amd64 [installed]

Additional environment details (AWS, VirtualBox, physical, etc.):

AWS

@openshift-ci-robot openshift-ci-robot added the kind/bug Categorizes issue or PR as related to a bug. label Nov 15, 2019
@mheon
Copy link
Member

mheon commented Nov 15, 2019

Can you run with --log-level=debug and paste the result, as well as any logs in syslog from conmon?

@sdarwin
Copy link
Author

sdarwin commented Nov 15, 2019

@mheon - very good idea - the conmon logs in /var/log/syslog shows the error:

Nov 15 22:50:24 ip-172-31-13-241 conmon: conmon 8e7b417faf5b76195947 <ninfo>: attach sock path: /var/run/libpod/socket/8e7b417faf5b761959470e1d08a460d4ebb7ae525d8eee88a7812e5e79475679/attach
Nov 15 22:50:24 ip-172-31-13-241 conmon: conmon 8e7b417faf5b76195947 <ninfo>: addr{sun_family=AF_UNIX, sun_path=/var/run/libpod/socket/8e7b417faf5b761959470e1d08a460d4ebb7ae525d8eee88a7812e5e79475679/attach}
Nov 15 22:50:24 ip-172-31-13-241 conmon: conmon 8e7b417faf5b76195947 <ninfo>: terminal_ctrl_fd: 16
Nov 15 22:50:24 ip-172-31-13-241 conmon: conmon 8e7b417faf5b76195947 <error>: Failed to mkfifo at /var/lib/containers/storage/overlay-containers/8e7b417faf5b761959470e1d08a460d4ebb7ae525d8eee88a7812e5e79475679/userdata/winsz: File exists

it says "File exists".

for completeness, here is the debug output of "podman start" also:

# podman start --log-level=debug 8e7
DEBU[0000] using conmon: "/usr/bin/conmon"              
DEBU[0000] Initializing boltdb state at /var/lib/containers/storage/libpod/bolt_state.db 
DEBU[0000] Using graph driver overlay                   
DEBU[0000] Using graph root /var/lib/containers/storage 
DEBU[0000] Using run root /var/run/containers/storage   
DEBU[0000] Using static dir /var/lib/containers/storage/libpod 
DEBU[0000] Using tmp dir /var/run/libpod                
DEBU[0000] Using volume path /var/lib/containers/storage/volumes 
DEBU[0000] Set libpod namespace to ""                   
DEBU[0000] [graphdriver] trying provided driver "overlay" 
DEBU[0000] cached value indicated that overlay is supported 
DEBU[0000] cached value indicated that metacopy is not being used 
DEBU[0000] cached value indicated that native-diff is usable 
DEBU[0000] backingFs=extfs, projectQuotaSupported=false, useNativeDiff=true, usingMetacopy=false 
DEBU[0000] Initializing event backend journald          
DEBU[0000] using runtime "/usr/bin/runc"                
WARN[0000] Error initializing configured OCI runtime crun: no valid executable found for OCI runtime crun: invalid argument 
INFO[0000] Found CNI network podman (type=bridge) at /etc/cni/net.d/87-podman-bridge.conflist 
DEBU[0000] overlay: mount_data=lowerdir=/var/lib/containers/storage/overlay/l/KPI72ZD43ORLU4PQBNTLGRGN7H:/var/lib/containers/storage/overlay/l/RCXFOQ5ZL5GLPMNTWGC2I6WBM3:/var/lib/containers/storage/overlay/l/EPS6NXK3IVOVGAQDSO76LGUASR,upperdir=/var/lib/containers/storage/overlay/5e913f3998bab492c31746f4c1d2fb3bed4f346ba12647a2caad3446c9c8039e/diff,workdir=/var/lib/containers/storage/overlay/5e913f3998bab492c31746f4c1d2fb3bed4f346ba12647a2caad3446c9c8039e/work 
DEBU[0000] mounted container "8e7b417faf5b761959470e1d08a460d4ebb7ae525d8eee88a7812e5e79475679" at "/var/lib/containers/storage/overlay/5e913f3998bab492c31746f4c1d2fb3bed4f346ba12647a2caad3446c9c8039e/merged" 
DEBU[0000] Created root filesystem for container 8e7b417faf5b761959470e1d08a460d4ebb7ae525d8eee88a7812e5e79475679 at /var/lib/containers/storage/overlay/5e913f3998bab492c31746f4c1d2fb3bed4f346ba12647a2caad3446c9c8039e/merged 
DEBU[0000] Made network namespace at /var/run/netns/cni-0631d181-0a0d-0cb2-1516-0e2ccbad1a85 for container 8e7b417faf5b761959470e1d08a460d4ebb7ae525d8eee88a7812e5e79475679 
INFO[0000] Got pod network &{Name:interesting_lederberg Namespace:interesting_lederberg ID:8e7b417faf5b761959470e1d08a460d4ebb7ae525d8eee88a7812e5e79475679 NetNS:/var/run/netns/cni-0631d181-0a0d-0cb2-1516-0e2ccbad1a85 Networks:[] RuntimeConfig:map[podman:{IP: PortMappings:[] Bandwidth:<nil> IpRanges:[]}]} 
INFO[0000] About to add CNI network cni-loopback (type=loopback) 
INFO[0000] Got pod network &{Name:interesting_lederberg Namespace:interesting_lederberg ID:8e7b417faf5b761959470e1d08a460d4ebb7ae525d8eee88a7812e5e79475679 NetNS:/var/run/netns/cni-0631d181-0a0d-0cb2-1516-0e2ccbad1a85 Networks:[] RuntimeConfig:map[podman:{IP: PortMappings:[] Bandwidth:<nil> IpRanges:[]}]} 
INFO[0000] About to add CNI network podman (type=bridge) 
DEBU[0000] [0] CNI result: Interfaces:[{Name:cni-podman0 Mac:ce:da:85:19:99:d2 Sandbox:} {Name:veth5bc236d0 Mac:2a:a1:47:79:f5:6c Sandbox:} {Name:eth0 Mac:5a:94:c2:a6:f8:ed Sandbox:/var/run/netns/cni-0631d181-0a0d-0cb2-1516-0e2ccbad1a85}], IP:[{Version:4 Interface:0xc42052cca8 Address:{IP:10.88.0.10 Mask:ffff0000} Gateway:10.88.0.1}], Routes:[{Dst:{IP:0.0.0.0 Mask:00000000} GW:<nil>}], DNS:{Nameservers:[] Domain: Search:[] Options:[]} 
INFO[0000] No non-localhost DNS nameservers are left in resolv.conf. Using default external servers: [nameserver 8.8.8.8 nameserver 8.8.4.4] 
INFO[0000] IPv6 enabled; Adding default IPv6 external servers: [nameserver 2001:4860:4860::8888 nameserver 2001:4860:4860::8844] 
DEBU[0000] /etc/system-fips does not exist on host, not mounting FIPS mode secret 
DEBU[0000] Setting CGroups for container 8e7b417faf5b761959470e1d08a460d4ebb7ae525d8eee88a7812e5e79475679 to machine.slice:libpod:8e7b417faf5b761959470e1d08a460d4ebb7ae525d8eee88a7812e5e79475679 
DEBU[0000] reading hooks from /usr/share/containers/oci/hooks.d 
DEBU[0000] reading hooks from /etc/containers/oci/hooks.d 
DEBU[0000] Created OCI spec for container 8e7b417faf5b761959470e1d08a460d4ebb7ae525d8eee88a7812e5e79475679 at /var/lib/containers/storage/overlay-containers/8e7b417faf5b761959470e1d08a460d4ebb7ae525d8eee88a7812e5e79475679/userdata/config.json 
DEBU[0000] /usr/bin/conmon messages will be logged to syslog 
DEBU[0000] running conmon: /usr/bin/conmon               args="[--api-version 1 -s -c 8e7b417faf5b761959470e1d08a460d4ebb7ae525d8eee88a7812e5e79475679 -u 8e7b417faf5b761959470e1d08a460d4ebb7ae525d8eee88a7812e5e79475679 -r /usr/bin/runc -b /var/lib/containers/storage/overlay-containers/8e7b417faf5b761959470e1d08a460d4ebb7ae525d8eee88a7812e5e79475679/userdata -p /var/run/containers/storage/overlay-containers/8e7b417faf5b761959470e1d08a460d4ebb7ae525d8eee88a7812e5e79475679/userdata/pidfile -l k8s-file:/var/lib/containers/storage/overlay-containers/8e7b417faf5b761959470e1d08a460d4ebb7ae525d8eee88a7812e5e79475679/userdata/ctr.log --exit-dir /var/run/libpod/exits --socket-dir-path /var/run/libpod/socket --log-level debug --syslog --conmon-pidfile /var/run/containers/storage/overlay-containers/8e7b417faf5b761959470e1d08a460d4ebb7ae525d8eee88a7812e5e79475679/userdata/conmon.pid --exit-command /usr/bin/podman --exit-command-arg --root --exit-command-arg /var/lib/containers/storage --exit-command-arg --runroot --exit-command-arg /var/run/containers/storage --exit-command-arg --log-level --exit-command-arg error --exit-command-arg --cgroup-manager --exit-command-arg systemd --exit-command-arg --tmpdir --exit-command-arg /var/run/libpod --exit-command-arg --runtime --exit-command-arg runc --exit-command-arg --storage-driver --exit-command-arg overlay --exit-command-arg --events-backend --exit-command-arg journald --exit-command-arg container --exit-command-arg cleanup --exit-command-arg 8e7b417faf5b761959470e1d08a460d4ebb7ae525d8eee88a7812e5e79475679]"
INFO[0000] Running conmon under slice machine.slice and unitName libpod-conmon-8e7b417faf5b761959470e1d08a460d4ebb7ae525d8eee88a7812e5e79475679.scope 
WARN[0000] Failed to add conmon to systemd sandbox cgroup: Unit libpod-conmon-8e7b417faf5b761959470e1d08a460d4ebb7ae525d8eee88a7812e5e79475679.scope already exists. 
DEBU[0000] Cleaning up container 8e7b417faf5b761959470e1d08a460d4ebb7ae525d8eee88a7812e5e79475679 
DEBU[0000] Tearing down network namespace at /var/run/netns/cni-0631d181-0a0d-0cb2-1516-0e2ccbad1a85 for container 8e7b417faf5b761959470e1d08a460d4ebb7ae525d8eee88a7812e5e79475679 
INFO[0000] Got pod network &{Name:interesting_lederberg Namespace:interesting_lederberg ID:8e7b417faf5b761959470e1d08a460d4ebb7ae525d8eee88a7812e5e79475679 NetNS:/var/run/netns/cni-0631d181-0a0d-0cb2-1516-0e2ccbad1a85 Networks:[] RuntimeConfig:map[podman:{IP: PortMappings:[] Bandwidth:<nil> IpRanges:[]}]} 
INFO[0000] About to del CNI network podman (type=bridge) 
DEBU[0000] unmounted container "8e7b417faf5b761959470e1d08a460d4ebb7ae525d8eee88a7812e5e79475679" 
ERRO[0000] unable to start container "8e7": container create failed (no logs from conmon): EOF 

@baude
Copy link
Member

baude commented Nov 17, 2019

@sdarwin can we also get a podman info ? Might help understanding all the versions involved. also @haircommander ptal, thoughts?

@mheon
Copy link
Member

mheon commented Nov 17, 2019 via email

@sdarwin
Copy link
Author

sdarwin commented Nov 17, 2019

I think the PR to clean up winsz might fix this as well

Yes.

can we also get a podman info ?

# podman info
host:
  BuildahVersion: 1.11.3
  CgroupVersion: v1
  Conmon:
    package: 'conmon: /usr/bin/conmon'
    path: /usr/bin/conmon
    version: 'conmon version 2.0.3, commit: unknown'
  Distribution:
    distribution: ubuntu
    version: "18.04"
  MemFree: 703287296
  MemTotal: 2088759296
  OCIRuntime:
    name: runc
    package: 'cri-o-runc: /usr/lib/cri-o-runc/sbin/runc'
    path: /usr/lib/cri-o-runc/sbin/runc
    version: 'runc version spec: 1.0.1-dev'
  SwapFree: 0
  SwapTotal: 0
  arch: amd64
  cpus: 1
  eventlogger: journald
  hostname: jenkins
  kernel: 4.15.0-1054-aws
  os: linux
  rootless: false
  uptime: 4m 2.76s
registries:
  blocked: null
  insecure: null
  search: null
store:
  ConfigFile: /etc/containers/storage.conf
  ContainerStore:
    number: 8
  GraphDriverName: overlay
  GraphOptions: {}
  GraphRoot: /var/lib/containers/storage
  GraphStatus:
    Backing Filesystem: extfs
    Native Overlay Diff: "true"
    Supports d_type: "true"
    Using metacopy: "false"
  ImageStore:
    number: 5
  RunRoot: /var/run/containers/storage
  VolumePath: /var/lib/containers/storage/volumes

@haircommander
Copy link
Collaborator

#4521 merged, are you able to verify with it @sdarwin ?

@sdarwin
Copy link
Author

sdarwin commented Nov 17, 2019

Fixed.

@sdarwin sdarwin closed this as completed Nov 17, 2019
@DaanDeMeyer
Copy link

Any estimate on when this fix will be released? Not being able to start containers a second time after stopping them is kind of hard to work around.

@vrothberg
Copy link
Member

@mheon will know when the next release happens.

@rhatdan
Copy link
Member

rhatdan commented Nov 25, 2019

We should have podman 1.6.3 released ASAP or something newer. 1.6.3 is already set up to be used on RHEL8.2, I believe.

@mheon
Copy link
Member

mheon commented Nov 25, 2019

I'll try and get a 1.6.4 out this week for 8.2 and Fedora. We need a few more patches that haven't been written yet for CGroups v2.

@ghost
Copy link

ghost commented Dec 3, 2019

I am getting the same error on podman container start ${container_name} on version 1.6.3

The log obtained from podman --log-level=debug container start rdis 2> podman.log is podman.log and the command used to create the container was

podman run --name rdis -p 6379:6379 -e REDIS_PASSWORD=${REDIS_PASSWORD} -d bitnami/redis:latest

I am also not able to start any of my other containers (these were all previously working containers) the output of podman info is podman_info.log
Any help or work around will be extremely appreciated. Sorry in advance if this is the wrong place for this, I did not want to open a new issue for this because this thread was fairly new and seemed relevant.

@miquecg
Copy link

miquecg commented Dec 5, 2019

This is also happening to me on Arch Linux with Podman 1.6.3 and conmon 2.0.3.

Steps to reproduce

➜  # ~ podman run -it alpine
/ # exit

➜  # ~ podman ps -a
CONTAINER ID  IMAGE                            COMMAND  CREATED         STATUS                     PORTS  NAMES
8946589d4831  docker.io/library/alpine:latest  /bin/sh  49 seconds ago  Exited (0) 47 seconds ago         beautiful_mendel

➜  # ~ podman start --log-level=debug 8946589d4831
(log output truncated)
ERRO[0000] unable to start container "8946589d4831": container create failed (no logs from conmon): EOF

podman info

host:
  BuildahVersion: 1.12.0-dev
  CgroupVersion: v1
  Conmon:
    package: Unknown
    path: /usr/bin/conmon
    version: 'conmon version 2.0.3, commit: eb5fa88c26fde5ce1e3f8a1d2a8a9498b2d7dbe6'
  Distribution:
    distribution: arch
    version: unknown
  MemFree: 3226333184
  MemTotal: 8313151488
  OCIRuntime:
    name: runc
    package: Unknown
    path: /usr/bin/runc
    version: |-
      runc version 1.0.0-rc9
      commit: d736ef14f0288d6993a1845745d6756cfc9ddd5a
      spec: 1.0.1-dev
  SwapFree: 4294963200
  SwapTotal: 4294963200
  arch: amd64
  cpus: 4
  eventlogger: journald
  hostname: <hostname>
  kernel: 5.4.1-arch1-1
  os: linux
  rootless: false
  uptime: 5h 36m 20.37s (Approximately 0.21 days)
registries:
  blocked: null
  insecure: null
  search:
  - docker.io
  - registry.fedoraproject.org
  - quay.io
  - registry.access.redhat.com
  - registry.centos.org
store:
  ConfigFile: /etc/containers/storage.conf
  ContainerStore:
    number: 3
  GraphDriverName: overlay
  GraphOptions:
    overlay.mountopt: nodev
  GraphRoot: /var/lib/containers/storage
  GraphStatus:
    Backing Filesystem: extfs
    Native Overlay Diff: "false"
    Supports d_type: "true"
    Using metacopy: "true"
  ImageStore:
    number: 1
  RunRoot: /var/run/containers/storage
  VolumePath: /var/lib/containers/storage/volumes

Complete log output.

@mheon
Copy link
Member

mheon commented Dec 5, 2019

We are preparing for a 1.6.4 release soon. That will include the patch that resolves the issue.

@ghost
Copy link

ghost commented Dec 6, 2019

Thank you for the update! for now I can create a new container, mount the same volumes (this is using postgres:alpine image from docker) and run it just fine — have not tried stopping it, I just hibernate the system between sessions and am able to get work done.

@delight
Copy link

delight commented Jan 22, 2020

Would someone please be so kind to publish a new release to the ppa for *buntu ?
PPA is still on the buggy 1.6.2 (1.6.2-1ubuntu18.04ppa1) version

https://launchpad.net/~projectatomic/+archive/ubuntu/ppa

Thnx

@TomSweeneyRedHat
Copy link
Member

ping @lsm5 thoughts?

@Enrico204
Copy link

@vrothberg
Copy link
Member

Yes, please refer to https://podman.io/getting-started/installation for the updated installation instructions.

@MahmoudChik
Copy link

Hi,
I have the same problem on ubuntu 16.04.
"contrainer create failed (no logs from common): EOF"

Any correction ?

@vrothberg
Copy link
Member

The issue is over three years old. Please open a new one. Given Ubuntu 16.04 is very old, you may be using an outdated version of Podman.

@github-actions github-actions bot added the locked - please file new issue/PR Assist humans wanting to comment on an old issue or PR with locked comments. label Sep 6, 2023
@github-actions github-actions bot locked as resolved and limited conversation to collaborators Sep 6, 2023
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
kind/bug Categorizes issue or PR as related to a bug. locked - please file new issue/PR Assist humans wanting to comment on an old issue or PR with locked comments.
Projects
None yet
Development

No branches or pull requests