Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Rebooted, dangling file in /var/lib/cni/networks/podman prevents container starting #3759

Closed
space88man opened this issue Aug 8, 2019 · 11 comments
Assignees
Labels

Comments

@space88man
Copy link

@space88man space88man commented Aug 8, 2019

Is this a BUG REPORT or FEATURE REQUEST? (leave only one on its own line)

/kind bug

Description

I did a systemctl reboot; encountered a orphan /var/lib/cni/networks/podman/10.88.0.20 file
which prevented the container from starting.

ERRO[0000] Error adding network: failed to allocate for range 0: requested IP address 10.88.0.20 is not available in range set 10.88.0.1-10.88.255.254 
ERRO[0000] Error while adding pod to CNI network "podman": failed to allocate for range 0: requested IP address 10.88.0.20 is not available in range set 10.88.0.1-10.88.255.254 
Error: unable to start container "freeswitch-init_1": error configuring network namespace for container 7690a7bc4960b76799d906dbc07a32d174a5e75581886349d3604d455e093cf7: failed to allocate for range 0: requested IP address 10.88.0.20 is not available in range set 10.88.0.1-10.88.255.254

Steps to reproduce the issue:

  1. Upgraded Fedora 30 host, rebooted, tried to start a container

Describe the results you received:

INFO[0000] Found CNI network podman (type=bridge) at /etc/cni/net.d/87-podman-bridge.conflist 
DEBU[0000] Made network namespace at /var/run/netns/cni-76dce0ef-599f-b625-7710-a7eeef4159e5 for container 7690a7bc4960b76799d906dbc07a32d174a5e75581886349d3604d455e093cf7 
INFO[0000] Got pod network &{Name:freeswitch-init_1 Namespace:freeswitch-init_1 ID:7690a7bc4960b76799d906dbc07a32d174a5e75581886349d3604d455e093cf7 NetNS:/var/run/netns/cni-76dce0ef-599f-b625-7710-a7eeef4159e5 PortMappings:[] Networks:[podman] NetworkConfig:map[podman:{IP:10.88.0.20}]} 
INFO[0000] About to add CNI network cni-loopback (type=loopback) 
DEBU[0000] overlay: mount_data=nodev,metacopy=on,lowerdir=/var/lib/containers/storage/overlay/l/Z6S3A3PN6PO5DCD4E3FZZOFLO6:/var/lib/containers/storage/overlay/l/II26Q5SUSAWGOVHLKY3MKC7GNU,upperdir=/var/lib/containers/storage/overlay/8cc72886f3925230caa200d3bce8b24599897b0c64ca5a6a4d6d56871f118d4d/diff,workdir=/var/lib/containers/storage/overlay/8cc72886f3925230caa200d3bce8b24599897b0c64ca5a6a4d6d56871f118d4d/work,context="system_u:object_r:container_file_t:s0:c362,c945" 
DEBU[0000] mounted container "7690a7bc4960b76799d906dbc07a32d174a5e75581886349d3604d455e093cf7" at "/var/lib/containers/storage/overlay/8cc72886f3925230caa200d3bce8b24599897b0c64ca5a6a4d6d56871f118d4d/merged" 
DEBU[0000] Created root filesystem for container 7690a7bc4960b76799d906dbc07a32d174a5e75581886349d3604d455e093cf7 at /var/lib/containers/storage/overlay/8cc72886f3925230caa200d3bce8b24599897b0c64ca5a6a4d6d56871f118d4d/merged 
INFO[0000] Got pod network &{Name:freeswitch-init_1 Namespace:freeswitch-init_1 ID:7690a7bc4960b76799d906dbc07a32d174a5e75581886349d3604d455e093cf7 NetNS:/var/run/netns/cni-76dce0ef-599f-b625-7710-a7eeef4159e5 PortMappings:[] Networks:[podman] NetworkConfig:map[podman:{IP:10.88.0.20}]} 
INFO[0000] About to add CNI network podman (type=bridge) 
ERRO[0000] Error adding network: failed to allocate for range 0: requested IP address 10.88.0.20 is not available in range set 10.88.0.1-10.88.255.254 
ERRO[0000] Error while adding pod to CNI network "podman": failed to allocate for range 0: requested IP address 10.88.0.20 is not available in range set 10.88.0.1-10.88.255.254 
DEBU[0000] Network is already cleaned up, skipping...   
DEBU[0000] unmounted container "7690a7bc4960b76799d906dbc07a32d174a5e75581886349d3604d455e093cf7" 
DEBU[0000] Cleaning up container 7690a7bc4960b76799d906dbc07a32d174a5e75581886349d3604d455e093cf7 
DEBU[0000] Network is already cleaned up, skipping...   
DEBU[0000] Container 7690a7bc4960b76799d906dbc07a32d174a5e75581886349d3604d455e093cf7 storage is already unmounted, skipping... 
ERRO[0000] unable to start container "freeswitch-init_1": error configuring network namespace for container 7690a7bc4960b76799d906dbc07a32d174a5e75581886349d3604d455e093cf7: failed to allocate for range 0: requested IP address 10.88.0.20 is not available in range set 10.88.0.1-10.88.255.254 

Describe the results you expected:
Container starts

Additional information you deem important (e.g. issue happens only occasionally):

Output of podman version:

Version:            1.4.4
RemoteAPI Version:  4
Go Version:         go1.12.7
OS/Arch:            linux/amd64

Output of podman info --debug:

debug:
  compiler: gc
  git commit: ""
  go version: go1.12.7
  podman version: 1.4.4
host:
  BuildahVersion: 1.9.0
  Conmon:
    package: podman-1.4.4-4.fc30.x86_64
    path: /usr/libexec/podman/conmon
    version: 'conmon version 1.0.0-dev, commit: 164df8af4e62dc759c312eab4b97ea9fb6b5f1fc'
  Distribution:
    distribution: fedora
    version: "30"
  MemFree: 7664599040
  MemTotal: 8340746240
  OCIRuntime:
    package: runc-1.0.0-93.dev.gitb9b6cc6.fc30.x86_64
    path: /usr/bin/runc
    version: |-
      runc version 1.0.0-rc8+dev
      commit: e3b4c1108f7d1bf0d09ab612ea09927d9b59b4e3
      spec: 1.0.1-dev
  SwapFree: 4294963200
  SwapTotal: 4294963200
  arch: amd64
  cpus: 4
  hostname: containers.localdomain
  kernel: 5.2.5-200.fc30.x86_64
  os: linux
  rootless: false
  uptime: 7m 27.03s
registries:
  blocked: null
  insecure: null
  search:
  - docker.io
  - registry.fedoraproject.org
  - quay.io
  - registry.access.redhat.com
  - registry.centos.org
store:
  ConfigFile: /etc/containers/storage.conf
  ContainerStore:
    number: 3
  GraphDriverName: overlay
  GraphOptions:
  - overlay.mountopt=nodev,metacopy=on
  GraphRoot: /var/lib/containers/storage
  GraphStatus:
    Backing Filesystem: xfs
    Native Overlay Diff: "false"
    Supports d_type: "true"
    Using metacopy: "true"
  ImageStore:
    number: 6
  RunRoot: /var/run/containers/storage
  VolumePath: /var/lib/containers/storage/volumes

Additional environment details (AWS, VirtualBox, physical, etc.):
In /var/lib/cni/networks/podman I have:

  1. 10.88.0.20: contains the ID of the container
  2. last_reserved_ip.0: contains 10.88.0.20
  3. lock

The guard file 10.88.0.20 must have been left behind from the reboot. After deleting the file, the container could start.

@space88man space88man changed the title requested IP address 10.88.0.20 is not available in range set 10.88.0.1-10.88.255.254 Rebooted, dangling file in /var/lib/cni/networks/podman prevents container starting Aug 8, 2019
@mheon

This comment has been minimized.

Copy link
Collaborator

@mheon mheon commented Aug 8, 2019

@space88man

This comment has been minimized.

Copy link
Author

@space88man space88man commented Aug 8, 2019

No; this was after a graceful reboot.

@mheon

This comment has been minimized.

Copy link
Collaborator

@mheon mheon commented Aug 8, 2019

@mccv1r0 Can you take a look at this? it looks like CNI didn't clean up address reservations after a clean reboot?

@dcbw

This comment has been minimized.

Copy link
Contributor

@dcbw dcbw commented Aug 8, 2019

@mheon the runtime is solely responsible for calling CNI DEL for every container that is no longer running but has not had DEL called on it. I know that podman/CRIO keep a cache of containers somewhere on-disk. When that cache is reconciled with what is actually running when CRIO/podman start, they need to call CNI DEL on every container in that cache list that is not currently running to allow the network plugin to clean up.

@rhatdan

This comment has been minimized.

Copy link
Member

@rhatdan rhatdan commented Aug 8, 2019

@dcbw Is this something we could configure to put these files on a tmpfs, so that we don't have to cleanup after a reboot?

@dcbw

This comment has been minimized.

Copy link
Contributor

@dcbw dcbw commented Aug 8, 2019

@rhatdan you have no idea what kind of information the network plugin has to clean up, so you have to tell the network plugin to clean up. Which is CNI DEL.

If your container is no longer running, and it wasn't given a CNI DEL, you must call CNI DEL to clean up.

@dcbw

This comment has been minimized.

Copy link
Contributor

@dcbw dcbw commented Aug 8, 2019

@mheon @rhatdan you have a container database:

// save container state to the database
func (c *Container) save() error {
	if err := c.runtime.state.SaveContainer(c); err != nil {
		return errors.Wrapf(err, "error saving container %s state", c.ID())
	}
	return nil
}

presumably that doesn't get blown away on restart. So after the next time podman runs (for any pod) it'll need to reconcile that database list with what's actually running and prune out the old stuff and call CNI DEL on those that aren't running. Or something like that.

@mheon

This comment has been minimized.

Copy link
Collaborator

@mheon mheon commented Aug 8, 2019

We do actually blow away most database state on reboot, on the assumption that none of it survived the reboot - what was running no longer is, what was mounted no longer is. It may be possible to work a CNI DEL into the process of refreshing the state post-reboot - I'll investigate.

@mheon

This comment has been minimized.

Copy link
Collaborator

@mheon mheon commented Aug 8, 2019

Alright, this bit promises to be complicated.

We wipe container state very early in the refresh process, because we can't retrieve containers otherwise - the state is not sane because of the restart, so all of our validation fails. The runtime never touches a state with the CNI result available after a reboot - it's gone by the time we have the ability to actually get containers.

We can change this to preserve the cached CNI result, but some bits of what we pass to OCICNI as part of TeardownPod are not going to survive - network namespace path, for example, is definitely gone, and I don't see a way of preserving it.

@mheon

This comment has been minimized.

Copy link
Collaborator

@mheon mheon commented Aug 8, 2019

Hmmm. If the network namespace path isn't strictly mandatory - I think we can probably call OCICNI's TeardownPod without preserving the old CNI result.

mheon added a commit to mheon/libpod that referenced this issue Sep 23, 2019
CNI expects that a DELETE be run before re-creating container
networks. If a reboot occurs quickly enough that containers can't
stop and clean up, that DELETE never happens, and Podman
currently wipes the old network info and thinks the state has
been entirely cleared. Unfortunately, that may not be the case on
the CNI side. Some things - like IP address reservations - may
not have been cleared.

To solve this, manually re-run CNI Delete on refresh. If the
container has already been deleted this seems harmless. If not,
it should clear lingering state.

Fixes: containers#3759

Signed-off-by: Matthew Heon <matthew.heon@pm.me>
@mheon

This comment has been minimized.

Copy link
Collaborator

@mheon mheon commented Sep 23, 2019

Should be fixed by #4086

mheon added a commit to mheon/libpod that referenced this issue Sep 24, 2019
CNI expects that a DELETE be run before re-creating container
networks. If a reboot occurs quickly enough that containers can't
stop and clean up, that DELETE never happens, and Podman
currently wipes the old network info and thinks the state has
been entirely cleared. Unfortunately, that may not be the case on
the CNI side. Some things - like IP address reservations - may
not have been cleared.

To solve this, manually re-run CNI Delete on refresh. If the
container has already been deleted this seems harmless. If not,
it should clear lingering state.

Fixes: containers#3759

Signed-off-by: Matthew Heon <matthew.heon@pm.me>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

5 participants
You can’t perform that action at this time.