make `podman system reset` handle external containers

[dustymabe]

/kind feature

Description

If I have "external" containers (like ones created by buildah or podman build then podman system reset will error:

[core@pcengines ~]$ podman system reset

WARNING! This will remove:
        - all containers
        - all pods
        - all images
        - all build cache
Are you sure you want to continue? [y/N] y
ERRO[0002] 31 errors occurred:
	* Image used by 769d78b7025b2b92072d2875f92474474dd36284a1d1cdb0480365b7ec534cc2: image is in use by a container
	* Image used by 4f834bf4657dbb9652a3e3be17b8595f32b5af28c2e2ebd4dcceeafc779449b9: image is in use by a container
	* Image used by 381b0e9504ed095875ac2093e67c14ce37edeb2881ecea0c4e49edb06a8f276b: image is in use by a container
	* Image used by 9fc744b584ce8ffcb26f0d593ecd1201d3fb17bbdd534607af7f56830800590b: image is in use by a container
	* Image used by ed0e6ed6ae5f0edcf6f7843f2903e26595a7efb64ddd31cd07617c05522c3bc5: image is in use by a container
	* Image used by 30748298264d7ccccf43ec43b875ce2189111dd47cd0947559f1ee0a7c7f9900: image is in use by a container
	* Image used by c949a4f5154931d24803966885bb2b01a8648470e0620876fb0e5dfc7de050a5: image is in use by a container
	* Image used by aa4bc49440ea59a757c04e22d410a62adf4ff062fe4a79e450067784cfdf23e3: image is in use by a container
	* Image used by d05b12db0a97d74a27558cb596091cd1547dc6303b718e97c9c2a8f87e0a1cc3: image is in use by a container
	* Image used by 00e39cc9be1d49e5e62313b2f3f62dfaa026303decb5fc9f2ae69004e597657d: image is in use by a container
	* Image used by ca865c272f46497d98f06706755978e7e822ce8263d899b604b11101b6cbda7c: image is in use by a container
	* Image used by 932c6ea5f622e8812ea3b471687990f5517aa181a6e42570f012fc25cbbc4b0d: image is in use by a container
	* Image used by ab9dc123c9c192b5b775887a252c6d7711ab0b50c74dacfaf50c64f1a458bb27: image is in use by a container
	* Image used by d966e95f4509fab473d1e5d373b0ef5e17d8fb313a6547306183c0b9b0c7bce1: image is in use by a container
	* Image used by b8b3bc88554c410a53566dc1f82d0344346a1dbf9af678ca96dd3266959ca2f6: image is in use by a container
	* Image used by a7159df01aaf47b0cd9d5510518ce4444d9695e2ba26f0fca7f8e03420455977: image is in use by a container
	* Image used by 4c5f25f44b1adca81b49a12d6bf4ee3ef4f59f6eed6c3ff0226551a3b6db62c9: image is in use by a container
	* Image used by 121986cc0d5edb829742521e1d4ff5ea60ddb0947ca8ddb1fb10941f940846f9: image is in use by a container
	* Image used by f4ab832832ad024cefefd50c6193a3a1ea1f603a3804758b1b7b486ac456a452: image is in use by a container
	* Image used by f8bbf11d04131a48621415ecf40a465d9132f846df2e319ba9037f4403ab44a4: image is in use by a container
	* Image used by 30a9613479d44daec05b62ede6a0d8876b817d3fde5a4b09631c44b8c60a5f9b: image is in use by a container
	* Image used by 119265f1cfe8a6ebed4a0399cf1c9e7d03cb02eb1c01c88b706f8a2a43373656: image is in use by a container
	* Image used by dea8f88952aebfdf3b4e183cfc8252873ec0cf5bba67f82105e92d6ff6cc1490: image is in use by a container
	* Image used by 4425ea2feeb99032952b3c0ad98a409b1ba964f774fb1b5c6f6b1f5fa58a6de7: image is in use by a container
	* Image used by ad3c42b69c9c7f2ad70e897b440ea57327f2bce45900f819bf0f9c142305d9f6: image is in use by a container
	* Image used by 11c5bab4545a3927bb06da633fa5d0a0226e4afd3e030f9cfcccda3b409b29c0: image is in use by a container
	* Image used by 19b9cb942b33a072ad5ae513fb61e53170865f4eccb3975d76ff78067e0fa6e1: image is in use by a container
	* Image used by 77b31ad28083421b60ae1e375eaac54c71bb85318fe2413d8e1972b513b400b8: image is in use by a container
	* Image used by bbcb8086c487825ed17547ee4a782faf89a180cc2e0a7c40eb61c638a1cc980a: image is in use by a container
	* Image used by 746f3ae2db1be3328d89c8ecde08ae20718c178206cadb170a85c516a11467ff: image is in use by a container
	* Image used by e6090965a7eedb7010c9325a6f20ed8d752916f43d44856700b8c924aa084f46: image is in use by a container

To an average user it's not obvious where these containers come from because podman ps -a doesn't show anything. However podman ps -a --external does.

We should be able to add some flag to the podman system reset command to scoop up these external containers and storage too.
Alternatively we find some other way to improve this user experience.

I was able to workaround with:

podman ps --external -a --format="{{.ID}}" | xargs podman rm

and then podman system reset could continue.

Output of podman info --debug:

[core@pcengines ~]$ podman info --debug
host:
  arch: amd64
  buildahVersion: 1.21.0
  cgroupControllers: []
  cgroupManager: systemd
  cgroupVersion: v2
  conmon:
    package: conmon-2.0.27-2.fc34.x86_64
    path: /usr/bin/conmon
    version: 'conmon version 2.0.27, commit: '
  cpus: 4
  distribution:
    distribution: fedora
    version: "34"
  eventLogger: journald
  hostname: pcengines
  idMappings:
    gidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 100000
      size: 65536
    uidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 100000
      size: 65536
  kernel: 5.12.9-300.fc34.x86_64
  linkmode: dynamic
  memFree: 3478781952
  memTotal: 4097011712
  ociRuntime:
    name: crun
    package: crun-0.20.1-1.fc34.x86_64
    path: /usr/bin/crun
    version: |-
      crun version 0.20.1
      commit: 0d42f1109fd73548f44b01b3e84d04a279e99d2e
      spec: 1.0.0
      +SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +CRIU +YAJL
  os: linux
  remoteSocket:
    path: /run/user/1000/podman/podman.sock
  security:
    apparmorEnabled: false
    capabilities: CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_NET_BIND_SERVICE,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT
    rootless: true
    seccompEnabled: true
    seccompProfilePath: /usr/share/containers/seccomp.json
    selinuxEnabled: true
  serviceIsRemote: false
  slirp4netns:
    executable: /usr/bin/slirp4netns
    package: slirp4netns-1.1.9-1.fc34.x86_64
    version: |-
      slirp4netns version 1.1.8+dev
      commit: 6dc0186e020232ae1a6fcc1f7afbc3ea02fd3876
      libslirp: 4.4.0
      SLIRP_CONFIG_VERSION_MAX: 3
      libseccomp: 2.5.0
  swapFree: 2047864832
  swapTotal: 2047864832
  uptime: 36h 26m 37.04s (Approximately 1.50 days)
registries:
  search:
  - registry.fedoraproject.org
  - registry.access.redhat.com
  - docker.io
  - quay.io
store:
  configFile: /var/home/core/.config/containers/storage.conf
  containerStore:
    number: 0
    paused: 0
    running: 0
    stopped: 0
  graphDriverName: overlay
  graphOptions: {}
  graphRoot: /var/home/core/.local/share/containers/storage
  graphStatus:
    Backing Filesystem: xfs
    Native Overlay Diff: "false"
    Supports d_type: "true"
    Using metacopy: "false"
  imageStore:
    number: 0
  runRoot: /run/user/1000/containers
  volumePath: /var/home/core/.local/share/containers/storage/volumes
version:
  APIVersion: 3.2.0
  Built: 1623248656
  BuiltTime: Wed Jun  9 14:24:16 2021
  GitCommit: ""
  GoVersion: go1.16.3
  OsArch: linux/amd64
  Version: 3.2.0

Package info (e.g. output of rpm -q podman or apt list podman):

podman-3.2.0-5.fc34.x86_64

[vrothberg]

Thanks for reaching out, @dustymabe!

I totally agree that Podman should handle external containers. I think that the image removal in system reset should behave just as podman rmi -af which will remove external containers.

@mheon @baude @rhatdan thoughts?

[baude]

If we are going to do this (which I support), I would like there to be a prompt stating there are external containers or a --force that includes them. In other words, an overt action says I want the nuclear option.

[dustymabe]

adding a --external flag might be sufficient?

Run once, see there are external containers (maybe we augment error message to be more helpful). Run a second time with --external.

[vrothberg]

I like @baude's proposal of displaying the external containers. To me, system reset is stronger than an rmi -f as it nukes everything, so I don't think we need a new flag for that.

[mheon]

My only concern is that podman build tends to leak Buildah intermediate containers, so I don't think there will be many cases where --external is not required

[dustymabe]

My only concern is that podman build tends to leak Buildah intermediate containers

how do we improve ^^

[mheon]

My memory on this is hazy, but: my understanding is that, when you Control-C (or otherwise abnormally terminate) a build midway through a RUN command, the code that would remove the intermediate container cannot fire (because it was stopped), so we end up with excess Buildah containers. I don't really know what we can do about that, given we don't really have a daemon to clean up - maybe have podman system prune prune them?

@nalind this sound accurate?

[nalind]

That's about right, yeah. We get killed along with the command being run. It's not out of the question that some tweaking to how we handle session/process group setup would change this, but it requires investigation I haven't personally done.

[flouthoc]

@dustymabe @vrothberg could you please give a try to this patch #10789 with podman system reset --force. While podman system reset without --force would give a prompt as @baude suggested.

[dustymabe]

I can try it if someone has a link to an rpm (scratch-build or copr, any rpm).

[flouthoc]

@dustymabe Feature is merged with master , I guess you should be able to use it with next minor release whenever its out where podman system reset --force should do the trick for the needed use-case.

@vrothberg @rhatdan @mheon I think we can safely close this issue.