Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

podman.sock: runtime error: index out of range [0] with length 0 #22561

Closed
heitorPB opened this issue May 1, 2024 · 17 comments
Closed

podman.sock: runtime error: index out of range [0] with length 0 #22561

heitorPB opened this issue May 1, 2024 · 17 comments
Labels
kind/bug Categorizes issue or PR as related to a bug.

Comments

@heitorPB
Copy link

heitorPB commented May 1, 2024

Issue Description

I upgraded my system and suddenly got error messages from some endpoints of the Podman socket.

Since updating all packages on my system, some socket endpoints are failing.

Steps to reproduce the issue

$ curl --unix-socket /var/run/user/1000/podman/podman.sock http://info
Not Found

$ curl --unix-socket /var/run/user/1000/podman/podman.sock http://localhost/info
{"cause":"runtime error: index out of range [0] with length 0","message":"runtime error: index out of range [0] with length 0","response":500}

$ curl --unix-socket /var/run/user/1000/podman/podman.sock http://libpod/info
{"cause":"runtime error: index out of range [0] with length 0","message":"runtime error: index out of range [0] with length 0","response":500}

But curl --unix-socket /var/run/user/1000/podman/podman.sock http://localhost/version works as expected (results bellow)

Describe the results you received

The returned payload contains a runtime error message:

$ curl --unix-socket /var/run/user/1000/podman/podman.sock http://localhost/info -s | jq
{
  "cause": "runtime error: index out of range [0] with length 0",
  "message": "runtime error: index out of range [0] with length 0",
  "response": 500
}

The logs I got from the podman.service:

$ journalctl --user-unit podman.service --since today -n 2 | tee podman.logs
mai 01 11:22:15 L14 podman[55658]: time="2024-05-01T11:22:15-03:00" level=warning msg="Recovering from API service endpoint handler panic: runtime error: index out of range [0] with length 0, goroutine 9245 [running]:\ngithub.com/containers/podman/v5/pkg/api/server.newServer.panicHandler.func6.1.1()\n\tgithub.com/containers/podman/v5/pkg/api/server/handler_panic.go:22 +0x74\npanic({0x1b5a920?, 0xc000a037b8?})\n\truntime/panic.go:770 +0x132\ngithub.com/containers/podman/v5/pkg/api/handlers/compat.getRuntimes(...)\n\tgithub.com/containers/podman/v5/pkg/api/handlers/compat/info.go:197\ngithub.com/containers/podman/v5/pkg/api/handlers/compat.GetInfo({0x1f83f50, 0xc000d8a600}, 0x1c490b7?)\n\tgithub.com/containers/podman/v5/pkg/api/handlers/compat/info.go:112 +0x11da\ngithub.com/containers/podman/v5/pkg/api/server.(*APIServer).apiWrapper(0xc0000a6000, 0x1d42900, {0x1f83f50, 0xc000d8a600}, 0xc000e8cb40, 0x0)\n\tgithub.com/containers/podman/v5/pkg/api/server/handler_api.go:64 +0x9ca\ngithub.com/containers/podman/v5/pkg/api/server.(*APIServer).registerInfoHandlers.(*APIServer).APIHandler.func2({0x1f83f50?, 0xc000d8a600?}, 0x1c58381?)\n\tgithub.com/containers/podman/v5/pkg/api/server/handler_api.go:24 +0x37\nnet/http.HandlerFunc.ServeHTTP(0xc000933620?, {0x1f83f50?, 0xc000d8a600?}, 0xc0005b4940?)\n\tnet/http/server.go:2166 +0x29\ngithub.com/containers/podman/v5/pkg/api/server.newServer.referenceIDHandler.func7.1({0x1f83f50, 0xc000d8a600}, 0xc000e8cb40)\n\tgithub.com/containers/podman/v5/pkg/api/server/handler_rid.go:38 +0x159\nnet/http.HandlerFunc.ServeHTTP(0x1f81e68?, {0x1f83f50?, 0xc000d8a600?}, 0x0?)\n\tnet/http/server.go:2166 +0x29\ngithub.com/gorilla/handlers.loggingHandler.ServeHTTP({{0x1f745e0, 0xc000120058}, {0x1f76160, 0xc000e019f8}, 0x1d42d00}, {0x1f81e68, 0xc000c5e1c0}, 0xc000e8cb40)\n\tgithub.com/gorilla/handlers@v1.5.2/logging.go:47 +0xef\ngithub.com/containers/podman/v5/pkg/api/server.newServer.panicHandler.func6.1({0x1f81e68?, 0xc000c5e1c0?}, 0xc0009337d0?)\n\tgithub.com/containers/podman/v5/pkg/api/server/handler_panic.go:29 +0x76\nnet/http.HandlerFunc.ServeHTTP(0xc000e8c900?, {0x1f81e68?, 0xc000c5e1c0?}, 0x4fe04f?)\n\tnet/http/server.go:2166 +0x29\ngithub.com/gorilla/mux.(*Router).ServeHTTP(0xc0001746c0, {0x1f81e68, 0xc000c5e1c0}, 0xc000e8c6c0)\n\tgithub.com/gorilla/mux@v1.8.1/mux.go:212 +0x1e2\nnet/http.serverHandler.ServeHTTP({0xc000933530?}, {0x1f81e68?, 0xc000c5e1c0?}, 0x6?)\n\tnet/http/server.go:3137 +0x8e\nnet/http.(*conn).serve(0xc000ad8750, {0x1f85858, 0xc000933470})\n\tnet/http/server.go:2039 +0x5e8\ncreated by net/http.(*Server).Serve in goroutine 28\n\tnet/http/server.go:3285 +0x4b4\n\ngoroutine 1 [chan receive, 26 minutes]:\ngithub.com/containers/podman/v5/pkg/api/server.(*APIServer).Serve(0xc0000a6000)\n\tgithub.com/containers/podman/v5/pkg/api/server/server.go:227 +0x168\ngithub.com/containers/podman/v5/cmd/podman/system.restService(0xc000542500, 0x2cec9c0, {{0x0, 0x0}, {0x0, 0x0}, 0x12a05f200, {0x0, 0x0}})\n\tgithub.com/containers/podman/v5/cmd/podman/system/service_abi.go:138 +0xb7f\ngithub.com/containers/podman/v5/cmd/podman/system.service(0x2c1d7e0, {0x2d552a0?, 0x0?, 0x0?})\n\tgithub.com/containers/podman/v5/cmd/podman/system/service.go:102 +0x265\ngithub.com/spf13/cobra.(*Command).execute(0x2c1d7e0, {0xc00014a030, 0x0, 0x0})\n\tgithub.com/spf13/cobra@v1.8.0/command.go:983 +0xaca\ngithub.com/spf13/cobra.(*Command).ExecuteC(0x2c07080)\n\tgithub.com/spf13/cobra@v1.8.0/command.go:1115 +0x3ff\ngithub.com/spf13/cobra.(*Command).Execute(...)\n\tgithub.com/spf13/cobra@v1.8.0/command.go:1039\ngithub.com/spf13/cobra.(*Command).ExecuteContext(...)\n\tgithub.com/spf13/cobra@v1.8.0/command.go:1032\nmain.Execute()\n\tgithub.com/containers/podman/v5/cmd/podman/root.go:115 +0xb4\nmain.main()\n\tgithub.com/containers/podman/v5/cmd/podman/main.go:60 +0x452\n\ngoroutine 39 [syscall]:\nos/signal.signal_recv()\n\truntime/sigqueue.go:152 +0x29\nos/signal.loop()\n\tos/signal/signal_unix.go:23 +0x13\ncreated by os/signal.Notify.func1.1 in goroutine 1\n\tos/signal/signal.go:151 +0x1f\n\ngoroutine 40 [select, 26 minutes]:\ngithub.com/containers/podman/v5/libpod/shutdown.Start.func1()\n\tgithub.com/containers/podman/v5/libpod/shutdown/handler.go:48 +0x87\ncreated by github.com/containers/podman/v5/libpod/shutdown.Start in goroutine 1\n\tgithub.com/containers/podman/v5/libpod/shutdown/handler.go:47 +0xf1\n\ngoroutine 41 [select, 26 minutes]:\ndatabase/sql.(*DB).connectionOpener(0xc0005964e0, {0x1f85890, 0xc000176370})\n\tdatabase/sql/sql.go:1246 +0x87\ncreated by database/sql.OpenDB in goroutine 1\n\tdatabase/sql/sql.go:824 +0x14c\n\ngoroutine 12 [select]:\ngithub.com/containers/podman/v5/libpod.(*Runtime).libimageEvents.func2()\n\tgithub.com/containers/podman/v5/libpod/runtime.go:726 +0xc9\ncreated by github.com/containers/podman/v5/libpod.(*Runtime).libimageEvents in goroutine 1\n\tgithub.com/containers/podman/v5/libpod/runtime.go:699 +0xfc\n\ngoroutine 13 [chan receive, 26 minutes]:\ngithub.com/containers/podman/v5/libpod.(*Runtime).startWorker.func1()\n\tgithub.com/containers/podman/v5/libpod/runtime_worker.go:8 +0x6c\ncreated by github.com/containers/podman/v5/libpod.(*Runtime).startWorker in goroutine 1\n\tgithub.com/containers/podman/v5/libpod/runtime_worker.go:7 +0x8e\n\ngoroutine 45 [chan receive, 26 minutes]:\ngithub.com/godbus/dbus/v5.newConn.func1()\n\tgithub.com/godbus/dbus/v5@v5.1.1-0.20230522191255-76236955d466/conn.go:302 +0x2c\ncreated by github.com/godbus/dbus/v5.newConn in goroutine 1\n\tgithub.com/godbus/dbus/v5@v5.1.1-0.20230522191255-76236955d466/conn.go:301 +0x4dc\n\ngoroutine 46 [IO wait, 26 minutes]:\ninternal/poll.runtime_pollWait(0x7f82a00b3e70, 0x72)\n\truntime/netpoll.go:345 +0x85\ninternal/poll.(*pollDesc).wait(0xc00019ea00?, 0xc000595e40?, 0x0)\n\tinternal/poll/fd_poll_runtime.go:84 +0x27\ninternal/poll.(*pollDesc).waitRead(...)\n\tinternal/poll/fd_poll_runtime.go:89\ninternal/poll.(*FD).ReadMsg(0xc00019ea00, {0xc000595e40, 0x10, 0x10}, {0xc0000f8628, 0x1000, 0x1000}, 0x40000000)\n\tinternal/poll/fd_unix.go:301 +0x38a\nnet.(*netFD).readMsg(0xc00019ea00, {0xc000595e40?, 0x7f82e833f108?, 0x10?}, {0xc0000f8628?, 0xc00023b3e0?, 0x0?}, 0x0?)\n\tnet/fd_posix.go:78 +0x31\nnet.(*UnixConn).readMsg(0xc0005ac060, {0xc000595e40?, 0x1?, 0x1925a00?}, {0xc0000f8628?, 0xc00055fdd0?, 0xcd69dc?})\n\tnet/unixsock_posix.go:115 +0x45\nnet.(*UnixConn).ReadMsgUnix(0xc0005ac060, {0xc000595e40?, 0xc00055fdb0?, 0x412170?}, {0xc0000f8628?, 0x0?, 0xc00055fdb0?})\n\tnet/unixsock.go:143 +0x36\ngithub.com/godbus/dbus/v5.(*oobReader).Read(0xc0000f8608, {0xc000595e40?, 0x1f7be50?, 0xc000298000?})\n\tgithub.com/godbus/dbus/v5@v5.1.1-0.20230522191255-76236955d466/transport_unix.go:41 +0x3c\nio.ReadAtLeast({0x1f77660, 0xc0000f8608}, {0xc000595e40, 0x10, 0x10}, 0x10)\n\tio/io.go:335 +0x90\nio.ReadFull(...)\n\tio/io.go:354\ngithub.com/godbus/dbus/v5.(*unixTransport).ReadMessage(0xc000012258)\n\tgithub.com/godbus/dbus/v5@v5.1.1-0.20230522191255-76236955d466/transport_unix.go:116 +0x21f\ngithub.com/godbus/dbus/v5.(*Conn).inWorker(0xc0005b2000)\n\tgithub.com/godbus/dbus/v5@v5.1.1-0.20230522191255-76236955d466/conn.go:389 +0x43\ncreated by github.com/godbus/dbus/v5.(*Conn).Auth in goroutine 1\n\tgithub.com/godbus/dbus/v5@v5.1.1-0.20230522191255-76236955d466/auth.go:118 +0x8dd\n\ngoroutine 49 [chan receive]:\ngithub.com/containers/common/pkg/servicereaper.reaper(0xc000140c60)\n\tgithub.com/containers/common@v0.58.2/pkg/servicereaper/service.go:41 +0x33\ncreated by github.com/containers/common/pkg/servicereaper.Start in goroutine 1\n\tgithub.com/containers/common@v0.58.2/pkg/servicereaper/service.go:35 +0x98\n\ngoroutine 14 [chan receive, 26 minutes]:\ngithub.com/containers/podman/v5/pkg/domain/infra.StartWatcher.func1()\n\tgithub.com/containers/podman/v5/pkg/domain/infra/runtime_libpod.go:305 +0x56\ncreated by github.com/containers/podman/v5/pkg/domain/infra.StartWatcher in goroutine 1\n\tgithub.com/containers/podman/v5/pkg/domain/infra/runtime_libpod.go:301 +0xab\n\ngoroutine 27 [chan receive, 26 minutes]:\ngithub.com/containers/podman/v5/pkg/api/server.(*APIServer).Serve.func2()\n\tgithub.com/containers/podman/v5/pkg/api/server/server.go:208 +0x30\ncreated by github.com/containers/podman/v5/pkg/api/server.(*APIServer).Serve in goroutine 1\n\tgithub.com/containers/podman/v5/pkg/api/server/server.go:207 +0xba\n\ngoroutine 28 [IO wait]:\ninternal/poll.runtime_pollWait(0x7f82a00b3d78, 0x72)\n\truntime/netpoll.go:345 +0x85\ninternal/poll.(*pollDesc).wait(0x9?, 0xe?, 0x0)\n\tinternal/poll/fd_poll_runtime.go:84 +0x27\ninternal/poll.(*pollDesc).waitRead(...)\n\tinternal/poll/fd_poll_runtime.go:89\ninternal/poll.(*FD).Accept(0xc0002fb580)\n\tinternal/poll/fd_unix.go:611 +0x2ac\nnet.(*netFD).accept(0xc0002fb580)\n\tnet/fd_unix.go:172 +0x29\nnet.(*UnixListener).accept(0xc0000b9e30?)\n\tnet/unixsock_posix.go:172 +0x16\nnet.(*UnixListener).Accept(0xc00059bc20)\n\tnet/unixsock.go:260 +0x30\nnet/http.(*Server).Serve(0xc0000a6000, {0x1f81a18, 0xc00059bc20})\n\tnet/http/server.go:3255 +0x33e\ngithub.com/containers/podman/v5/pkg/api/server.(*APIServer).Serve.func3()\n\tgithub.com/containers/podman/v5/pkg/api/server/server.go:219 +0x32\ncreated by github.com/containers/podman/v5/pkg/api/server.(*APIServer).Serve in goroutine 1\n\tgithub.com/containers/podman/v5/pkg/api/server/server.go:218 +0x153\n\ngoroutine 9246 [IO wait]:\ninternal/poll.runtime_pollWait(0x7f82a00b38a0, 0x72)\n\truntime/netpoll.go:345 +0x85\ninternal/poll.(*pollDesc).wait(0xc000943f00?, 0xc000933541?, 0x0)\n\tinternal/poll/fd_poll_runtime.go:84 +0x27\ninternal/poll.(*pollDesc).waitRead(...)\n\tinternal/poll/fd_poll_runtime.go:89\ninternal/poll.(*FD).Read(0xc000943f00, {0xc000933541, 0x1, 0x1})\n\tinternal/poll/fd_unix.go:164 +0x27a\nnet.(*netFD).Read(0xc000943f00, {0xc000933541?, 0xc000882068?, 0x0?})\n\tnet/fd_posix.go:55 +0x25\nnet.(*conn).Read(0xc0008fb528, {0xc000933541?, 0xc000014008?, 0xc000a1c188?})\n\tnet/net.go:179 +0x45\nnet/http.(*connReader).backgroundRead(0xc000933530)\n\tnet/http/server.go:681 +0x37\ncreated by net/http.(*connReader).startBackgroundRead in goroutine 9245\n\tnet/http/server.go:677 +0xba\n"
mai 01 11:22:15 L14 podman[55658]: time="2024-05-01T11:22:15-03:00" level=info msg="Request Failed(Internal Server Error): runtime error: index out of range [0] with length 0"

Describe the results you expected

I expected the api to not error out. Instead, to return the information without crashing.

podman info output

$ podman version
Client:       Podman Engine
Version:      5.0.2
API Version:  5.0.2
Go Version:   go1.22.2
Built:        Mon Dec 31 21:00:00 1979
OS/Arch:      linux/amd64


$ podman info
host:
  arch: amd64
  buildahVersion: 1.35.3
  cgroupControllers:
  - cpu
  - io
  - memory
  - pids
  cgroupManager: systemd
  cgroupVersion: v2
  conmon:
    package: Unknown
    path: /nix/store/z19pwnaidxiy4ry5jp4aa7m0sk5x42qr-conmon-2.1.10/bin/conmon
    version: 'conmon version 2.1.10, commit: '
  cpuUtilization:
    idlePercent: 96.42
    systemPercent: 1.24
    userPercent: 2.34
  cpus: 16
  databaseBackend: sqlite
  distribution:
    codename: uakari
    distribution: nixos
    version: "24.05"
  eventLogger: journald
  freeLocks: 2036
  hostname: L14
  idMappings:
    gidmap:
    - container_id: 0
      host_id: 100
      size: 1
    - container_id: 1
      host_id: 100000
      size: 65536
    uidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 100000
      size: 65536
  kernel: 6.6.28
  linkmode: dynamic
  logDriver: journald
  memFree: 4042944512
  memTotal: 15497019392
  networkBackend: netavark
  networkBackendInfo:
    backend: netavark
    dns:
      package: Unknown
      path: /nix/store/fiwb4kcxj74dvfwnwhksvi4m2wp0ma3m-podman-5.0.2/libexec/podman/aardvark-dns
      version: aardvark-dns 1.10.0
    package: Unknown
    path: /nix/store/fiwb4kcxj74dvfwnwhksvi4m2wp0ma3m-podman-5.0.2/libexec/podman/netavark
    version: netavark 1.7.0
  ociRuntime:
    name: crun
    package: Unknown
    path: /nix/store/394d1gwshxgj53l488va9xna5vfnqm9j-crun-1.14.4/bin/crun
    version: |-
      crun version 1.14.4
      commit: 1.14.4
      rundir: /run/user/1000/crun
      spec: 1.0.0
      +SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +CRIU +YAJL
  os: linux
  pasta:
    executable: /nix/store/fiwb4kcxj74dvfwnwhksvi4m2wp0ma3m-podman-5.0.2/libexec/podman/pasta
    package: Unknown
    version: |
      pasta 2024_04_05.954589b
      Copyright Red Hat
      GNU General Public License, version 2 or later
        <https://www.gnu.org/licenses/old-licenses/gpl-2.0.html>
      This is free software: you are free to change and redistribute it.
      There is NO WARRANTY, to the extent permitted by law.
  remoteSocket:
    exists: true
    path: /run/user/1000/podman/podman.sock
  security:
    apparmorEnabled: false
    capabilities: CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_NET_BIND_SERVICE,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT
    rootless: true
    seccompEnabled: true
    seccompProfilePath: ""
    selinuxEnabled: false
  serviceIsRemote: false
  slirp4netns:
    executable: /nix/store/fiwb4kcxj74dvfwnwhksvi4m2wp0ma3m-podman-5.0.2/libexec/podman/slirp4netns
    package: Unknown
    version: |-
      slirp4netns version 1.2.3
      commit: c22fde291bb35b354e6ca44d13be181c76a0a432
      libslirp: 4.7.0
      SLIRP_CONFIG_VERSION_MAX: 4
      libseccomp: 2.5.5
  swapFree: 17179865088
  swapTotal: 17179865088
  uptime: 0h 52m 12.00s
  variant: ""
plugins:
  authorization: null
  log:
  - k8s-file
  - none
  - passthrough
  - journald
  network:
  - bridge
  - macvlan
  - ipvlan
  volume:
  - local
registries:
  search:
  - docker.io
  - quay.io
store:
  configFile: /home/h/.config/containers/storage.conf
  containerStore:
    number: 1
    paused: 0
    running: 0
    stopped: 1
  graphDriverName: overlay
  graphOptions: {}
  graphRoot: /home/h/.local/share/containers/storage
  graphRootAllocated: 458047488000
  graphRootUsed: 128015925248
  graphStatus:
    Backing Filesystem: zfs
    Native Overlay Diff: "true"
    Supports d_type: "true"
    Supports shifting: "false"
    Supports volatile: "true"
    Using metacopy: "false"
  imageCopyTmpDir: /var/tmp
  imageStore:
    number: 30
  runRoot: /run/user/1000/containers
  transientStore: false
  volumePath: /home/h/.local/share/containers/storage/volumes
version:
  APIVersion: 5.0.2
  Built: 315532800
  BuiltTime: Mon Dec 31 21:00:00 1979
  GitCommit: ""
  GoVersion: go1.22.2
  Os: linux
  OsArch: linux/amd64
  Version: 5.0.2


$ curl --unix-socket /var/run/user/1000/podman/podman.sock http://localhost/version -s | jq
{
  "Platform": {
    "Name": "linux/amd64/nixos-24.05"
  },
  "Components": [
    {
      "Name": "Podman Engine",
      "Version": "5.0.2",
      "Details": {
        "APIVersion": "5.0.2",
        "Arch": "amd64",
        "BuildTime": "1979-12-31T21:00:00-03:00",
        "Experimental": "false",
        "GitCommit": "",
        "GoVersion": "go1.22.2",
        "KernelVersion": "6.6.28",
        "MinAPIVersion": "4.0.0",
        "Os": "linux"
      }
    },
    {
      "Name": "Conmon",
      "Version": "conmon version 2.1.10, commit: ",
      "Details": {
        "Package": "Unknown"
      }
    },
    {
      "Name": "OCI Runtime (crun)",
      "Version": "crun version 1.14.4\ncommit: 1.14.4\nrundir: /run/user/1000/crun\nspec: 1.0.0\n+SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +CRIU +YAJL",
      "Details": {
        "Package": "Unknown"
      }
    }
  ],
  "Version": "5.0.2",
  "ApiVersion": "1.41",
  "MinAPIVersion": "1.24",
  "GitCommit": "",
  "GoVersion": "go1.22.2",
  "Os": "linux",
  "Arch": "amd64",
  "KernelVersion": "6.6.28",
  "BuildTime": "1979-12-31T21:00:00-03:00"
}

Podman in a container

No

Privileged Or Rootless

Rootless

Upstream Latest Release

Yes

Additional environment details

Running on NixOs, unstable nixpkgs.

Additional information

I found this issue when i3status-rs errored out on my Docker stats. It connects to a socket to retrieve the data and before the update all was fine.

@heitorPB heitorPB added the kind/bug Categorizes issue or PR as related to a bug. label May 1, 2024
@Luap99
Copy link
Member

Luap99 commented May 2, 2024

Do you have any special containers.conf configured? I can see the place where it can panic but it should not happen with a default config.

@heitorPB
Copy link
Author

heitorPB commented May 2, 2024

❯ cat /etc/containers/containers.conf
[engine]
init_path = "/nix/store/zgwgcbzq2qz15mvnn67ks3km0amdi7y9-catatonit-0.2.0/bin/catatonit"

[network]
cni_plugin_dirs = ["/nix/store/lz1ll9v7hf6877v5zzqrgvszbqrkv1xj-cni-plugins-1.4.1/bin"]
network_backend = "netavark"

Is this a non-default config?

I can see the place where it can panic

Do you see a reason why it can panic, to handle it with an explanation?

@kevincox
Copy link

kevincox commented May 2, 2024

I'm seeing this on NixOS using the default podman config.

	virtualisation.podman = {
		enable = true;
		dockerSocket.enable = true;
	};
% tree /etc/containers
/etc/containers
├── containers.conf -> /etc/static/containers/containers.conf
├── networks
├── policy.json -> /etc/static/containers/policy.json
├── registries.conf -> /etc/static/containers/registries.conf
└── storage.conf -> /etc/static/containers/storage.conf

2 directories, 4 files
% cat /etc/containers/containers.conf
[engine]
init_path = "/nix/store/zgwgcbzq2qz15mvnn67ks3km0amdi7y9-catatonit-0.2.0/bin/catatonit"

[network]
cni_plugin_dirs = ["/nix/store/lz1ll9v7hf6877v5zzqrgvszbqrkv1xj-cni-plugins-1.4.1/bin"]
network_backend = "netavark"
% ls -l /etc/containers/networks
total 0
% cat /etc/containers/policy.json
{
    "default": [
        {
            "type": "insecureAcceptAnything"
        }
    ],
    "transports":
        {
            "docker-daemon":
                {
                    "": [{"type":"insecureAcceptAnything"}]
                }
        }
}
% cat /etc/containers/registries.conf
[registries]
[registries.block]
registries = []

[registries.insecure]
registries = []

[registries.search]
registries = ["docker.io", "quay.io"]
% cat /etc/containers/storage.conf
[storage]
driver = "overlay"
graphroot = "/var/lib/containers/storage"
runroot = "/run/containers/storage"

Downgrading to 4.9.3 fixes it. 5.x seems to have trouble.

@Luap99
Copy link
Member

Luap99 commented May 2, 2024

❯ cat /etc/containers/containers.conf
[engine]
init_path = "/nix/store/zgwgcbzq2qz15mvnn67ks3km0amdi7y9-catatonit-0.2.0/bin/catatonit"

[network]
cni_plugin_dirs = ["/nix/store/lz1ll9v7hf6877v5zzqrgvszbqrkv1xj-cni-plugins-1.4.1/bin"]
network_backend = "netavark"

Is this a non-default config?

I can see the place where it can panic

Do you see a reason why it can panic, to handle it with an explanation?

The default config is no config at all but this won't likely work in the weird nix world. Anyhow non of the keys there should pose any issues so this should not be super relevant AFAIK. Are there nay configs in the user home dir, or /usr/share/containers?

Also could you run podman --log-level trace info that should show some runtime information.

The panic happens because you seem to have a OCIruntime defined without a path:

runtime.Runtime = dockerSystem.Runtime{Path: paths[0], Args: nil}

Of course podman should never panic and handle this case (here likely just ignore it) but I don't see how this can happen unless someone manually specified a runtimes map in containers.conf

@Luap99
Copy link
Member

Luap99 commented May 2, 2024

i.e. a config file like this reproduces

[engine]
[engine.runtimes]
nonstandard_runtime_name = []

@kevincox
Copy link

kevincox commented May 2, 2024

Are there nay configs in the user home dir, or /usr/share/containers

I was testing on the system wide socket so it should only be using the global config.

@heitorPB
Copy link
Author

heitorPB commented May 2, 2024

The panic happens because you seem to have a OCIruntime defined without a path:

What is the standard path for it? I can modify the config file and try again.

I have no user-specific config in ~/.local/share/containers/, only storage related.
I get the same errors regardless of system wide, user wide socket:

❯ curl --unix-socket /var/run/docker.sock http://localhost/info
{"cause":"runtime error: index out of range [0] with length 0","message":"runtime error: index out of range [0] with length 0","response":500}

❯ curl --unix-socket /var/run/user/1000/podman/podman.sock http://localhost/info
{"cause":"runtime error: index out of range [0] with length 0","message":"runtime error: index out of range [0] with length 0","response":500}

@Luap99
Copy link
Member

Luap99 commented May 2, 2024

The config paths would be /usr/share/containers/containers.conf /etc/containers/containers.conf ~/.config/containers/containers.conf + possible .d directories

I doubt that both of you would have any of this set but still I cannot find a reason in code why a path would be unset unless a invalid config was present. I also cannot see any difference with 4.9/5.0 in that regard.

@kevincox
Copy link

kevincox commented May 2, 2024

I wonder if this could be the problem. I see that some of these groups are empty:

https://github.com/NixOS/nixpkgs/blob/23f69f50f72ae19d7bbc7271abe30d8bc859207f/pkgs/applications/virtualization/podman/hardcode-paths.patch

cc @WxNzEMof

@Luap99
Copy link
Member

Luap99 commented May 2, 2024

Oh yes this patch breaks it. Instead of commenting out the paths it should delete the map entry there.

@Luap99
Copy link
Member

Luap99 commented May 2, 2024

Oh yes this patch breaks it. Instead of commenting out the paths it should delete the map entry there.

I mean it doesn't really matter I guess, we should fix it in podman no matter what and then it should not make a functional difference fot that path

@kevincox
Copy link

kevincox commented May 2, 2024

Ok, so for a quick fix it should be possible to just comment out the whole stanzas? Then podman can consider the UX of this in a future release?

@heitorPB
Copy link
Author

heitorPB commented May 3, 2024

Not sure I followed all points here. The complete fix involves handling the error in Podman and adding the path in NixOs packages?

@cyrillzadra
Copy link

cyrillzadra commented May 8, 2024

Is there another workaround besides just downgrading to podman 4.9.3?

@Luap99
Copy link
Member

Luap99 commented May 8, 2024

Well for starters did you report this to nixos that there patch broke this? Maybe get them to revert it or fix to not have runtimes without paths defined as mentioned above.

Or well submit a patch to podman to not make it panic on "invalid" configs which should happen regardless if nix fixes it in
the config or not.

WxNzEMof added a commit to WxNzEMof/nixpkgs that referenced this issue May 9, 2024
Change the approach used to integrate runtimes, in order to:

- Better support macOS
- Make obscure OCI runtimes optional
- Work around a panic due to runtimes having no paths
  (see containers/podman#22561)
@kevincox
Copy link

NixOS/nixpkgs#310182 fixes the patch that was triggering the issue in NixOS. Do you want to leave this open to handle empty lists more gracefully? Or should this be closed?

SuperSandro2000 pushed a commit to SuperSandro2000/nixpkgs that referenced this issue May 12, 2024
Change the approach used to integrate runtimes, in order to:

- Better support macOS
- Make obscure OCI runtimes optional
- Work around a panic due to runtimes having no paths
  (see containers/podman#22561)
SuperSandro2000 pushed a commit to SuperSandro2000/nixpkgs that referenced this issue May 14, 2024
Change the approach used to integrate runtimes, in order to:

- Better support macOS
- Make obscure OCI runtimes optional
- Work around a panic due to runtimes having no paths
  (see containers/podman#22561)
SuperSandro2000 pushed a commit to SuperSandro2000/nixpkgs that referenced this issue May 14, 2024
Change the approach used to integrate runtimes, in order to:

- Better support macOS
- Make obscure OCI runtimes optional
- Work around a panic due to runtimes having no paths
  (see containers/podman#22561)
SuperSandro2000 pushed a commit to SuperSandro2000/nixpkgs that referenced this issue May 14, 2024
Change the approach used to integrate runtimes, in order to:

- Better support macOS
- Make obscure OCI runtimes optional
- Work around a panic due to runtimes having no paths
  (see containers/podman#22561)
SuperSandro2000 pushed a commit to SuperSandro2000/nixpkgs that referenced this issue May 14, 2024
Change the approach used to integrate runtimes, in order to:

- Better support macOS
- Make obscure OCI runtimes optional
- Work around a panic due to runtimes having no paths
  (see containers/podman#22561)
@Luap99
Copy link
Member

Luap99 commented May 15, 2024

Fixed in #22640

@Luap99 Luap99 closed this as completed May 15, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
kind/bug Categorizes issue or PR as related to a bug.
Projects
None yet
Development

No branches or pull requests

4 participants