Failed to decode the keys ["machine"]

[KhalilSantana]

Is this a BUG REPORT or FEATURE REQUEST? (leave only one on its own line)

/kind bug

Description

Running any podman command, like podman --version on podman >= 3.3.0 results in the following warnings:

% podman --version
WARN[0000] Failed to decode the keys ["machine"] from "/usr/share/containers/containers.conf". 
WARN[0000] Failed to decode the keys ["machine"] from "/usr/share/containers/containers.conf". 
podman version 3.3.1

Both root-full and root-less podman processes are affected.
Steps to reproduce the issue:

1. Install podman 3.3.1 on Archlinux

2. Run any podman command like podman --version or podman ps -a

3. Obverve the warnings/output printed

Describe the results you received:
Podman printed warnings (see above)

Describe the results you expected:
Podman's default config file shouldn't cause warnings during normal usage

Additional information you deem important (e.g. issue happens only occasionally):
The latest podman 3.2.X available for my distro doesn't have this issue. Only the 3.3.X branch

Output of podman version:

podman version 3.3.1

Output of podman info --debug:

(paste your output here)

Package info (e.g. output of rpm -q podman or apt list podman):

host:
  arch: amd64
  buildahVersion: 1.22.3
  cgroupControllers: []
  cgroupManager: systemd
  cgroupVersion: v2
  conmon:
    package: /usr/bin/conmon pertence a conmon 1:2.0.30-1
    path: /usr/bin/conmon
    version: 'conmon version 2.0.30, commit: 2792c16f4436f1887a7070d9ad99d9c29742f38a'
  cpus: 8
  distribution:
    distribution: arch
    version: unknown
  eventLogger: journald
  hostname: andromeda
  idMappings:
    gidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 100000
      size: 65536
    uidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 165536
      size: 65536
  kernel: 5.14.7-arch1-1
  linkmode: dynamic
  memFree: 15988346880
  memTotal: 50467069952
  ociRuntime:
    name: crun
    package: /usr/bin/crun pertence a crun 1.0-2
    path: /usr/bin/crun
    version: |-
      crun version 1.0
      commit: 139dc6971e2f1d931af520188763e984d6cdfbf8
      spec: 1.0.0
      +SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +YAJL
  os: linux
  remoteSocket:
    path: /run/user/1000/podman/podman.sock
  security:
    apparmorEnabled: false
    capabilities: CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_NET_BIND_SERVICE,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT
    rootless: true
    seccompEnabled: true
    seccompProfilePath: /etc/containers/seccomp.json
    selinuxEnabled: false
  serviceIsRemote: false
  slirp4netns:
    executable: /usr/bin/slirp4netns
    package: /usr/bin/slirp4netns pertence a slirp4netns 1.1.12-1
    version: |-
      slirp4netns version 1.1.12
      commit: 7a104a101aa3278a2152351a082a6df71f57c9a3
      libslirp: 4.6.1
      SLIRP_CONFIG_VERSION_MAX: 3
      libseccomp: 2.5.2
  swapFree: 4294963200
  swapTotal: 4294963200
  uptime: 3h 59m 18.71s (Approximately 0.12 days)
registries: {}
store:
  configFile: /home/khalil/.config/containers/storage.conf
  containerStore:
    number: 0
    paused: 0
    running: 0
    stopped: 0
  graphDriverName: btrfs
  graphOptions: {}
  graphRoot: /home/khalil/.local/share/containers/storage
  graphStatus:
    Build Version: 'Btrfs v5.13.1 '
    Library Version: "102"
  imageStore:
    number: 1
  runRoot: /run/user/1000/containers
  volumePath: /home/khalil/.local/share/containers/storage/volumes
version:
  APIVersion: 3.3.1
  Built: 1630517266
  BuiltTime: Wed Sep  1 14:27:46 2021
  GitCommit: 4c5283fabff2de5145838f1847a5a7b2b1fbc0a5-dirty
  GoVersion: go1.17
  OsArch: linux/amd64
  Version: 3.3.1

Have you tested with the latest version of Podman and have you checked the Podman Troubleshooting Guide? (https://github.com/containers/podman/blob/master/troubleshooting.md)

Yes

Additional environment details (AWS, VirtualBox, physical, etc.):
Physical host running an AMD64-compatible processor.

[KhalilSantana]

The contents of /usr/share/containers/containers.conf:

# The containers configuration file specifies all of the available configuration
# command-line options/flags for container engine tools like Podman & Buildah,
# but in a TOML format that can be easily modified and versioned.

# Please refer to containers.conf(5) for details of all configuration options.
# Not all container engines implement all of the options.
# All of the options have hard coded defaults and these options will override
# the built in defaults. Users can then override these options via the command
# line. Container engines will read containers.conf files in up to three
# locations in the following order:
#  1. /usr/share/containers/containers.conf
#  2. /etc/containers/containers.conf
#  3. $HOME/.config/containers/containers.conf (Rootless containers ONLY)
#  Items specified in the latter containers.conf, if they exist, override the
# previous containers.conf settings, or the default settings.

[containers]

# List of annotation. Specified as
# "key = value"
# If it is empty or commented out, no annotations will be added
#
#annotations = []

# Used to change the name of the default AppArmor profile of container engine.
#
#apparmor_profile = "container-default"

# Default way to to create a cgroup namespace for the container
# Options are:
# `private` Create private Cgroup Namespace for the container.
# `host`    Share host Cgroup Namespace with the container.
#
#cgroupns = "private"

# Control container cgroup configuration
# Determines  whether  the  container will create CGroups.
# Options are:
# `enabled`   Enable cgroup support within container
# `disabled`  Disable cgroup support, will inherit cgroups from parent
# `no-conmon` Do not create a cgroup dedicated to conmon.
#
#cgroups = "enabled"

# List of default capabilities for containers. If it is empty or commented out,
# the default capabilities defined in the container engine will be added.
#
default_capabilities = [
  "CHOWN",
  "DAC_OVERRIDE",
  "FOWNER",
  "FSETID",
  "KILL",
  "NET_BIND_SERVICE",
  "SETFCAP",
  "SETGID",
  "SETPCAP",
  "SETUID",
  "SYS_CHROOT"
]

# A list of sysctls to be set in containers by default,
# specified as "name=value",
# for example:"net.ipv4.ping_group_range=0 0".
#
default_sysctls = [
  "net.ipv4.ping_group_range=0 0",
]

# A list of ulimits to be set in containers by default, specified as
# "<ulimit name>=<soft limit>:<hard limit>", for example:
# "nofile=1024:2048"
# See setrlimit(2) for a list of resource names.
# Any limit not specified here will be inherited from the process launching the
# container engine.
# Ulimits has limits for non privileged container engines.
#
#default_ulimits = [
#  "nofile=1280:2560",
#]

# List of devices. Specified as
# "<device-on-host>:<device-on-container>:<permissions>", for example:
# "/dev/sdc:/dev/xvdc:rwm".
# If it is empty or commented out, only the default devices will be used
#
#devices = []

# List of default DNS options to be added to /etc/resolv.conf inside of the container.
#
#dns_options = []

# List of default DNS search domains to be added to /etc/resolv.conf inside of the container.
#
#dns_searches = []

# Set default DNS servers.
# This option can be used to override the DNS configuration passed to the
# container. The special value "none" can be specified to disable creation of
# /etc/resolv.conf in the container.
# The /etc/resolv.conf file in the image will be used without changes.
#
#dns_servers = []

# Environment variable list for the conmon process; used for passing necessary
# environment variables to conmon or the runtime.
#
#env = [
#  "PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin",
#  "TERM=xterm",
#]

# Pass all host environment variables into the container.
#
#env_host = false

# Default proxy environment variables passed into the container.
# The environment variables passed in include:
# http_proxy, https_proxy, ftp_proxy, no_proxy, and the upper case versions of
# these. This option is needed when host system uses a proxy but container
# should not use proxy. Proxy environment variables specified for the container
# in any other way will override the values passed from the host.
#
#http_proxy = true

# Run an init inside the container that forwards signals and reaps processes.
#
#init = false

# Container init binary, if init=true, this is the init binary to be used for containers.
#
#init_path = "/usr/libexec/podman/catatonit"

# Default way to to create an IPC namespace (POSIX SysV IPC) for the container
# Options are:
# `private` Create private IPC Namespace for the container.
# `host`    Share host IPC Namespace with the container.
#
#ipcns = "private"

# keyring tells the container engine whether to create
# a kernel keyring for use within the container.
#
#keyring = true

# label tells the container engine whether to use container separation using
# MAC(SELinux) labeling or not.
# The label flag is ignored on label disabled systems.
#
#label = true

# Logging driver for the container. Available options: k8s-file and journald.
#
#log_driver = "k8s-file"

# Maximum size allowed for the container log file. Negative numbers indicate
# that no size limit is imposed. If positive, it must be >= 8192 to match or
# exceed conmon's read buffer. The file is truncated and re-opened so the
# limit is never exceeded.
#
#log_size_max = -1

# Specifies default format tag for container log messages.
# This is useful for creating a specific tag for container log messages.
# Containers logs default to truncated container ID as a tag.
#
#log_tag = ""

# Default way to to create a Network namespace for the container
# Options are:
# `private` Create private Network Namespace for the container.
# `host`    Share host Network Namespace with the container.
# `none`    Containers do not use the network
#
#netns = "private"

# Create /etc/hosts for the container.  By default, container engine manage
# /etc/hosts, automatically adding  the container's  own  IP  address.
#
#no_hosts = false

# Default way to to create a PID namespace for the container
# Options are:
# `private` Create private PID Namespace for the container.
# `host`    Share host PID Namespace with the container.
#
#pidns = "private"

# Maximum number of processes allowed in a container.
#
#pids_limit = 2048

# Copy the content from the underlying image into the newly created volume
# when the container is created instead of when it is started. If false,
# the container engine will not copy the content until the container is started.
# Setting it to true may have negative performance implications.
#
#prepare_volume_on_create = false

# Indicates the networking to be used for rootless containers
#
#rootless_networking = "slirp4netns"

# Path to the seccomp.json profile which is used as the default seccomp profile
# for the runtime.
#
#seccomp_profile = "/usr/share/containers/seccomp.json"

# Size of /dev/shm. Specified as <number><unit>.
# Unit is optional, values:
# b (bytes), k (kilobytes), m (megabytes), or g (gigabytes).
# If the unit is omitted, the system uses bytes.
#
#shm_size = "65536k"

# Set timezone in container. Takes IANA timezones as well as "local",
# which sets the timezone in the container to match the host machine.
#
#tz = ""

# Set umask inside the container
#
#umask = "0022"

# Default way to to create a User namespace for the container
# Options are:
# `auto`        Create unique User Namespace for the container.
# `host`    Share host User Namespace with the container.
#
#userns = "host"

# Number of UIDs to allocate for the automatic container creation.
# UIDs are allocated from the "container" UIDs listed in
# /etc/subuid & /etc/subgid
#
#userns_size = 65536

# Default way to to create a UTS namespace for the container
# Options are:
# `private`        Create private UTS Namespace for the container.
# `host`    Share host UTS Namespace with the container.
#
#utsns = "private"

# List of volumes. Specified as
# "<directory-on-host>:<directory-in-container>:<options>", for example:
# "/db:/var/lib/db:ro".
# If it is empty or commented out, no volumes will be added
#
#volumes = []

# The network table contains settings pertaining to the management of
# CNI plugins.

[secrets]
#driver = "file"

[secrets.opts]
#root = "/example/directory"

[network]

# Path to directory where CNI plugin binaries are located.
#
#cni_plugin_dirs = [
#  "/usr/local/libexec/cni",
#  "/usr/libexec/cni",
#  "/usr/local/lib/cni",
#  "/usr/lib/cni",
#  "/opt/cni/bin",
#]

# The network name of the default CNI network to attach pods to.
#
#default_network = "podman"

# The default subnet for the default CNI network given in default_network.
# If a network with that name does not exist, a new network using that name and
# this subnet will be created.
# Must be a valid IPv4 CIDR prefix.
#
#default_subnet = "10.88.0.0/16"

# Path to the directory where CNI configuration files are located.
#
#network_config_dir = "/etc/cni/net.d/"

[engine]
# Index to the active service
#
#active_service = production

# Cgroup management implementation used for the runtime.
# Valid options "systemd" or "cgroupfs"
#
#cgroup_manager = "systemd"

# Environment variables to pass into conmon
#
#conmon_env_vars = [
#  "PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin"
#]

# Paths to look for the conmon container manager binary
#
#conmon_path = [
#  "/usr/libexec/podman/conmon",
#  "/usr/local/libexec/podman/conmon",
#  "/usr/local/lib/podman/conmon",
#  "/usr/bin/conmon",
#  "/usr/sbin/conmon",
#  "/usr/local/bin/conmon",
#  "/usr/local/sbin/conmon"
#]

# Specify the keys sequence used to detach a container.
# Format is a single character [a-Z] or a comma separated sequence of
# `ctrl-<value>`, where `<value>` is one of:
# `a-z`, `@`, `^`, `[`, `\`, `]`, `^` or `_`
#
#detach_keys = "ctrl-p,ctrl-q"

# Determines whether engine will reserve ports on the host when they are
# forwarded to containers. When enabled, when ports are forwarded to containers,
# ports are held open by as long as the container is running, ensuring that
# they cannot be reused by other programs on the host. However, this can cause
# significant memory usage if a container has many ports forwarded to it.
# Disabling this can save memory.
#
#enable_port_reservation = true

# Environment variables to be used when running the container engine (e.g., Podman, Buildah).
# For example "http_proxy=internal.proxy.company.com".
# Note these environment variables will not be used within the container.
# Set the env section under [containers] table, if you want to set environment variables for the container.
#
#env = []

# Selects which logging mechanism to use for container engine events.
# Valid values are `journald`, `file` and `none`.
#
#events_logger = "journald"

# A is a list of directories which are used to search for helper binaries.
#
#helper_binaries_dir = [
#  "/usr/local/libexec/podman",
#  "/usr/local/lib/podman",
#  "/usr/libexec/podman",
#  "/usr/lib/podman",
#]

# Path to OCI hooks directories for automatically executed hooks.
#
#hooks_dir = [
#  "/usr/share/containers/oci/hooks.d",
#]

# Manifest Type (oci, v2s2, or v2s1) to use when pulling, pushing, building
# container images. By default image pulled and pushed match the format of the
# source image. Building/committing defaults to OCI.
#
#image_default_format = ""

# Default transport method for pulling and pushing for images
#
#image_default_transport = "docker://"

# Maximum number of image layers to be copied (pulled/pushed) simultaneously.
# Not setting this field, or setting it to zero, will fall back to containers/image defaults.
#
#image_parallel_copies = 0

# Default command to run the infra container
#
#infra_command = "/pause"

# Infra (pause) container image name for pod infra containers.  When running a
# pod, we start a `pause` process in a container to hold open the namespaces
# associated with the  pod.  This container does nothing other then sleep,
# reserving the pods resources for the lifetime of the pod.
#
#infra_image = "k8s.gcr.io/pause:3.4.1"

# Specify the locking mechanism to use; valid values are "shm" and "file".
# Change the default only if you are sure of what you are doing, in general
# "file" is useful only on platforms where cgo is not available for using the
# faster "shm" lock type.  You may need to run "podman system renumber" after
# you change the lock type.
#
#lock_type** = "shm"

# Indicates if Podman is running inside a VM via Podman Machine.
# Podman uses this value to do extra setup around networking from the
# container inside the VM to to host.
#
#machine_enabled = false

# MultiImageArchive - if true, the container engine allows for storing archives
# (e.g., of the docker-archive transport) with multiple images.  By default,
# Podman creates single-image archives.
#
#multi_image_archive = "false"

# Default engine namespace
# If engine is joined to a namespace, it will see only containers and pods
# that were created in the same namespace, and will create new containers and
# pods in that namespace.
# The default namespace is "", which corresponds to no namespace. When no
# namespace is set, all containers and pods are visible.
#
#namespace = ""

# Path to the slirp4netns binary
#
#network_cmd_path = ""

# Default options to pass to the slirp4netns binary.
# For example "allow_host_loopback=true"
#
#network_cmd_options = ["enable_ipv6=true",]

# Whether to use chroot instead of pivot_root in the runtime
#
#no_pivot_root = false

# Number of locks available for containers and pods.
# If this is changed, a lock renumber must be performed (e.g. with the
# 'podman system renumber' command).
#
#num_locks = 2048

# Whether to pull new image before running a container
#
#pull_policy = "missing"

# Indicates whether the application should be running in remote mode. This flag modifies the
# --remote option on container engines. Setting the flag to true will default
# `podman --remote=true` for access to the remote Podman service.
#
#remote = false

# Default OCI runtime
#
#runtime = "crun"

# List of the OCI runtimes that support --format=json.  When json is supported
# engine will use it for reporting nicer errors.
#
#runtime_supports_json = ["crun", "runc", "kata", "runsc", "krun"]

# List of the OCI runtimes that supports running containers with KVM Separation.
#
#runtime_supports_kvm = ["kata", "krun"]

# List of the OCI runtimes that supports running containers without cgroups.
#
#runtime_supports_nocgroups = ["crun", "krun"]

# Default location for storing temporary container image content.  Can be overridden with the TMPDIR environment
# variable.  If you specify "storage", then the location of the
# container/storage tmp directory will be used.
# image_copy_tmp_dir="/var/tmp"

# Number of seconds to wait without a connection
# before the `podman system service` times out and exits
#
#service_timeout = 5

# Directory for persistent engine files (database, etc)
# By default, this will be configured relative to where the containers/storage
# stores containers
# Uncomment to change location from this default
#
#static_dir = "/var/lib/containers/storage/libpod"

# Number of seconds to wait for container to exit before sending kill signal.
#
#stop_timeout = 10

# map of service destinations
#
#[service_destinations]
#  [service_destinations.production]
#     URI to access the Podman service
#     Examples:
#       rootless "unix://run/user/$UID/podman/podman.sock" (Default)
#       rootfull "unix://run/podman/podman.sock (Default)
#       remote rootless ssh://engineering.lab.company.com/run/user/1000/podman/podman.sock
#       remote rootfull ssh://root@10.10.1.136:22/run/podman/podman.sock
#
#    uri = "ssh://user@production.example.com/run/user/1001/podman/podman.sock"
#    Path to file containing ssh identity key
#    identity = "~/.ssh/id_rsa"

# Directory for temporary files. Must be tmpfs (wiped after reboot)
#
#tmp_dir = "/run/libpod"

# Directory for libpod named volumes.
# By default, this will be configured relative to where containers/storage
# stores containers.
# Uncomment to change location from this default.
#
#volume_path = "/var/lib/containers/storage/volumes"

# Paths to look for a valid OCI runtime (crun, runc, kata, runsc, krun, etc)
[engine.runtimes]
#crun = [
#  "/usr/bin/crun",
#  "/usr/sbin/crun",
#  "/usr/local/bin/crun",
#  "/usr/local/sbin/crun",
#  "/sbin/crun",
#  "/bin/crun",
#  "/run/current-system/sw/bin/crun",
#]

#kata = [
#  "/usr/bin/kata-runtime",
#  "/usr/sbin/kata-runtime",
#  "/usr/local/bin/kata-runtime",
#  "/usr/local/sbin/kata-runtime",
#  "/sbin/kata-runtime",
#  "/bin/kata-runtime",
#  "/usr/bin/kata-qemu",
#  "/usr/bin/kata-fc",
#]

#runc = [
#  "/usr/bin/runc",
#  "/usr/sbin/runc",
#  "/usr/local/bin/runc",
#  "/usr/local/sbin/runc",
#  "/sbin/runc",
#  "/bin/runc",
#  "/usr/lib/cri-o-runc/sbin/runc",
#]

#runsc = [
#  "/usr/bin/runsc",
#  "/usr/sbin/runsc",
#  "/usr/local/bin/runsc",
#  "/usr/local/sbin/runsc",
#  "/bin/runsc",
#  "/sbin/runsc",
#  "/run/current-system/sw/bin/runsc",
#]

#krun = [
#  "/usr/bin/krun",
#  "/usr/local/bin/krun",
#]

[engine.volume_plugins]
#testplugin = "/run/podman/plugins/test.sock"

[machine]
# Number of CPU's a machine is created with.
#
#cpus=1

# The size of the disk in GB created when init-ing a podman-machine VM.
#
#disk_size=10

# The image used when creating a podman-machine VM.
#
#image = "testing"

# Memory in MB a machine is created with.
#
#memory=2048

# The [machine] table MUST be the last entry in this file.
# (Unless another table is added)
# TOML does not provide a way to end a table other than a further table being
# defined, so every key hereafter will be part of [machine] and not the
# main config.

[nikitakuchur]

I got this bug after containers-common v0.46.0 update. The machine table is a new feature and I think it has not been supported in podman yet. I found a temporary fix, you can just comment out "[machine]" in your config file.

[rhatdan]

archlinux must have pulled the latest containers.conf file from the main branch to an older podman. I am updating Podman now in main to support the new fields.

[wirewc]

Arch user, can confirm commenting the [machine] lines out of the files solved it for me. Files I had to edit:

• /usr/share/containers/containers.conf
• /etc/containers/containers.conf

I've been using buildah for while on that machine, it was definitely time to mv storage.conf.pacnew storage.conf to resolve some other issues.

[marusak]

This is not just the case on Arch. Current Fedora-35 has the same issue. Any podman call has bunch of warnings:

# podman run -dit busybox sh
WARN[0000] Failed to decode the keys ["machine"] from "/usr/share/containers/containers.conf". 
WARN[0000] Failed to decode the keys ["machine"] from "/usr/share/containers/containers.conf". 
WARN[0000] Failed to decode the keys ["machine"] from "/usr/share/containers/containers.conf". 

[wirewc]

@marusak comment out the line in you config that has [machine] and your issue will go away. Upstream with buildah and podman the changes were made and your distro require you to update your config file. The only change will be the [machine] and nothing in that group is specified looking at the error message.

[marusak]

Right, I was just trying to say, that clean fedora-35 image has this issue as well.

[rhatdan]

rpm -q podman

[rhatdan]

podman 3.4 should not have a problem.

yum -y update podman --enable-repo=updates-testing

[marusak]

podman-3.4.0-0.11.rc2.fc35.x86_64 was affected, updating to 3.4.0-1.fc35.x86_64 works.