podman hangs on build

[alechenninger]

Issue Description

podman build hangs indefinitely. I tried a couple of different Dockerfiles which used to work.

Steps to reproduce the issue

Steps to reproduce the issue

1. podman machine stop
2. podman machine start
3. podman build .

Describe the results you received

podman build hangs indefinitely.

Sometimes when I cancel the build with ctrl-c, then podman starts responding with:

$ podman version
Error: failed to connect: dial tcp [::1]:49780: connect: connection refused

But this is intermittent and doesn't always occur.

podman run works with various images.

Describe the results you expected

I expected the build to start.

podman info output

$ podman info
host:
  arch: arm64
  buildahVersion: 1.29.0
  cgroupControllers:
  - cpu
  - io
  - memory
  - pids
  cgroupManager: systemd
  cgroupVersion: v2
  conmon:
    package: conmon-2.1.5-1.fc37.aarch64
    path: /usr/bin/conmon
    version: 'conmon version 2.1.5, commit: '
  cpuUtilization:
    idlePercent: 94.62
    systemPercent: 4.73
    userPercent: 0.65
  cpus: 4
  distribution:
    distribution: fedora
    variant: coreos
    version: "37"
  eventLogger: journald
  hostname: localhost.localdomain
  idMappings:
    gidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 100000
      size: 1000000
    uidmap:
    - container_id: 0
      host_id: 501
      size: 1
    - container_id: 1
      host_id: 100000
      size: 1000000
  kernel: 6.1.11-200.fc37.aarch64
  linkmode: dynamic
  logDriver: journald
  memFree: 1425494016
  memTotal: 2049871872
  networkBackend: netavark
  ociRuntime:
    name: crun
    package: crun-1.8-1.fc37.aarch64
    path: /usr/bin/crun
    version: |-
      crun version 1.8
      commit: 0356bf4aff9a133d655dc13b1d9ac9424706cac4
      rundir: /run/user/501/crun
      spec: 1.0.0
      +SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +CRIU +LIBKRUN +WASM:wasmedge +YAJL
  os: linux
  remoteSocket:
    exists: true
    path: /run/user/501/podman/podman.sock
  security:
    apparmorEnabled: false
    capabilities: CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_NET_BIND_SERVICE,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID
    rootless: true
    seccompEnabled: true
    seccompProfilePath: /usr/share/containers/seccomp.json
    selinuxEnabled: true
  serviceIsRemote: true
  slirp4netns:
    executable: /usr/bin/slirp4netns
    package: slirp4netns-1.2.0-8.fc37.aarch64
    version: |-
      slirp4netns version 1.2.0
      commit: 656041d45cfca7a4176f6b7eed9e4fe6c11e8383
      libslirp: 4.7.0
      SLIRP_CONFIG_VERSION_MAX: 4
      libseccomp: 2.5.3
  swapFree: 0
  swapTotal: 0
  uptime: 0h 2m 55.00s
plugins:
  authorization: null
  log:
  - k8s-file
  - none
  - passthrough
  - journald
  network:
  - bridge
  - macvlan
  volume:
  - local
registries:
  search:
  - docker.io
store:
  configFile: /var/home/core/.config/containers/storage.conf
  containerStore:
    number: 0
    paused: 0
    running: 0
    stopped: 0
  graphDriverName: overlay
  graphOptions: {}
  graphRoot: /var/home/core/.local/share/containers/storage
  graphRootAllocated: 106769133568
  graphRootUsed: 2810376192
  graphStatus:
    Backing Filesystem: xfs
    Native Overlay Diff: "true"
    Supports d_type: "true"
    Using metacopy: "false"
  imageCopyTmpDir: /var/tmp
  imageStore:
    number: 0
  runRoot: /run/user/501/containers
  transientStore: false
  volumePath: /var/home/core/.local/share/containers/storage/volumes
version:
  APIVersion: 4.4.1
  Built: 1675940263
  BuiltTime: Thu Feb  9 05:57:43 2023
  GitCommit: ""
  GoVersion: go1.19.5
  Os: linux
  OsArch: linux/arm64
  Version: 4.4.1

Podman in a container

No

Privileged Or Rootless

Rootless

Upstream Latest Release

Yes

Additional environment details

MacOS 12.6.3

Additional information

Additional information like issue happens only occasionally or issue happens with a particular architecture or on a particular setting

[alechenninger]

w/ debug logging:

$ podman build --log-level debug --platform linux/arm64 . -t asl:latest
INFO[0000] podman filtering at log level debug
DEBU[0000] Called build.PersistentPreRunE(podman build --log-level debug --platform linux/arm64 . -t asl:latest)
DEBU[0000] SSH Ident Key "/Users/ahenning/.ssh/podman-machine-default" SHA256:c6Fc/izfStF1E0acV0HgG8xvup/SRfAGC/xJygNCNK4 ssh-ed25519
DEBU[0000] DoRequest Method: GET URI: http://d/v4.4.2/libpod/_ping
DEBU[0000] Loading registries configuration "/etc/containers/registries.conf"
DEBU[0000] Found credentials for registry.redhat.io in credential helper containers-auth.json in file /Users/ahenning/.config/containers/auth.json
DEBU[0000] DoRequest Method: POST URI: http://d/v4.4.2/libpod/build

[flouthoc]

@alechenninger Could you check if size of context directory is really big ? or is there any COPY/ADD statement which is adding a very big file, the reason I'm asking because sometimes it takes time to upload file to remote server.

[rhatdan]

@alechenninger when building from a MAC or WIndows, the entire context directory gets copied into the MAC, if the context directory is huge this will take some time.

[alechenninger]

Yeah, that was it. I had a ~4gb log file creep up on me. I'm not sure what hung the other builds, but things are working now. Thank you.

[Danvil]

@rhatdan I am running into a similar problem, but my context directory is "intentionally" huge. Is there a way to tell podman to ignore certain sub directories?

[rhatdan]

You can use a .containerignore file, but I am not sure if this is processed on the client side or the server side.

[sneko]

Suffering from the same issue. I don't get the logic:

1. I'm in a monorepository
2. I have a Dockerfile at the root to gather 2 "dist" folders from ./apps/app ./apps/api, I use:

COPY ./apps/app/dist /app/frontend
COPY ./apps/api /app/backend

So as you can see I'm not doing a COPY ./, so why it's taking so long? If I ignore everything in my .dockerignore with **/* and using docker build --log-level trace -t aei-test . I see the logs that it's skipping folders in the monorepo that were not targetted by COPY commands. Why is that?

I ended doing:

**/node_modules
!/apps/app/dist/node_modules

To avoid other subproject node_modules but still, it takes like ~20 seconds before really starting the Dockerfile steps.

It's really a huge pain and makes no sense to me :(