"Failed to create volume" when executing a task (Running Concourse CI in Docker)

[petermoberg-smarsh]

Summary

While working through the Concourse tutorial here https://concoursetutorial.com/basics/task-hello-world.html I wanted to run the hello-world task to make sure that my concourse installation is working. I downloaded the tutorial git repo and proceded with a fly execute on the hello-world task. Instead of getting the output that the tutorial was describing I get an error message saying that "failed to create volume". I tried other tasks available in the tutorial and received the same error.

Steps to reproduce

After Concourse is up and running with Docker I logged in to Concourse, created a team and then created a target called ci that points to the new team.

After that I just did

$ fly -t ci execute -c task_hello_world.yml

Expected results

Expecting to see the task being executed successfully and eventually outputting "Hello world".

Actual results

Fly output:

$ fly -t ci execute -c task_hello_world.yml

initializing
initializing check: image
selected worker: e928147d49b7
run check: find or create container on worker e928147d49b7: failed to create volume
run check: find or create container on worker e928147d49b7: failed to create volume
errored

Worker logs:

{"timestamp":"2022-04-06T17:05:15.368512739Z","level":"info","source":"baggageclaim","message":"baggageclaim.api.volume-server.get-volume.get-volume.volume-not-found","data":{"session":"4.1.4.1","volume":"331d2d8e-8615-465d-434b-cf06c74c6e83"}}

{"timestamp":"2022-04-06T17:05:15.368568316Z","level":"info","source":"baggageclaim","message":"baggageclaim.api.volume-server.get-volume.volume-not-found","data":{"session":"4.1.4","volume":"331d2d8e-8615-465d-434b-cf06c74c6e83"}}

{"timestamp":"2022-04-06T17:05:24.561909118Z","level":"info","source":"baggageclaim","message":"baggageclaim.api.volume-server.get-volume.get-volume.volume-not-found","data":{"session":"4.1.14.1","volume":"2ba233eb-42e0-45d8-5a40-b75167cd7277"}}

{"timestamp":"2022-04-06T17:05:24.561969501Z","level":"info","source":"baggageclaim","message":"baggageclaim.api.volume-server.get-volume.volume-not-found","data":{"session":"4.1.14","volume":"2ba233eb-42e0-45d8-5a40-b75167cd7277"}}

{"timestamp":"2022-04-06T17:05:24.575712968Z","level":"error","source":"baggageclaim","message":"baggageclaim.api.volume-server.create-volume-async.failed-to-create","data":{"error":"invalid argument","handle":"2ba233eb-42e0-45d8-5a40-b75167cd7277","privileged":true,"session":"4.1.15","strategy":{"type":"cow","volume":"331d2d8e-8615-465d-434b-cf06c74c6e83"}}}

Additional context

Concourse CI is running in a Docker Compose environment.
Underlying hardware is a Synology NAS running DSM 7.0.1 (Linux with btrfs filesystem).

Here follows the docker compose filed that I used:

services:
  web:
    image: concourse/concourse
    command: web
    ports: ["19201:8080"]
    volumes: ["/volume1/docker/concourse-ci/web:/concourse-keys"]
    environment:
      CONCOURSE_EXTERNAL_URL: <redacted>
      CONCOURSE_POSTGRES_HOST: <redacted>
      CONCOURSE_POSTGRES_PORT: <redacted>
      CONCOURSE_POSTGRES_USER: <redacted>
      CONCOURSE_POSTGRES_PASSWORD: <redacted>
      CONCOURSE_POSTGRES_DATABASE: concourse     
      CONCOURSE_GITLAB_CLIENT_ID: <redacted>
      CONCOURSE_GITLAB_CLIENT_SECRET: <redacted>
      CONCOURSE_GITLAB_HOST: <redacted>
      CONCOURSE_MAIN_TEAM_GITLAB_USER: <redacted>
    logging:
      driver: "json-file"
      options:
        max-file: "5"
        max-size: "10m"

  worker:
    image: concourse/concourse
    command: worker
    privileged: true
    depends_on: [web]
    volumes: ["/volume1/docker/concourse-ci/worker:/concourse-keys", "/volume1/docker/concourse-ci/worker-state:/worker-state"]
    links: [web]
    stop_signal: SIGUSR2
    environment:
      CONCOURSE_TSA_HOST: web:2222
      # enable DNS proxy to support Docker's 127.x.x.x DNS server
      CONCOURSE_GARDEN_DNS_PROXY_ENABLE: "true"
      CONCOURSE_WORK_DIR: "/worker-state"
    logging:
      driver: "json-file"
      options:
        max-file: "5"
        max-size: "10m"

Triaging info

• Concourse version: 7.7.1
• Browser (if applicable): N/A
• Did this used to work? First time trying this so can't tell.

[petermoberg-smarsh]

Some updates:

If I set the following environment variables on the worker I can get my pipeline to work.

CONCOURSE_BAGGAGECLAIM_DRIVER: "naive"
CONCOURSE_RUNTIME: "houdini"

It seems like having the driver be either: detect, btrfs or overlay will not work.
The only driver that works is naive

If I don't set the runtime to houdini I get this error:

run check: find or create container on worker f3bc672b4ab3: failed to retrieve kernel parameter "net.ipv4.tcp_keepalive_time": open /proc/sys/net/ipv4/tcp_keepalive_time: no such file or directory

[petermoberg-smarsh]

Not sure what is going on but when I tried again I now get this error:

initializing
initializing check: image
selected worker: 4a7bff5e4a55
selected worker: 4a7bff5e4a55
waiting for docker to come up...
Pulling busybox@sha256:14d4f50961544fdb669075c442509f194bdc4c0e344bde06e35dbd55af842a38...
docker.io/library/busybox@sha256:14d4f50961544fdb669075c442509f194bdc4c0e344bde06e35dbd55af842a38: Pulling from library/busybox
554879bb3004: Pulling fs layer
554879bb3004: Verifying Checksum
554879bb3004: Download complete
failed to register layer: ApplyLayer exit status 1 stdout:  stderr: remount /, flags: 0x84000: invalid argument

I am not sure why I got the image to download before but now it fails everytime with above message...

[taylorsilva]

Did you try containerd as the container runtime by chance?

Houdini and naive as your runtime and storage driver basically mean you're not using containers at all.

We generally recommend using overlay and containerd.

[ponytin]

I use overlay and containerd on ubuntu 20.04 with cgroup v2 enabled, and facing the same issue too. But I found it's only happened to worker-state mounted in nfs volume. It's ok to have worker-state be mapped to the local drive.

[ChrisJBurns]

@ponytin How would you do this via the Helm Chart? Currently using the following we get the error in the issue:

  values:
    image: URL/third-party/concourse
    imageTag: 7.8.3
    imagePullPolicy: Always
    concourse:
      web:
        nodeSelector: |
          Capability: k8s-concourse-workers
      worker:
        autoscaling:
          maxReplicas: 3
          minReplicas: 3
        baggageclaim:
          driver: overlay
        runtime: containerd
        containerd:
          dnsProxyEnable: false
    worker:
       kind: Deployment
       replicas: 3

[gsaslis]

Hitting the same issue on a mac m2, with concourse 7.10.0 and

      CONCOURSE_WORKER_RUNTIME: houdini
      CONCOURSE_WORKER_BAGGAGECLAIM_DRIVER: naive
      CONCOURSE_WORKER_BAGGAGECLAIM_DISABLE_USER_NAMESPACES: true

[edwardCao]

I have the same issue with Mac M1 with concourse but I can trigger jobs for a couple of times then I start to have this error. I have to restart docker-compose and then it will work again for a while.

    image: concourse/concourse:7.9.1-ubuntu
    platform: linux/amd64
      CONCOURSE_WORKER_BAGGAGECLAIM_DRIVER: naive
      CONCOURSE_WORKER_BAGGAGECLAIM_DISABLE_USER_NAMESPACES: true
      CONCOURSE_WORKER_RUNTIME: houdini

[gsaslis]

@edwardCao I've got a relatively stable concourse deployment (not entirely without its issues) on arm64 (with podman desktop) and :

    image: rdclda/concourse:7.9.1
    platform: linux/arm64
    environment:
          CONCOURSE_WORKER_RUNTIME: "containerd"
          CONCOURSE_WORKER_BAGGAGECLAIM_DRIVER: overlay

wanna give that a try ?