Resources failing in Kubernetes with Google Container-Optimized OS after upgrade to 3.1.0

[ese]

Bug Report

• Concourse version: 3.1.0
• Deployment type (Docker):
• Infrastructure/IaaS: Kubernetes

After upgrade to 3.1.0 all git and time resources are failing checks with:

runc create: exit status 1: container_linux.go:264: starting container process caused "process_linux.go:339: container init caused \"rootfs_linux.go:57: mounting \\\"/worker-state/3.1.0/assets/bin/init\\\" to rootfs \\\"/worker-state/volumes/live/26e7c69d-69fc-4f0f-507d-4b30c461a78f/volume\\\" at \\\"/worker-state/volumes/live/26e7c69d-69fc-4f0f-507d-4b30c461a78f/volume/tmp/garden-init\\\" caused \\\"open /worker-state/volumes/live/26e7c69d-69fc-4f0f-507d-4b30c461a78f/volume/tmp/garden-init: permission denied\\\"\""

Other resources seems to check fine

[devdavidkarlsson]

It appears that something has happened to how the permissions of the volumes are handled... related? #1231

[vito]

Does a recreate of the worker, blowing away the volumes/worker state, fix this?

[ese]

@vito Recreate workers totally from scratch (volumes and all stuff) didn't help. We finally downgrade to 3.0.1

[jaredstehler]

I'm seeing a similar error for a docker build step after upgrading to 3.1.1:

mount: permission denied (are you root?)

Is this the same issue or a new one? I'm using the pinned v1.12.6 version for the ECR auth workaround:

- name: docker-image-2
  type: docker-image
  privileged: true
  source:
    repository: concourse/docker-image-resource
    tag: docker-1.12.6

I've recreated workers.

[robertjsullivan]

We are seeing something similiar on 3.1.0:

runc create: exit status 1: container_linux.go:262: starting container process caused "process_linux.go:339: container init caused "rootfs_linux.go:57: mounting \"/var/vcap/data/baggageclaim/volumes/live/b367226d-376c-42a1-7802-4d5962b835a2/volume\" to rootfs \"/var/vcap/data/garden/graph/aufs/mnt/ba10b98473cc0ffa00c8b004a6adeb98e4c3897a5cb99e7cb9a56f93ad72a259\" at \"/var/vcap/data/garden/graph/aufs/mnt/ba10b98473cc0ffa00c8b004a6adeb98e4c3897a5cb99e7cb9a56f93ad72a259/scratch\" caused \"mkdir /var/vcap/data/garden/graph/aufs/mnt/ba10b98473cc0ffa00c8b004a6adeb98e4c3897a5cb99e7cb9a56f93ad72a259/scratch: permission denied\"""

recreating the workers did not fix it.

[robertjsullivan]

We were able to fix our issue by switching to image_resource: instead of image: in our task config.

[clarafu]

@jaredstehler Confirmed that that doesn't seem to work. Thanks for the pointer!

[schuth]

I am facing the same issue when running Concourse: 3.1.1 running on Kubernetes. All resources fail.

[superbeeny]

Oddly I'm only seeing the cf-resource fail with the failed to create volume, a few retries will usually get it to work. I have the Baggage claim logs

baggageclaim.zip

[zaa]

Unfortunately we too experience the issue with Concourse 3.1.1 on AWS (running on Kubernetes using the helm chart). OS: Debian Jessie. Baggageclaim driver: overlay.

The problem can be reproduced with help of the following pipeline:

jobs:
  - name: test
    plan:
      - task: test
        config:
          platform: linux
          image_resource:
            type: docker-image
            source:
              repository: alpine
          run:
            path: ls
            args: ["-la", "."]

Concourse successfully pulls the docker image, but stumbles on running the task in non-privileged container:

{"timestamp":"1497359514.769297361","source":"guardian","message":"guardian.api.garden-server.create.creating","log_level":0,"data":{"request":{"Handle":"c2888395-d0bb-4e78-5da2-265b128f72c3","GraceTime":0,"RootFSPath":"raw:///concourse-work-dir/volumes/live/9481101c-fc45-475f-453e-072880b49f12/volume","BindMounts":[{"src_path":"/concourse-work-dir/volumes/live/573b155c-76ab-4ccf-4f59-5ccad44a2b30/volume","dst_path":"/scratch","mode":1}],"Network":"","Privileged":false,"Limits":{"bandwidth_limits":{},"cpu_limits":{},"disk_limits":{},"memory_limits":{},"pid_limits":{}}},"session":"3.1.216"}}
{"timestamp":"1497359514.769345284","source":"guardian","message":"guardian.create.start","log_level":1,"data":{"handle":"c2888395-d0bb-4e78-5da2-265b128f72c3","session":"189"}}
{"timestamp":"1497359514.769390821","source":"guardian","message":"guardian.create.containerizer-create.start","log_level":1,"data":{"handle":"c2888395-d0bb-4e78-5da2-265b128f72c3","session":"189.2"}}
{"timestamp":"1497359514.786541939","source":"guardian","message":"guardian.create.containerizer-create.depot-create.started","log_level":1,"data":{"handle":"c2888395-d0bb-4e78-5da2-265b128f72c3","session":"189.2.1"}}
{"timestamp":"1497359514.787511349","source":"guardian","message":"guardian.create.containerizer-create.depot-create.finished","log_level":1,"data":{"handle":"c2888395-d0bb-4e78-5da2-265b128f72c3","session":"189.2.1"}}
{"timestamp":"1497359514.787638426","source":"guardian","message":"guardian.create.containerizer-create.lookup.started","log_level":0,"data":{"handle":"c2888395-d0bb-4e78-5da2-265b128f72c3","session":"189.2.2"}}
{"timestamp":"1497359514.787839890","source":"guardian","message":"guardian.create.containerizer-create.lookup.finished","log_level":0,"data":{"handle":"c2888395-d0bb-4e78-5da2-265b128f72c3","session":"189.2.2"}}
{"timestamp":"1497359514.787889004","source":"guardian","message":"guardian.create.containerizer-create.create.creating","log_level":1,"data":{"bundle":"/concourse-work-dir/depot/c2888395-d0bb-4e78-5da2-265b128f72c3","bundlePath":"/concourse-work-dir/depot/c2888395-d0bb-4e78-5da2-265b128f72c3","handle":"c2888395-d0bb-4e78-5da2-265b128f72c3","id":"c2888395-d0bb-4e78-5da2-265b128f72c3","logPath":"/concourse-work-dir/depot/c2888395-d0bb-4e78-5da2-265b128f72c3/create.log","pidFilePath":"/concourse-work-dir/depot/c2888395-d0bb-4e78-5da2-265b128f72c3/pidfile","runc":"/concourse-work-dir/3.1.1/assets/bin/runc","session":"189.2.3"}}
{"timestamp":"1497359514.888193846","source":"guardian","message":"guardian.create.containerizer-create.create.runc","log_level":0,"data":{"bundle":"/concourse-work-dir/depot/c2888395-d0bb-4e78-5da2-265b128f72c3","handle":"c2888395-d0bb-4e78-5da2-265b128f72c3","message":"exit status 1","session":"189.2.3"}}
{"timestamp":"1497359514.888237238","source":"guardian","message":"guardian.create.containerizer-create.create.runc","log_level":0,"data":{"bundle":"/concourse-work-dir/depot/c2888395-d0bb-4e78-5da2-265b128f72c3","handle":"c2888395-d0bb-4e78-5da2-265b128f72c3","message":"container_linux.go:264: starting container process caused \"process_linux.go:339: container init caused \\\"rootfs_linux.go:57: mounting \\\\\\\"/concourse-work-dir/3.1.1/assets/bin/init\\\\\\\" to rootfs \\\\\\\"/concourse-work-dir/volumes/live/9481101c-fc45-475f-453e-072880b49f12/volume\\\\\\\" at \\\\\\\"/concourse-work-dir/volumes/live/9481101c-fc45-475f-453e-072880b49f12/volume/tmp/garden-init\\\\\\\" caused \\\\\\\"open /concourse-work-dir/volumes/live/9481101c-fc45-475f-453e-072880b49f12/volume/tmp/garden-init: permission denied\\\\\\\"\\\"\"\n","session":"189.2.3"}}
{"timestamp":"1497359514.888260841","source":"guardian","message":"guardian.create.containerizer-create.create.runc","log_level":0,"data":{"bundle":"/concourse-work-dir/depot/c2888395-d0bb-4e78-5da2-265b128f72c3","handle":"c2888395-d0bb-4e78-5da2-265b128f72c3","message":"container_linux.go:264: starting container process caused \"process_linux.go:339: container init caused \\\"rootfs_linux.go:57: mounting \\\\\\\"/concourse-work-dir/3.1.1/assets/bin/init\\\\\\\" to rootfs \\\\\\\"/concourse-work-dir/volumes/live/9481101c-fc45-475f-453e-072880b49f12/volume\\\\\\\" at \\\\\\\"/concourse-work-dir/volumes/live/9481101c-fc45-475f-453e-072880b49f12/volume/tmp/garden-init\\\\\\\" caused \\\\\\\"open /concourse-work-dir/volumes/live/9481101c-fc45-475f-453e-072880b49f12/volume/tmp/garden-init: permission denied\\\\\\\"\\\"\"\n","session":"189.2.3"}}
{"timestamp":"1497359514.888289213","source":"guardian","message":"guardian.create.containerizer-create.create.finished","log_level":1,"data":{"bundle":"/concourse-work-dir/depot/c2888395-d0bb-4e78-5da2-265b128f72c3","handle":"c2888395-d0bb-4e78-5da2-265b128f72c3","session":"189.2.3"}}

If I set the task's privileged attribute to "true" it starts working.

I tinkered with the configuration of the worker a bit and found out that the issue disappears when I switch the worker to "naive" baggageclaim driver (start the worker with --baggageclaim-driver=naive command line flag). I presume the issue has something to do with running non-privileged containers using runc from a volume backed by the overlay fs driver.

[schuth]

This is still an issue for me with 3.2.1 - any plans to fix this?

[markstgodard]

I can also confirm I'm getting similar error on 3.3.0 deployed to k8s 1.6.4 on GKE via the helm chart:

Example output from a git resource failure

runc create: exit status 1: container_linux.go:264: starting container process caused "process_linux.go:339: container init caused \"rootfs_linux.go:57: mounting \\\"/concourse-work-dir/3.3.0/assets/bin/init\\\" to rootfs \\\"/concourse-work-dir/volumes/live/724a870f-a34b-4cd2-5509-125b652e0a77/volume\\\" at \\\"/concourse-work-dir/volumes/live/724a870f-a34b-4cd2-5509-125b652e0a77/volume/tmp/garden-init\\\" caused \\\"open /concourse-work-dir/volumes/live/724a870f-a34b-4cd2-5509-125b652e0a77/volume/tmp/garden-init: permission denied\\\"\""

[ese]

It seems to be related with some kernel param

I can reproduce it with kernels
4.4.35+ (Container-Optimized OS -google cloud-)
4.4.65-k8s (debian kubernetes kops)

It works fine in:
4.4.0 (ubuntu xenial)
4.9.24 coreos

Pipeline to test

---
jobs:
  - name: test-overlay
    plan:
      - get: time
      - task: test-task
        config:
          platform: linux
          image_resource:
            type: docker-image
            source:
              repository: busybox
          run:
            path: sh
            args:
              - -c
              - |
                echo hello world

resources:
  - name: time
    type: time
    source:
      interval: 1m

[jtarchie]

@ese, thanks for reporting. I don't have the needed expertise to reproduce this issue with k8. It would be really useful if you could provide steps to reproduce this locally, using minikube.

[vito]

@ese That's a great lead, thanks! Can anyone else confirm this?

We may just have to document/detect a minimum kernel requirement for the overlay volume driver.

[markstgodard]

I can confirm my Concourse 3.3.0 deployment to GKE k8s 1.6.4 cluster, workers running: linux kernel 4.4.35+ has the issue.

From kubectl get nodes

System Info:
 Kernel Version:                4.4.35+
 OS Image:                      Container-Optimized OS from Google
 Operating System:              linux
 Architecture:                  amd64
 Container Runtime Version:     docker://1.11.2
 Kubelet Version:               v1.6.4
 Kube-Proxy Version:            v1.6.4

[watawuwu]

I tinkered with the configuration of the worker a bit and found out that the issue disappears when I switch the worker to "naive" baggageclaim driver (start the worker with --baggageclaim-driver=naive command line flag).

I was able to avoid it if I started the worker with --baggageclaim-driver=naive as environment variables.

# spec.template.spec.containers[].env:
            - name: CONCOURSE_BAGGAGECLAIM_DRIVER
              value: "naive"

Kubernetes 1.6.4(on GKE)
Concourse 3.3.0(from Helm concourse-0.1.3)

[zaa]

The issue might have something to do with user namespaces support in Linux kernel versions less than 4.8.

From https://github.com/oracle/crashcart#known-issues

crashcart doesn't work with user namespaces prior to kernel 4.8. In earlier versions of the kernel, when you attempt to mount a device inside a mount namespace that is a child of a user namespace, the kernel returns EPERM. The logic was changed in 4.8 so that it is possible as long as the caller of mount is in the init userns.

Sounds familiar.

[viglesiasce]

Thanks for chasing this @avanier!! Much appreciated.

[avanier]

Hiya all,

After much poking and prodding of our friends at Google, we've just tested GKE with Kubernetes 1.7.8, which brings Container Optimized OS milestone 61, which has Docker running on Overlay2... and it still gives the same error.

(╯°□°)╯︵ ┻━┻

I'm at a loss for words right now... 😞

[avanier]

So... to put an epilogue to this saga, we're now running web in on GKE, and our workers now run off CoreOS Stable, bootstrapped with Ignition, in GCE.

(ﾉ≧∇≦)ﾉ ﾐ ┸━┸

We're always willing to test, should anyone want to give another go at this.

Edit: Settled on an even cuter table-flip-guy.

[yaolingling]

@zaa , thanks for your solution. I have solved this problem. Only need to add one line "CONCOURSE_BAGGAGECLAIM_DRIVER: naive" in the file "docker-compose.yml" to config the concourse-worker's "environment"

[rjain-pivotal]

This is still an issue with the Concourse 3.8.0 on GKE with the following version...

Client Version: version.Info{Major:"1", Minor:"8", GitVersion:"v1.8.4", GitCommit:"9befc2b8928a9426501d3bf62f72849d5cbcd5a3", GitTreeState:"clean", BuildDate:"2017-11-20T05:28:34Z", GoVersion:"go1.8.3", Compiler:"gc", Platform:"darwin/amd64"}

Server Version: version.Info{Major:"1", Minor:"8+", GitVersion:"v1.8.1-gke.1", GitCommit:"aba494e68a76583d2d7d1b9c97e4a97a19c3a920", GitTreeState:"clean", BuildDate:"2017-10-27T23:54:39Z", GoVersion:"go1.8.3b4", Compiler:"gc", Platform:"linux/amd64"}

{"error":"failed to create btrfs filesystem: exit status 32"}}

[philss]

I had a similar problem with k8s on GKE 1.8.7 deploying Concourse using Helm (also, using cos).

I tried to use the version 3.9.0 of Concourse with btrfs without success. The deploy worked, but when I was trying to execute a build, it was showing "No workers" in red. After deleting the Helm installation (with helm del --purge concourse) and reinstalling with naive option, it worked.

imageTag: "3.9.0"
concourse:
  baggageclaimDriver: naive

[vito]

Is this still a problem for anyone? It's been quiet for a while now, and I wonder if any changes have been made out-of-band (i.e. on Google's side) to resolve this.

[avanier]

Errr... I believe @fkoclas mentioned something to us a while ago about changes made by Google on GKE. I'll try and stir answers out of knowledgeable people by end of day.

cc @Typositoire ?

[Typositoire]

I do recall @fkoclas saying that GKE now support nodes with at least Overlay2 (No idea what happened with BtrFS)

[rjain-pivotal]

I can give this a try on GKE and report back

…

On Fri, Jul 6, 2018 at 10:29 AM Yann David ***@***.***> wrote: I do recall @fkoclas <https://github.com/fkoclas> saying that GKE now support nodes with at least Overlay2 (No idea what happened with BtrFS) — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#1230 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AJm9M4dLFvMzhxF-m6HW8Gdp9e-12WA-ks5uD3RPgaJpZM4NrNc5> .

-- Rajesh Jain Advisory Cloud Foundry Platform Architect, Pivotal <http://www.pivotal.io/> e:rajesh.jain@pivotal.io | m: 2487900082 | t: @rjain15 <http://www.twitter.com/rjain15> | l: @rjain15 <https://www.linkedin.com/in/rjain15>

[vito]

@rjain-pivotal Thanks!

[fkoclas]

Kubernetes v1.10.5-gke.0 on GKE (cos image - not ubuntu) ships with a recent kernel, 4.14.22, which shouldn't have problems with overlay2.

[mrparkers]

We are in the process of upgrading Concourse from 3.8 (using overlay driver and Ubuntu image) to the latest version, and I spun up a few test GKE clusters today to see what the driver compatibility looks for the different GKE image types on a more recent Kubernetes version.

Running on 1.10.4-gke.2 with version 3.14.1 of Concourse...

• Ubuntu image with overlay driver works fine
• Ubuntu image with btrfs driver works fine
• COS image with overlay driver works fine
• COS image with btrfs does not work - fails with failed to create btrfs filesystem: exit status 32 in the logs

Given this, we are probably going to stay on the Ubuntu image and switch to the btrfs driver when upgrading our production instance to 3.14.1.

[evelant]

This is still happening. Went to install concourse via helm in my gke version 1.9.6.gke2 cluster on cos and ran into this issue.

[chrishiestand]

@AndrewMorsillo what baggage claim driver are you using?

I've been running concourse 3.14.1 on GKE (via helm chart). I've used the naive driver on cos and btrfs on ubuntu with no such issues.

[evelant]

I was trying to user overlay. Switched to naive and it works but it prints a big warning about how it isn't recommended to use it.

[xchapter7x]

im seeing this issue on:

 Kernel Version:             4.4.121-k8s
 OS Image:                   Debian GNU/Linux 8 (jessie)
 Operating System:           linux
 Architecture:               amd64
 Container Runtime Version:  docker://17.3.2
 Kubelet Version:            v1.9.3
 Kube-Proxy Version:         v1.9.3

NAME            	CHART VERSION	APP VERSION	DESCRIPTION
stable/concourse	2.0.1        	4.2.1      	Concourse is a simple and scalable CI system.

[cirocosta]

Hey,

Given that we're way past 3.1.0 and the issue is specifically about Concourse on a GKE node backed by COS, and we test on GKE such cases (

concourse/topgun/k8s/baggageclaim_drivers_test.go

Lines 67 to 96 in e7f017b

	Entry("with btrfs on cos", Case{
	Driver: "btrfs",
	NodeImage: "cos",
	ShouldWork: false,
	}),
	Entry("with btrfs on ubuntu", Case{
	Driver: "btrfs",
	NodeImage: "ubuntu",
	ShouldWork: true,
	}),
	Entry("with overlay on cos", Case{
	Driver: "overlay",
	NodeImage: "cos",
	ShouldWork: true,
	}),
	Entry("with overlay on ubuntu", Case{
	Driver: "overlay",
	NodeImage: "ubuntu",
	ShouldWork: true,
	}),
	Entry("with naive on cos", Case{
	Driver: "naive",
	NodeImage: "cos",
	ShouldWork: true,
	}),
	Entry("with naive on ubuntu", Case{
	Driver: "naive",
	NodeImage: "ubuntu",
	ShouldWork: true,
	}),

), I'll close this one.

Feel free to open other issues around other IaaSes or platforms you're seeing problems!

Thanks