Performance regression with 'overlay' driver and privileged containers

[jeanbza]

Bug Report

• Concourse version: 3.3.1
• Deployment type (BOSH/Docker/binary): binary
• Infrastructure/IaaS: AWS
• Browser (if applicable): chrome
• Did this used to work? Yes

We are noticing that various tasks are hanging for 1-10m before executing. (the ui doesn't show the 'loading' icon spinning, just hangs)

Our setup:

• 2 workers with 16gb RAM and 4 cores, 1 web with 8gb and 4 cores (IIRC)
• Postgres db lives on the web VM
• Workers and web never get above 10% CPU
• Three pipelines with < 5 jobs, 1 pipeline with ~20 jobs, some of which have 3000+ builds (most are low hundreds though)

Things we have tried:

• Set build_logs_to_retain: 20 per @jtarchie

This was also seen by @krishicks in the Slack concourse#general channel, I believe.

[krishicks]

I have seen this issue repeatedly.

[pms1969]

I've seen this as well. Just been on slack, and directed here.

Concourse Configuration: binary deployment to AWS clusters. 2 ATC's, 2 Linux Worker, and 2 winows Workers. internal LB for workers to talk to ATC's, External LB for hitting the UI. CodeDeploy over regular aws ami's.

Slack discussion begins here: https://concourseci.slack.com/archives/C07RY25QF/p1500654086732765

As per the slack discussion, I tend to see this leading up to problems with the linux workers. Number of volumes grows until the workers just start giving up, and then I end up having to rebuild the workers.

Happy to try and get any other information that may be required when it's available.

[jeanbza]

Dropping to v3.1.1 and recreating our db did nothing. Dropping to 2.x, I bet it's that file system change thing.

[jeanbza]

Dropping to 2.7.7 fixed our issues.

[markstgodard]

I was using the following and DID NOT see this issue:

• Concourse 3.0.1 (standalone binaries)
• Google Compute Engine VMs (Ubuntu 14.04)
• baggage claim driver: btrfs

I recently upgraded to Concourse 3.3.4, and I DO see this issue.

I upgraded to Concourse 3.1.1 in between 3.0.1 and 3.3.4 and I don't believe I saw the issue, but I could be mistaken.

Note: we are using the default baggage claim driver: overlay fs in 3.3.4

It seems like certain Jobs are affected by it (more so than others).

I have one job that builds a docker image (see below)

All the tasks run quick, with no lag... but when it gets to the docker-image-resource "put" task for "app-image" it takes 6+ minutes for it to start spinning...

During that 6+ minutes, it visually just looks like it has not started.

I also see the there is a container listed for that task when I do a fly containers
However I cannot intercept into it... (get a ssh bad handshake... so I assume its because it has NOT yet spun up the container ? )

- name: build-image-backend
  serial_groups: [build-docker]
  plan:
  - aggregate:
    - get: ci-runner-image
    - get: ci-scripts
    - get: ci-secrets
    - get: backend-version
      params: {bump: patch}
    - get: backend-dev
      passed: [unit-tests-backend]
      trigger: true
  - aggregate:
    - get: base-backend-image
      params: {save: true}
      passed: [build-base-backend]
    - task: prep-docker-image
      image: ci-runner-image
      file: ci-scripts/tasks/backend/prep-docker-backend.yml
  - put: backend-image
    params:
      load_base: base-backend-image
      build: output
      tag: backend-version/number
      tag_as_latest: true
    get_params:
      skip_download: true
  - put: backend-version
    params:
      file: backend-version/number

Before:

After:

During this 6+ minutes, I did a "watch" on fly volumes for the container handle that is associated with that put task. I see a few container entries for fly containers for that handle... then after about 6 minutes, I do see a new volume entry show up in the watch fly volumes | grep the-container-handle ... so seems like its related to fs or volumes?

It would be great to get this fixed.. also if there is something I could look into... logs or anything to help track down, would be appreciated.

We got alot of gains with the new caches feature, but lost them with some CI pipelines w/ this noticeable lag :)

Cheers

[vito]

With the switch to overlay by default in 3.1, the tradeoff is that privileged tasks/resources (i.e. the Docker Image resource) have a performance penalty.

Unfortunately with container tech as it is today, you either get instability (btrfs) or slowness (overlay). We chose the latter.

Here are a few paths forward:

• Improve the UI feedback, so you at least know what it's doing and can know it's not stuck (so more than just a spinner). We've been generally in need of finer-grained progress indicators on the build page. (/cc @jma @Lindsayauchin)
• Look for optimizations in the runtime (/cc @topherbullock).
• Pray to the kernel gods that shiftfs gets merged and we can kill this nasty performance overhead.
• Add the ability to limit the inputs to a put, if it's the case that it's taking so long to transfer data that isn't actually needed by the task/put. See I should be able to control which artifacts a put receives #1202

[krishicks]

I'd prefer the instability over the slowness.

This slowness means we end up waiting 3-5 minutes regularly for tasks to start.

When you add to that the fact that volumes in 3.0.1+ often get reaped sooner than they should, and we have to download 7GB files again from the Internet, the time it takes jobs to run becomes absurd.

[vito]

Then you can configure the driver back to btrfs. Is there an issue for the volume reaping bug? This is the first I've heard of it.

…

On Tue, Aug 15, 2017, 11:14 AM krishicks ***@***.***> wrote: I'd prefer the instability over the slowness. This slowness means we end up waiting 3-5 minutes regularly for tasks to start. When you add to that the fact that volumes in 3.0.1+ often get reaped sooner than they should, and we have to download 7GB files again from the Internet, the time it takes jobs to run becomes absurd. — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#1404 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAAHWMhtJ6KnTYdaa07JPaa7U7Bqs8jcks5sYbXggaJpZM4OeaDJ> .

[krishicks]

I'll switch our driver back to btrfs for now.

I've added #1475. I was told previously to make a reproducible test case, but I haven't prioritized doing that.

[jeanbza]

Also switched ours to btrfs. Hope that shiftfs gets in!

[timrchavez]

Did the folk that switched back to btrfs notice a performance improvement? Have you noticed increased instability? cc: @krishicks, @jadekler

FWIW the slowness and the lack of feedback drive dev-folk here bonkers.

[krishicks]

We did see improvement when switching back. I had forgotten about this issue and realized it was affecting me on a new Concourse deploy. Thanks for reminding me to switch drivers!

…

On Thu, Dec 28, 2017 at 09:56 Timothy R. Chavez ***@***.***> wrote: Did the folk that switched back to btrfs notice a performance improvement? Have you noticed increased instability? cc: @krishicks <https://github.com/krishicks>, @jadekler <https://github.com/jadekler> FWIW the slowness *and* the lack of feedback drive dev-folk here bonkers. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#1404 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAE1_b0JHOcZuTwx5Q-jTahjHWPVg-lzks5tE9ZcgaJpZM4OeaDJ> .

[jeanbza]

@timrchavez We definitely saw performance improvements switching back. I'm a couple months removed from managing concourse now so I'm unfortunately unable to remember specifics :(

[vito]

For anyone not following along in #1966, we found and fixed the source of a lot of the btrfs instability that led us to switching to overlay in the first place. We can now more easily recommend that people switch back to it with the next release of Concourse (3.9), and we'll consider changing the default in the future. Until we either change the default or find a way to improve the overlay performance (not likely), I'll leave this issue open.

[timrchavez]

I can now confirm that we see a dramatic performance improvement by switching back to btrfs. No more long pauses before a dind task starts running. Looking forward to the v3.9.0 release to get those stability fixes.

[ezraroi]

So after upgrading to 3.10.0 i removed the CONCOURSE_BAGGAGECLAIM_DRIVER='btrfs' env from our workers and looks like the issue returned now... we see 10+ minutes spinning build steps

[xtremerui]

@ezraroi refer to https://github.com/concourse/baggageclaim/blob/68422aa9cea1e86cf925e4aacbf696bfd81a0545/baggageclaimcmd/driver_linux.go#L43

is it possible for some reason your worker falls into the kernelSupportsOverlay and picks up overlay driver again? By default if it detects your file system supports btrfs it will use that.

[adarshaj]

I still see this issue in 3.14.1 even after switching to btrfs driver.. the job just keeps waiting and sometimes has 'waiting for docker to come up..' for more than 15 minutes..

[krishicks]

I feel the "waiting for docker to come up" error is unrelated. I have, however, also seen that on 3.14.1. The job was waiting for 15 hours.

[adarshaj]

Ah okay, may be it was just coincidence in my case then? I ran with 3.14.1 with overlay a few times and hadn't seen, after switching to btrfs I saw it in every run, so probably did the mistake of inferring correlation as causation..

Any work around for it?

[0815fox]

I see that issue on my concourse setup, version 4.2.1. Is there any information that I should provide?

[sbkg0002]

We have the same issue with 4.2.1 with the binary release.

[xtremerui]

@0815fox @sbkg0002 what filesystem are you using?

In concourse CI it is also using overlay now and we don't have this issue. So your performance degradation might caused by something else but filesystem.

[sbkg0002]

We changed overlay into btrfs this morning and the speed is back again!
Binary release on Amazon Linux 2.

@0815fox change the fs driver.