Add ability to not use loopback dev for btrfs

[kcmannem]

@cirocosta and I noticed when concourse starts trashing memory. The loop0 device used by btrfs creates lots of load. We can get around this by attaching an ephemeral disk to our workers through the cloud-config. Then use that device as the btrfs volume. This will reduce alot of administrative overhead used by concourse.

In the flamegraph posted below, during high load situation (eg max containers reached) kswap+loop0+telegraph use about 60% of system resources. We need to reduce this.

[kcmannem]

@ddadlani @topherbullock

[cirocosta]

Hey, another detail before breaking this down to other possible issues - it's also interesting to see how that filepath.Walk that we perform might end up being quite expensive depending on the volumes that we have.

If I understood correctly, such "walking" goes through every file under that volume:

https://github.com/concourse/baggageclaim/blob/6fee2c8029decd1325b5e9d6d4069ea4a7c4578d/volume/driver/btrfs.go#L40-L59

Is that correct? If so, perhaps it'd be good to somehow remove that big walk as that means at least one syscall for each possible file in the volume 🤔 Wdyt?

Thanks!

[ddadlani]

To do this, we would likely need to change baggageclaim. The current behaviour is to create a loopback device on startup and to use that, we would need a way to bypass this behaviour and pass it the mount point of an external disk or something.

[vito]

@ddadlani It should only do that if it detects that the disk isn't already formatted as btrfs.

[cirocosta]

(the if that @vito mentioned lives here: https://github.com/concourse/baggageclaim/blob/6fee2c8029decd1325b5e9d6d4069ea4a7c4578d/baggageclaimcmd/driver_linux.go#L64-L78 - I was very recently looking at that portion 👀 )

[ddadlani]

Thanks for pointing that out, looks like it should be straightforward to test with an ephemeral disk for volumes 👍

[ddadlani]

I recall that we couldn't remove the filepath.Walk because of concourse/baggageclaim#21 (comment)

Do we still want to test btrfs with an ephemeral disk?

[kcmannem]

@ddadlani prob at some point but it’s fallen out of our priorities. I think we should revert the default to overlay as we’ve had good times with it.