Run docker as buildkite agent with userns-remap

[lox]

This uses the --userns-remap feature in recent docker daemons to remap the uid of root in docker containers to the buildkite-agent users on the host.

[toolmantim]

I don't know enough to really approve this. I'll let you merge as you need, or get someone else to put some eyeballs on it.

[lox]

I'm going to merge this after I put out a point release. I'm thinking of doing it as 2.3.0 @toolmantim, feels?

[toolmantim]

The release plan sounds good, testing aside!

[lox]

Possibly an rc1 so we can test if before it hits 2.3.0.

[vektah]

What happens to other user ids in the container? eg nobody writing doctrine cache files out to a volume mount? I think will land in following user ids so there will be stuff owned by 501+$uid in that directory, so you probably still need the permission fixing scripts. but I'm not sure how mapping that to a range of 1 works. maybe it just flat out breaks?

👍 for an RC, this is a whole lot of change.

The list of breaks from #32:

• Using --privileged mode flag on docker run
• sharing PID or NET namespaces with the host (--pid=host or --net=host)
• sharing a network namespace with an existing container (--net=container:other)
• sharing an IPC namespace with an existing container (--ipc=container:other)
• A --readonly container filesystem (this is a Linux kernel restriction against remounting with modified flags of a currently mounted filesystem when inside a user namespace)
• external (volume or graph) drivers which are unaware/incapable of using daemon user mappings

[lox]

Yeah, that is a pretty epic list huh.

[lox]

I might try and make this opt-in.

[toolmantim]

Maybe a v3.0 milestone feature?

[toolmantim]

What if we let this feature be turned on using an ENV var?

If we do add it as another option on the stack setup screen, might need to make it clear that it's not "buildkite-agent" the binary we're taking about here in the description?

If it's a "this could be buggy, so you better know what you're doing" then perhaps we just say:

Enables experimental Docker userns-remap support

[lox]

The issue is that it needs to have the system docker daemon started with the setting

[toolmantim]

Ah of course! 🙃

[lox]

If this is an optional setting, are you ok with a 2.3.0 release @toolmantim?