Add support for volume mounting in docker backend

[detro]

When using Docker as provider, Travis Worker doesn't currently allow to control the options for the command:

docker run [OPTIONS] IMAGE [COMMAND] [ARG...]

Right now, only providing a CMD for [COMMAND] is allowed.

Allowing to define some default [OPTIONS] used by the worker when launching containers could be beneficial to, for example, provide a list of volumes to mount in the container.

I can try to provide a PR in the next days, but my knowledge of Go is close to 0.
In the meantime, what do you guys think?

Thanks
Ivan

[singpolyma]

This is exactly what I was just looking for.

[meatballhat]

Given that the docker backend operates over the docker API, as opposed to a shell invocation, I'm not sure how such an OPTIONS config will work in practice (see

worker/backend/docker.go

Lines 188 to 198 in f810296

	dockerConfig := &docker.Config{
	Cmd: p.runCmd,
	Image: imageID,
	Memory: int64(p.runMemory),
	Hostname: fmt.Sprintf("testing-docker-%s", uuid.NewRandom()),
	}
	dockerHostConfig := &docker.HostConfig{
	Privileged: p.runPrivileged,
	Memory: int64(p.runMemory),
	}

). Are there specific options that can be supported that will address your needs?

[singpolyma]

Volumes to mount would be the one I've run into.

[meatballhat]

I'm in favor of a volume mounting configuration option, yep 👍

[meatballhat]

@detro is volume mounting enough? What other capabilities do you need?

[detro]

IMHO allowing anything the Docker GO client API allow would be the best resolution to this.

But I understand that that might be too much to work on, given we could only present the "volume mounting" as an example.

I'd leave to the final implementer the ultimate call: at the end of the day I said I was going to provide a PR but I didn't. So I'll accept any outcome :)

SIDENOTE: How did I circumvent this in our Travis CI Enterprise? I have created a container off the Travis official base container (the java one) and added a VOLUME /volume_name to it. The I use that as our lang in the .travis.yml and, when launching containers from within a Travis Job container, I tell Docker to launch it by mounting all the volumes I have in the launching container. Did I explain myself badly? It's because it's very inceptiony.

[meatballhat]

@detro Allowing anything the client API allows sounds like a maintenance fun time to me 😁. I think your solution for getting a volume mount is reasonable, but I can also see the value in providing first-class support for volume mounting. If you're OK with it, I'd like to alter the title of this issue to be volume-specific. Alternatively, I can start a separate issue specifically for volume mounting feature discussion. What do you think?

[detro]

I totally understand.

Go with the renaming and volume only: I think it's already a great extra feature to have.

Maintenance is super important in modern software, and I can't counter argue that supporting "anything the API do" would be a nightmare ;)

Thanks a lot for your help!

Ivan De Marino
Sent from Mobile (sorry for typos)

On 17 Nov 2016, at 21:06, Dan Buch notifications@github.com wrote:

@detro Allowing anything the client API allows sounds like a maintenance fun time to me 😁. I think your solution for getting a volume mount is reasonable, but I can also see the value in providing first-class support for volume mounting. If you're OK with it, I'd like to alter the title of this issue to be volume-specific. Alternatively, I can start a separate issue specifically for volume mounting feature discussion. What do you think?

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub, or mute the thread.

[Kazade]

Just to clarify, until this issue is fixed, is it currently impossible to access the cloned git repo from inside a Docker container (given that -v and --volumes-from don't seem to do anything)?

[meatballhat]

@Kazade I think I need more detail about what you're trying to do in order to comment 😃

[majormoses]

I would love to see this as I was trying to work on a use case but got blocked by this.

The thought was to test multiple rubies with kitchen in a docker container but to mount the rubies from travis so we could still leverage cache.

[Kazade]

@meatballhat I have a Docker image that has a custom toolchain that I need to use to compile the code from my repo. Without volume mounting I have to re-clone the repo and checkout the branch into the container :(

[meatballhat]

I began working on adding support for arbitrary read-only and read-write volume mounts, only to find that the docker client library we've been using is ill-equipped for the job. I expect what we will have to do first is #351.

[detro]

That's precisely why I didn't open a PR: I'm not versed in Go but I thought I could wing it. But when I realised the library for Docker needed to be updated too, my impact could have been way too big. So I gave up. We remain hopeful for an official-maintainer fix :)

[majormoses]

Yay for progress: #362

[meatballhat]

Closed via #373 🎉

To be released 🔜

[Kazade]

Do we have an ETA of when this will be released? Kinda eagerly waiting for it 😉

[meatballhat]

@Kazade Sorry for the delay! I'm hoping to get a release out today 👍

[meatballhat]

@Kazade I take it back! 3.4.0 has the goods!

[meatballhat]

@Kazade More specifically, this support landed in 3.2.0.

[Kazade]

Is there an example of how this works somewhere? I've tried this:

env:
 global:
  - TRAVIS_WORKER_DOCKER_BINDS='/home:/home'

And loads of variations on that, but I can't seem to get anything to mount in the Docker container, it doesn't seem to have any affect. Am I using it wrong?

[mcraveiro]

I've got the same question as @Kazade, need to use this so an example would be fantastic. Thanks in advance.

[majormoses]

See the comment here: #373 (comment)

[ffledgling]

It looks like issue shouldn't be marked as closed since the fix seems to be only for enterprise customers of Travis? (per comment linked above). It'd be really nice to be able to mount the build workspace (if not arbitrary directories) into the container.

[meatballhat]

@ffledgling Mounting arbitrary directories is not something we intend to offer for hosted Travis CI given the security implications. Please see #373 (comment) ❤️

[meatballhat]

@ffledgling looking through the linked commit (ffledgling/pdns-go-tooling@58f1840) I wonder if we're talking about different things. On sudo-enabled Linux, running docker however you choose should work fine (with the exception of docker-in-docker). If you are seeing otherwise, please be sure to contact support@travis-ci.org.

[detro]

Hey folks, I have started using this feature (it's awesome). But it's not documented anywhere. I'd mend that ;)

Or, if the doc is a repository, happy to contribute.

[detro]

Found it: https://github.com/travis-ci/docs-travis-ci-com/blob/master/user/enterprise/worker-configuration.md.
PR incoming.

[detro]

PR ready travis-ci/docs-travis-ci-com#1762