Proposal: Config file for default command line options

[bfirsh]

Docker needs a way of specifying options to "docker run" in a file.

You shouldn't have to write all this:

docker run -i -t --volumes-from=shykes-dev -v /var/run/docker.sock:/var/run/docker.sock -u 0 --name project -h project shykes/devbox bash

Instead, this could be in a file called .docker/config:

[run]
interactive=true
tty=true
volumes-from=shykes-dev
volume=/var/run/docker.sock:/var/run/docker.sock
user=0
name=project
host=project
image=shykes/devbox
cmd=bash

Then you can just run docker run in the same directory as that file.

This is a bit like how .git/config works, where you can, for example, specify options under a [log] section that overrides options on git log.

You could also specify what file to use with the --config option, like so:

docker --config=/etc/docker/config run

If you want to disable the config and specify options manually, there could be a --no-config option.

Environment variables could be specified by using the env-file option.

[shykes]

Probably that --config flag should come before run? Eg shouldn't it be this:

docker --config=/etc/docker/config

And NOT this?

docker run --config=/etc/docker/config

[bfirsh]

Yeah, it could do. The [run] section isn't to call the run command though, it's so you can specify options for other commands too. E.g. you could have a [build] section that specified build options.

Like .git/config there could also be a [core] section where you could specify stuff like the Docker host, etc.

[crosbymichael]

Few questions:

1. Is this all done client side? No sending the config to the server, it's parsed and the correct requests are built?
2. Image being a part of the config is a little weird to me. If i ship some software and I have a config and Dockerfile in the same repo how will that work? Maybe docker run --build to build and run the container in my cwd?
3. How are you going to handle arrays? -v -v -v ?

[bfirsh]

@crosbymichael

1. Yep, all client-side.
2. Agreed, it is a bit odd. I'd really like images and build paths to be the same thing... so you could do docker run . bash. But that's complicated, and should probably be continued in Build and run in one command #3763.
3. Some options I can think of:
   • Splitting by comma. E.g. `volumes=/foo:bar,/baz:/bar
   • Duplicate keys (but breaks the ini format)
   • A separate section, e.g., [run-volumes], where key/value pairs could be listed

[thaJeztah]

I like ideas that make it easier to work with docker and reduce the amount of typing required. Having said that, I don't think this proposal is really "there" (yet). I'll try to describe my thoughts and questions when reading this proposal;

• I'm not sure I want the config file to be hidden; it will be easy to overlook, which may lead to unexpected results if the user is unaware that a config-file is present ("why is it running detached? I didn't specify -d?). This may lead to a lot of hard-to-answer support questions; asking "What options did you use to run that container" will no longer be sufficient.
• The $project vars in the example; are those environment variables that will be expanded? What if the variable doesn't exist? How should the user be aware that such variable must be defined?
• How are arguments merged / combined if they are provided both via the command-line and inside the config file? For example, what if I use -v /var/run/docker.sock:/foobar/docker.sock; Will it override the volumes in the config? Append it?
• Allowing volumes-from to be specified will inevitably lead to the next question; will Docker also take care of starting those containers?
• If I use --use-config=/some/path/config, will paths inside the config be relative to the config-file, relative to the location that the command is ran from?

In general;

• I'd prefer the config to not be invoked automatically, maybe by calling docker run . (use config in current location)?
• Not sure if "config" is the best name for this. As I see it, the proposal describes a file-format for running a container - a file that offers to run what Dockerfile offers for build (should it be called DockerRun?)
• It feels a bit like re-inventing fig.yml, using .ini-format in stead of yaml. (.ini seems not so flexible when arrays should be specified), albeit without the 'orchestration'

Sorry if the comments are a bit disorganised - was writing them while overthinking the implications of this proposal. Hope they are useful anyhow 😄

[tianon]

I think if it's entirely client-side, that's just one more barrier for API-implementers, since then we'll see 14 different competing interpretations of parsing our "config" file spring up.

[shykes]

I don't think we should merge cli and config. If you specify 1 or more
command-line argument, the default from the file should be discarded
entirely.

On Friday, July 25, 2014, Sebastiaan van Stijn notifications@github.com
wrote:

I like ideas that make it easier to work with docker and reduce the amount
of tying required. Having said that, I don't think this proposal is really
"there". I'll try to describe my thoughts and questions when reading this
proposal;

• I'm not sure I want the config file to be hidden; it will be easy
  to overlook, which may lead to unexpected results if the user is unaware
  that a config-file is present ("why is it running detached? I didn't
  specify -d?). This may lead to a lot of hard-to-answer support
  questions; asking "What options did you use to run that container" will no
  longer be sufficient.
• The $project vars in the example; are those environment variables
  that will be expanded? What if the variable doesn't exist? How should the
  user be aware that such variable must be defined?
• How are arguments merged / combined if they are provided both via
  the command-line and inside the config file? For example, what if I use -v
  /var/run/docker.sock:/foobar/docker.sock; Will it override the
  volumes in the config? Append it?
• Allowing volumes-from to be specified will inevitably lead to the
  next question; will Docker also take care of starting those containers?
• If I use --use-config=/some/path/config, will paths inside the
  config be relative to the config-file, relative to the location that the
  command is ran from?

In general;

• I'd prefer the config to not be invoked automatically, maybe by
  calling docker run . (use config in current location)?
• Not sure if "config" is the best name for this. As I see it, the
  proposal describes a file-format for running a container - a file
  that offers to run what Dockerfile offers for build (should it be
  called DockerRun?)
• It feels a bit like re-inventing fig.yml, using .ini-format in stead
  of yaml. (.ini seems not so flexible when arrays should be specified),
  albeit without the 'orchestration'

Sorry if the comments are a bit disorganised - was writing them while
overthinking the implications of this proposal. Hope they are useful anyhow [image:
😄]

—
Reply to this email directly or view it on GitHub
#7232 (comment).

[thaJeztah]

Combining config with command-line arguments may be troublesome in other cases as well. For example, if I specify a conflicting argument (config says --detached and command-line argument is --interactive)

[thaJeztah]

@tianon would calling it Dockerrun and sending the file to the server (just like Dockerfile) be a better approach?

[tianon]

I think sending it to the server would be, yeah, but it's not my call; I'm just an interested party here. :)

[thaJeztah]

@tianon so am I. Not technically-savvy enough to provide the code for such things, but try to give (hopefully useful) input where possible 😄

[SvenDowideit]

I also would like a presets file that gives me defaults - similar to how boot2docker-cli uses the toml module to auto-preset the pflags

But for image build and run defaults, I would really like to (also?) embed them into the Dockerfile and Image - its just as hidden from use as a .docker/config, but its portable - a user gets the defaults both when they git clone app and when they docker pull app

What I was dreaming of originally, was to push the toml marshaling into a pflags wrapper, so the coder experience would be as it is today, and the defaulting would 'just happen'.

This also gets hard when options can modify the defaults other options.. (which is a good reason to discard the defaults entirely when the user specifies some.)

[thaJeztah]

..embed them into the Dockerfile and Image..

Thinking about this; what if someone specified --privileged as default. Is that something we want?

But you do make a good point; I was thinking of Dockerfiles and/or examples distributed via GitHub, but people downloading an image from Docker Hub will only get the Image, no option to get additional data (read.me, Dockerrun).

I like the idea of adding it to the Dockerfile, but it worries me that 'run-time' and 'build-time' settings are getting tangled up.

[tianon]

I think the way git handles this is a great example; when you "git clone",
the repo you are cloning can't provide new flags for the clone call, but
there are several config files read for changing the default behavior for
local things (machine global file, then user global file, then the repo
specific config which isn't managed inside the repo itself but is instead
managed in the local repo metadata).

[bfirsh]

@thaJeztah I agree with you that enabling options in a hidden config file is going to be surprising. I wonder why it isn't surprising with Git. Perhaps because it isn't often used and doesn't make large changes?

You're right about the $project vars. That was a mistake and I've removed them.

I think we should require you to specify the location of the config file, or at least require that you say you want to use it so there isn't surprising behaviour.

[SvenDowideit]

@bfirsh and @vieux I was working on some pain points in the complicated github.com/ogier/pflag and github.com/BurntSushi/toml flags with presets code that we have in boot2docker, and thought I'd see what happens if I replace it with docker's mflag, plus adding a really simple flags.PrintProfile() and flags.ReadProfile().

SvenDowideit@39ed2b8 is the result. (plus I added IP and IPMask as they were needed in b2d-cli)

If you like the direction I've taken it, I'll continue by adding tests for it. In Docker, we can then add some code to read the specified file, chop it up into the current command's section using the [run] section labels, and its almost done :)

in Boot2Docker we do currently mix presets and cmdline, but we also don't have arrays of values... - but this can easily be changed.

[vieux]

@SvenDowideit SGTM

[bfirsh]

Docker Machine has a concept of an "active" machine, where we want the Docker client to be automatically configured to talk to that machine – i.e. having the -H and --auth flags automatically set.

We could use environment variables, but there's no neat way to do this without a manual step. These are the current instructions after creating a machine:

INFO[0078] "test" has been created and is now the active machine. To point Docker at this machine, run: export DOCKER_HOST=$(machine url) DOCKER_AUTH=identity

What I would quite like to do is to use a configuration file to set these options. I think the first version could just be the top-level Docker options in an ini-format file. For example in ~/.docker/config:

[general]
host=tcp://10.1.2.3:2376
auth=identity
log-level=debug

The naming of the top-level key could do with some thought, but we can rename in the future without much pain. Ideas: [engine], [client]?

One concern is automatically generating this file from Machine, which may overwrite a user-written file. I wonder if we need some kind of config.d system...

@shykes @crosbymichael @jfrazelle @vieux @thaJeztah @SvenDowideit @tianon @ehazlett thoughts?

[ehazlett]

What if instead of overwriting from Machine we just parse and adjust the needed options? I think if they are using Machine they would expect it to be re-configured so we wouldn't have to worry about that specific value.

[thaJeztah]

Ideally I would expect Docker Machine to be able to manage multiple machines and be able to switch to a different machine just by usingDocker Machine, e.g. machine active production (or machine switch production - I don't like "active" as an instruction, but that's preference). This also works around the current assumption that bash is used as shell and would allow machine to run on Windows in future.

I don't think overwriting settings is necessary if the ini-file uses inheritance (example);

[global]
host = unix:///var/run/docker.sock
auth = identity

[staging : global]
host = tcp://123.456.789

each machine can have its own config and inherit settings from the global configuration.

[bfirsh]

@thaJeztah Machine already has a concept of an active machine and can switch between them. I don't think we don't want Docker to be aware of Machine. Instead we want Machine to be able to instruct Docker which host it should be talking to.

Two reasons:

1. Loose coupling. Docker doesn't need to be aware of Machine existing.
2. Machines don't have fixed IP addresses, so Machine needs to execute a bit of code to figure out where they are.

[thaJeztah]

@bfirsh Yes, you're absolutely right; the actual "switching" should be handled by docker, machine should only instruct the docker-client which machine/host to use.

wrt switching hosts (thinking out loud here); I can see this possibly conflicting with using swarm, i.e. if machine create automatically switches to the newly created machine, it will reset the host/ip for swarm (when using environment variables for that; the current examples for swarm explicitly use -H <swarm_ip:swarm_port>)

[bfirsh]

@thaJeztah Yes, that could be a problem. I think environment variables and the -H flag should override whatever is in a config file.

[thaJeztah]

There's also the case of hosts that are not maintained/created by Swarm or Machine. If Docker only keeps track of the current host, then switching to another host will overwrite that information. Which brings us back to #7232 (comment)

A possible solution would be to have Docker keep a list of host-configurations and allow switching between them. Machine, Swarm and other software would be able to add/update/remove a configuration in that list.

But that overlaps a lot with the functionality that Machine offers (apart from not offering the actual creation and provisioning of new hosts).

[jessfraz]

so is this compose?

[thaJeztah]

I think @bfirsh hijacked his own issue, describing a different use case starting from this #7232 (comment)

I suggest to;

• close this issue; the original use case is covered by compose; adding similar features to Docker itself would only confuse users
• create a new proposal based on the use case described in Proposal: Config file for default command line options #7232 (comment) and below.

[duglin]

where are we on this?

We have a config file that can be used to store additional (common) config properties (like DockerHost).

I'm not sure we should use a "config file" for properties that specific to just one command though. The example specified in the original comment ( #7232 (comment) ) is odd to me - that feels more like using a text file to represent the cmd line flags - I would call that more like a compose yml than a config file.

[duglin]

@bfirsh can we close this?
We now have a ~/.docker/config.json file which can (and is) used to persist some cli options - like ps's filter. Whether we use it for any particular option can probably be discussed in individual PRs.

[vdemeester]

I do agree with @duglin, I think we should close this one 😉

[thaJeztah]

Ok I agree and I'll be the one that did it /o\