New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Cannot load hosts in configuration json file when starting docker daemon #22339

Closed
allencloud opened this Issue Apr 26, 2016 · 28 comments

Comments

Projects
None yet
@allencloud
Copy link
Contributor

allencloud commented Apr 26, 2016

Output of docker version:

root@10-11-1-156:~# docker version
Client:
 Version:      1.11.0
 API version:  1.23
 Go version:   go1.5.4
 Git commit:   4dc5990
 Built:        Wed Apr 13 18:34:23 2016
 OS/Arch:      linux/amd64

Server:
 Version:      1.11.0
 API version:  1.23
 Go version:   go1.5.4
 Git commit:   4dc5990
 Built:        Wed Apr 13 18:34:23 2016
 OS/Arch:      linux/amd64

Output of docker info:

root@10-11-1-156:~# docker info
Containers: 7
 Running: 0
 Paused: 0
 Stopped: 7
Images: 207
Server Version: 1.11.0
Storage Driver: aufs
 Root Dir: /var/lib/docker/aufs
 Backing Filesystem: extfs
 Dirs: 234
 Dirperm1 Supported: false
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
 Volume: local
 Network: bridge null host
Kernel Version: 3.13.0-83-generic
Operating System: Ubuntu 14.04.3 LTS
OSType: linux
Architecture: x86_64
CPUs: 1
Total Memory: 1.955 GiB
Name: 10-11-1-156
ID: NDA5:5CVE:Y7Y4:MQJZ:IQVN:ISZ4:5V2Q:JUMU:MGPR:BWGF:LKSR:OW5E
Docker Root Dir: /var/lib/docker
Debug mode (client): false
Debug mode (server): false
Registry: https://index.docker.io/v1/
WARNING: No swap limit support

Additional environment details (AWS, VirtualBox, physical, etc.):

two config file, one as flag, one for config loading

root@10-11-1-156:~# cat /etc/default/docker
DOCKER_OPTS="-H unix:///var/run/docker.sock"
root@10-11-1-156:~# cat /etc/docker/daemon.json
{"hosts":["tcp://127.0.0.1:2376"]}

when execute service docker restart:

root@10-11-1-156:~# service docker restart
docker stop/waiting
start: Job failed to start

check docker daemon's log:

Waiting for /var/run/docker.sock
unable to configure the Docker daemon with file /etc/docker/daemon.json: the following directives are specified both as a flag and in the configuration file: hosts: (from flag: [unix:///var/run/docker.sock], from file: [tcp://127.0.0.1:2376])

And I was following (https://docs.docker.com/engine/reference/commandline/daemon/#daemon-configuration-file) to add hosts in /etc/docker/daemon.json. While fails to start docker daemon.

@thaJeztah

This comment has been minimized.

Copy link
Member

thaJeztah commented Apr 26, 2016

This is expected; you cannot specify options both as a flag and in the configuration file (daemon.json). If you change your DOCKER_OPTS to DOCKER_OPTS="" and restart, then it should work. We explicitly don't "merge" these configurations.

I'll close this issue, as I don't think there's a bug here, but feel free to comment 👍

@thaJeztah thaJeztah closed this Apr 26, 2016

@ghost

This comment has been minimized.

Copy link

ghost commented Jun 12, 2016

This is a problem if you are using the default systemd unit file, which is hardcoded to start dockerd with -H fd://.

quick workaround (Debian 8)): create /etc/systemd/system/docker.service.d/docker.conf

[Service]
ExecStart=
ExecStart=/usr/bin/dockerd

then systemctl daemon-reload

pyotr777 added a commit to pyotr777/kportal that referenced this issue Jul 13, 2016

Remove /etc/docker/daemon.json or docker won't start.
moby/moby#22339
Permissions for log directory.
CUR_DIR in start_server.sh is not used.
@zoechi

This comment has been minimized.

Copy link
Contributor

zoechi commented Jul 19, 2016

@magentic thanks that helped

for me it was

ExecStart=/usr/bin/docker daemon

instead of

ExecStart=/usr/bin/dockerd
@asbjornenge

This comment has been minimized.

Copy link
Contributor

asbjornenge commented Oct 6, 2016

This just hit me also.

Considering the default config breaks with a daemon.json containing hosts - where hosts is a supported config option (and a very common one I would think?) should we not consider this a bug still!?

//cc @thaJeztah

@samrocketman

This comment has been minimized.

Copy link

samrocketman commented Oct 18, 2016

This is definitely a bug in the systemd configuration. I have DOCKER_OPTS="" but systemd on Ubuntu 16.04 starts the docker daemon with -H fd://. I want to configure docker with TLS over the network.

Please re-open this critical bug I consider this a blocking and breaking bug.

$ grep 'ExecStart' /lib/systemd/system/docker.service
ExecStart=/usr/bin/dockerd -H fd:// $DOCKER_OPTS

Considering I'm trying to use daemon.json because /etc/default/docker states:

# Here in Debian, this file is sourced by:
#   - /etc/init.d/docker (sysvinit)
#   - /etc/init/docker (upstart)
#   - systemd's docker.service

# Use of this file for configuring your Docker daemon is discouraged.

# The recommended alternative is "/etc/docker/daemon.json", as described in:
#   https://docs.docker.com/v1.11/engine/reference/commandline/daemon/#daemon-configuration-file

I recommend either fixing this bug or removing the message Use of this file for configuring your Docker daemon is discouraged. from /etc/default/docker to avoid confusion.

# apt-cache show docker.io
Package: docker.io
Priority: optional
Section: universe/admin
Installed-Size: 54232
Maintainer: Ubuntu Developers <ubuntu-devel-discuss@lists.ubuntu.com>
Original-Maintainer: Paul Tagliamonte <paultag@debian.org>
Architecture: amd64
Version: 1.12.1-0ubuntu13~16.04.1
Replaces: docker (<< 1.5~)
Depends: adduser, containerd (>= 0.2.3~), iptables, runc (>= 1.0.0~rc1~), init-system-helpers (>= 1.18~), lsb-base (>= 4.1+Debian11ubuntu7), libapparmor1 (>= 2.6~devel), libc6 (>= 2.14), libdevmapper1.02.1 (>= 2:1.02.97)
Recommends: ca-certificates, cgroupfs-mount | cgroup-lite, git, ubuntu-fan, xz-utils, apparmor
Suggests: aufs-tools, btrfs-tools, debootstrap, docker-doc, rinse, zfs-fuse | zfsutils
Breaks: docker (<< 1.5~)
Filename: pool/universe/d/docker.io/docker.io_1.12.1-0ubuntu13~16.04.1_amd64.deb
Size: 10581186
MD5sum: 08676d065c43fc6bcb450cf52c782ed5
SHA1: 8f51b8c7b0fbc3eb879e13a76286cbe13b096d34
SHA256: 0987a665d090d90b0096fbdcab7d18316cad6a1754dbc70f56ab74c36ba53c51
...
@thaJeztah

This comment has been minimized.

Copy link
Member

thaJeztah commented Oct 18, 2016

Sorry, I missed @asbjornenge's ping #22339 (comment)

@samrocketman there's no bug in the systemd configuration, to override defaults in a systemd unit file, you can use a drop-in file, as described here https://docs.docker.com/engine/admin/systemd/#/custom-docker-daemon-options

Producing an error if both a flag and an option in daemon.json are provided was a design decision when implementing that (in general, flags should always have precedense over configuration files); automatically merging options was not an option, as this would lead to unexpected results (was the intent to override an option, or to add to an option?)

I can see that the combination of the default unit-file, and using daemon.json to try to override this particular setting is making it more difficult to use. I'm not sure what the best option is (given the above explanation); can you open a new issue to propose what you think is the correct change?

I'd rather not re-open this issue, as the original issue/question is resolved, so please open a new issue.

@samrocketman

This comment has been minimized.

Copy link

samrocketman commented Oct 18, 2016

@thaJeztah I went ahead and proposed a patch to the exact script which is causing this for me. See linked PR #27473.

@samrocketman

This comment has been minimized.

Copy link

samrocketman commented Oct 18, 2016

@thaJeztah I'll open a new issue. Now that I read the description more closely it's somewhat similar but not the same as this issue.

EDIT: I'll just let this PR serve as my issue unless I'm required otherwise.

@sandersaares

This comment has been minimized.

Copy link

sandersaares commented Dec 8, 2016

This issue impacts me. I would enjoy a resolution that does not require expenditure of energy on finding all the different places where things are configured.

pyotr777 added a commit to pyotr777/kportal that referenced this issue Feb 16, 2017

Remove /etc/docker/daemon.json or docker won't start.
moby/moby#22339
Permissions for log directory.
CUR_DIR in start_server.sh is not used.
@Justluckyg

This comment has been minimized.

Copy link

Justluckyg commented Jul 24, 2017

Is this the same issue as this?

[root@DB-arb files]# systemctl status docker.service
● docker.service - Docker Application Container Engine
Loaded: loaded (/usr/lib/systemd/system/docker.service; enabled; vendor preset: disabled)
Drop-In: /etc/systemd/system/docker.service.d
└─docker.conf
Active: failed (Result: exit-code) since Mon 2017-07-24 23:19:26 CXT; 15s ago
Docs: https://docs.docker.com
Process: 4774 ExecStart=/usr/bin/dockerd --bip=172.30.0.0/16 (code=exited, status=1/FAILURE)
Main PID: 4774 (code=exited, status=1/FAILURE)

Jul 24 23:19:26 DB-arb dockerd[4774]: time="2017-07-24T23:19:26.628798234+07:00" level=info msg="[graphdriver] using prior storage driver: devicemapper"
Jul 24 23:19:26 DB-arb dockerd[4774]: time="2017-07-24T23:19:26.641562669+07:00" level=info msg="Graph migration to content-addressability took 0.00 seconds"
Jul 24 23:19:26 DB-arb dockerd[4774]: time="2017-07-24T23:19:26.642103632+07:00" level=warning msg="mountpoint for pids not found"
Jul 24 23:19:26 DB-arb dockerd[4774]: time="2017-07-24T23:19:26.642573572+07:00" level=info msg="Loading containers: start."
Jul 24 23:19:26 DB-arb dockerd[4774]: time="2017-07-24T23:19:26.667569811+07:00" level=info msg="Firewalld running: true"
Jul 24 23:19:26 DB-arb dockerd[4774]: Error starting daemon: Error initializing network controller: Error creating default "bridge" network: fail...dy in use
Jul 24 23:19:26 DB-arb systemd[1]: docker.service: main process exited, code=exited, status=1/FAILURE
Jul 24 23:19:26 DB-arb systemd[1]: Failed to start Docker Application Container Engine.
Jul 24 23:19:26 DB-arb systemd[1]: Unit docker.service entered failed state.
Jul 24 23:19:26 DB-arb systemd[1]: docker.service failed.
Hint: Some lines were ellipsized, use -l to show in full.
`

@zhaoyao91

This comment has been minimized.

Copy link

zhaoyao91 commented Nov 1, 2017

Thanks @asbjornenge, real time saver.
This really should be considered as a bug.
According to the doc:

Note: You cannot set options in daemon.json that have already been set on daemon startup as a flag. On systems that use systemd to start the Docker daemon, -H is already set, so you cannot use the hosts key in daemon.json to add listening addresses. See https://docs.docker.com/engine/admin/systemd/#custom-docker-daemon-options for how to accomplish this task with a systemd drop-in file.

But when I turned to the reference, there isn't solution for this problem.

@binman-docker

This comment has been minimized.

Copy link

binman-docker commented Nov 13, 2017

Looks like it was documented here: https://docs.docker.com/engine/admin/#troubleshoot-conflicts-between-the-daemonjson-and-startup-scripts

I'd still like to see a modification of the packaging or Docker itself so that people don't have to work around this. Anyone using TLS over API but also the socket will need to specify their hosts, and people upgrading from another version where they used a host config in daemon.json will have Docker break until they end up finding this issue or the new doc via their favorite search engine.

Maybe docker engine should prefer settings in daemon.json over those supplied by the flag, or vice versa.

@kekkokk

This comment has been minimized.

Copy link

kekkokk commented Nov 17, 2017

we use daemon.json to avoid to discriminate what file to change based on the system (upstart/systemV, etc.)
But if we have to manually empty the DOCKER_OPTS, it has no sense.
so dameon should have the priority

@lifeofguenter

This comment has been minimized.

Copy link

lifeofguenter commented Feb 25, 2018

why is this issue closed?

@tdaniely

This comment has been minimized.

Copy link

tdaniely commented Feb 27, 2018

This is so stupid.

@ZQQ1024

This comment has been minimized.

Copy link

ZQQ1024 commented Feb 28, 2018

This hits me also.I am using docker remote api and tlsverify on Ubuntu 16.04 LTS, and have to chang the hosts.I can not change hosts in /etc/docker/daemon.json, and I solved it by overriding docker.service.

@samrocketman

This comment has been minimized.

Copy link

samrocketman commented Feb 28, 2018

Same, I still override it. @thaJeztah is it worth reconsidering this a bug? It still affects me but I gave up trying to change your mind until people started reporting it again.

@bk201sama

This comment has been minimized.

Copy link

bk201sama commented Jul 10, 2018

So the question still exists. ghost's resolution worked.

@jindov

This comment has been minimized.

Copy link

jindov commented Jul 12, 2018

If you're using AWS ec2, you can put some stuff on userdata:

echo '{"registry-mirrors": [ "http://10.16.1.163:5000" ], "max-concurrent-downloads": 5, "hosts" : ["tcp://0.0.0.0:2375", "unix:///var/run/docker.sock"]}' > /etc/docker/daemon.json
sed -i 's/dockerd\ \-H\ fd\:\/\//dockerd/g' /lib/systemd/system/docker.service
systemctl daemon-reload
systemctl restart docker
@wiese

This comment has been minimized.

Copy link

wiese commented Jul 18, 2018

In the light of this mentioning #14491 (override.conf) seems worthwhile - more explicit than seding a 3rd party file for sure.

@jacksindhu

This comment has been minimized.

Copy link

jacksindhu commented Aug 1, 2018

error during connect: Get https://172.31.20.213:2376/v1.27/version: x509: certificate signed by unknown authority
[root@ip-172-31-20-213 certs]# dockerd --tlsverify --tlscacert=ca.pem --tlscert=server-cert.pem --tlskey=server-key.pem -H=0.0.0.0:2376 unable to configure the Docker daemon with file /etc/docker/daemon.json: the following directives are specified both as a flag and in the configuration file: hosts: (from flag: [0.0.0.0:2376], from file: [tcp://0.0.0.0:2376 unix:///var/run/docker.sock]), tlscacert: (from flag: ca.pem, from file: /etc/docker/certs/ca.pem), tlscert: (from flag: server-cert.pem, from file: /etc/docker/certs/server-cert.pem), tlskey: (from flag: server-key.pem, from file: /etc/docker/certs/server-key.pem), tlsverify: (from flag: true, from file: true)

@cgyim1992

This comment has been minimized.

Copy link

cgyim1992 commented Aug 17, 2018

this hit me, I solved by change config in

/lib/systemd/system/docker.service

as

#ExecStart=/usr/bin/dockerd -H fd://
ExecStart=/usr/bin/dockerd

by erase the '-H fd://', the /etc/docker/daemon.json would take effect, but you should also be sure that your DOCKER_OPTS="" in your /etc/default/docker
NOTE: you should do

systemctl daemon-reload

to refresh the system service daemon config file ,and

service docker restart 

should work fine .

@duckie

This comment has been minimized.

Copy link

duckie commented Nov 2, 2018

[Service] ExecStart= ExecStart=/usr/bin/dockerd

The ExecStart= with no value is important here and I dont get why

@thaJeztah

This comment has been minimized.

Copy link
Member

thaJeztah commented Nov 2, 2018

The ExecStart= with no value is important here and I dont get why

When you use a systemd drop-in ("override") file, you are specifying options to override, or add to the main systemd unit file.

However, a systemd unit file is allowed to have multiple ExecStart instructions (see the systemd man page. If you specify ExecStart in your "override" file, it is added to the existing list of ExecStart instructions in the main systemd unit file (and those ExecStart commands will be joined together).

To override / replace the ExecStart in the main systemd unit file, it is therefore needed to first reset the ExecStart, and then define the new ExecStart to replace it. So;

Example content of your override file (e.g./etc/systemd/system/docker.service.d/my-override.conf);

# This line resets / "removes" the original ExecStart as was defined in the main systemd unit file
ExecStart=

# This line defines the new ExecStart to use _instead_
ExecStart=/usr/bin/dockerd

Never edit the original systemd unit file, because doing so will mark it as "edited by user", which means that if you upgrade Docker to a newer version (install a newer version of the rpm / deb package), and that version has an updated systemd unit-file, the file won't be updated automatically with the new version.

@chrisinmtown

This comment has been minimized.

Copy link

chrisinmtown commented Nov 2, 2018

I hope this isn't a dumb question, would someone please comment how the command-line options are set when dockerd is launched by snap, which seems to be the case in Ubuntu 18.04? The system is configured as follows after installing docker version 18.06.1-ce. I see an ExecStart line but it looks nothing like what is posted in this thread.

# cat /etc/systemd/system/snap.docker.dockerd.service 
[Unit]
# Auto-generated, DO NOT EDIT
Description=Service for snap application docker.dockerd
Requires=snap-docker-321.mount
Wants=network-online.target
After=snap-docker-321.mount network-online.target
X-Snappy=yes

[Service]
ExecStart=/usr/bin/snap run docker.dockerd
SyslogIdentifier=docker.dockerd
Restart=on-failure
WorkingDirectory=/var/snap/docker/321
TimeoutStopSec=30
Type=simple

[Install]
WantedBy=multi-user.target

Thanks in advance.

@thaJeztah

This comment has been minimized.

Copy link
Member

thaJeztah commented Nov 2, 2018

@chrisinmtown the snap package is likely maintained by Ubuntu / canonical, and is not the official Docker package maintained by Docker. Probably best to contact the canonical maintainers (or install the offficial Docker package https://docs.docker.com/install/linux/docker-ce/ubuntu/ )

@duckie

This comment has been minimized.

Copy link

duckie commented Nov 28, 2018

@thaJeztah thanks for the explanation.

However, -H is definitely used by the systemd unit file provided in the official docker-ce package, at least for the RHEL family. Therefore, it should be considered a packaging bug. -H should not be used in the unit file so that it stays as generic as possible, and the default -H value should be provided in the default daemon.json file instead.

Unless there is a rationale to force the host value ? I cant see any valid reason.

@dsanders11

This comment has been minimized.

Copy link

dsanders11 commented Dec 15, 2018

@thaJeztah, any response to @duckie? The use of -H is clearly visible in docker-ce's systemd file, and was introduced in the last version where it wasn't used before. This broke several working configurations.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment