docker-compose slow on docker for mac os beta

[Erwyn]

docker-compose is slow with docker for mac os beta on my home network. Here is my workaround for now:

• docker-compose up (take ages)
• shut down wifi
• docker-compose up (really fast)
• re-enable wifi

I do not reproduce the issue on another network than mine, my work network for instance do not make it slow. I already had a potentially related issue with the docker client itself which couldn't pull any image (going to bizarre local ips instead of the docker hub registry) but it has been fixed since one of the latest docker for mac os beta update.

The issue is not reproduced against the docker-toolbox, only the "native" docker for mac.

My version of docker-compose is : docker-compose version 1.7.0, build 0d7bf73
My version of docker for mac is: Version 1.11.1-beta10 (build: 6662)

The docker-compose file I'm trying to run is:

#docker-compose.yml
sync-engine:
  build: nylas-sync-engine
  ports:
    - 5555:5555
  links:
    - mysql:mysql
    - redis:redis
  hostname: sync-engine
  log_opt:
    max-size: "10m"
    max-file: "10"

mysql:
  image: mysql
  environment:
    - MYSQL_ROOT_PASSWORD=whateverpassword
  volumes:
    - nylas_mysql:/var/lib/mysql
  log_opt:
    max-size: "10m"
    max-file: "10"

redis:
  image: redis
  volumes:
    - nylas_redis:/data
  log_opt:
    max-size: "10m"
    max-file: "10"

[thaJeztah]

ping @FrenchBen 😄

[smyth64]

+1

[smyth64]

got the same issue :D

[thaJeztah]

@smith64fx problem also goes away if you turn off your WiFi?

[smyth64]

@stijn Yes, when i turn off wifi everything works like charm

Von meinem iPhone gesendet

Am 05.05.2016 um 00:26 schrieb Sebastiaan van Stijn notifications@github.com:

@smith64fx problem also goes away if you turn off your WiFi?

—
You are receiving this because you were mentioned.
Reply to this email directly or view it on GitHub

[Erwyn]

@smith64fx I see from your signature that you are likely from Germany (or Austria/switzerland). Do you mind me asking what is your Internet provider? I'm wondering if we have the same, and if it could be coming from the box which does not look like a very good piece of hardware/software and would not act as thought by docker.

(I am with Vodafone and I have their easybox-whatever)

[holstvoogd]

Same issue on a wired network (corporate network), as soon as I un plug it everything is very fast. So I'm positive it is not related to you specific provider.

I've looked at the verbose output, there is no obvious error (nor anywhere in the system logs). Did notice this though:

With no networking, these lines are grouped together:

docker attach <- (u'd90162cbfd7b312c28488a209440641b2d9c9f99e8eb59082f20ddf7d84e7b7e', stderr=True, stream=True, stdout=True)
docker attach -> <generator object _multiplexed_response_stream_helper at 0x1045f20a0>
docker start <- (u'd90162cbfd7b312c28488a209440641b2d9c9f99e8eb59082f20ddf7d84e7b7e')

Without networking, I get ~100-200 lines of 'compose.parallel.feed_queue: Pending: set([])' between the attach <- and where it returns with attach -> ...

Before this point there is a lot more happening with network as well, but mostly just more calls to inspect image etc it seems.

I've attached the output of compose --verbose up for bot situations. The compose file is just two containers straight from the hub.

with-networking.txt
without-networking.txt

[Erwyn]

@holstvoogd oh, ok for the provider. Thanks for the info, I was a bit worried :)

[FrenchBen]

@Erwyn @smith64fx To confirm, you're always connected (hard-wired) and at the same time use the same network with wifi?

[smyth64]

@FrenchBen No it's only in the wifi network in my home. At my office it's super fast. But the thing is, everything else at home runs fast, except docker-compose ^_^"

[Erwyn]

@FrenchBen same as @smith64fx. It's only wifi at home so there is no wired connection involved. And as I can see @holstvoogd seems to have the same issue with a wired connection.

[eugenesia]

I am noticing the same issue with Docker for Mac Beta. docker-compose up is slow when Wifi is enabled, fast when Wifi is disabled.

• Docker Compose version: docker-compose version 1.7.1, build 0a9ab35
• Docker version: Docker version 1.11.1, build 5604cbe
• OS version: OS X El Capitan 10.11.4

[krydos]

Hi, Is there any news about this?

I have same issue. Compose/Docker/OSX versions are same as @eugenesia has.
I use WiFi at home and at office and at my home it works incredibly slow. At office it works as expected (fast).

I thought maybe it is something related to DNS server of my ISP (home and office internet providers are different) and I tried to use some public DNS including Google's one but it didn't help.

If I turn WiFi off then docker-compose works really fast

[FrenchBen]

@krydos A new release should be coming out this week with some speed improvements

[runand]

I have the same issue, after updating to docker for mac 1.11.1-beta13, the problem persists. The workaround by switching the network(s) off and on again does not work anymore, the services start fast when switching the network off, but when switching the network on again, the services is no longer accessible and the docker daemon is not responding

docker ps
Error response from daemon: Bad response from Docker engine

• Docker Compose version: docker-compose version 1.7.1, build 0a9ab35
• Docker version: Docker version 1.11.1-beta13, build 7975
• OS version: OS X El Capitan 10.11.5

[jewilmeer]

I've had the same issues, and found a post (sorry, lost the ref) mentioning docker-compose is trying to resolve localunixsocket.local. You can get insight into the dns lookup by running sudo tcpdump -A -s0 -nni en0 port 53

For now I've pointed localunixsocket.local to localhost in my /etc/hosts. Now everything is speedy again.

127.0.0.1 localunixsocket.local

[thaJeztah]

Thanks @jewilmeer, that looks useful

[smyth64]

I switched back using virtualbox with docker-machine. Problem doesn't exist and it's up to 10x faster than Docker Mac Beta!

[thaJeztah]

@smith64fx please keep your comments constructive; it's a beta, it's not a finished product yet, so bugs and performance issues are expected. It's exactly these kind of issues that need reporting (and testing) to resolve them.

[seeekr]

super awesome comment, @jewilmeer! For me I had to add a few more addresses to /etc/hosts which I discovered using your tcpdump command:

# (yours)
127.0.0.1 localunixsocket.local
# additional ones -- be sure to replace bracketed thing with the output of `hostname`
127.0.0.1 localunixsocket.home <my hostname>.home

After those additions -- speedy! Actually, amazingly speedy, seems a good bunch faster than when using Docker Toolbox! woop woop :) (Though that may be a highly subjective observation!)

[justincormack]

That is pretty odd, but seems to point at what the underlying issue is...

[ScottKelly]

I'm currently facing this same issue and I can't resolve the problem.
I've tried the previous suggestions for editing /etc/hosts. On our office network, docker is extremely slow. On home networks or using tethering to a cell phone removes all slow downs and everything is snappy.

We're using docker-compose with a web container linked to four service containers (postgres, redis, rabbitmq, elasticsearch). Connecting to any of the four service containers directly from OSX is fine. It's only slow when trying to connect from the web container to any of the service containers.

Running tcpdump -vvv -s 0 -l -n port 53 gives the following output when tethered to a cell phone

tcpdump: data link type PKTAP
tcpdump: listening on pktap, link-type PKTAP (Packet Tap), capture size 262144 bytes
16:54:34.236026 IP (tos 0x0, ttl 64, id 25341, offset 0, flags [none], proto UDP (17), length 69)
    172.20.10.4.56662 > 172.20.10.1.53: [udp sum ok] 5785+ PTR? 1.0.19.172.in-addr.arpa. (41)
16:54:34.239617 IP (tos 0x0, ttl 64, id 29137, offset 0, flags [none], proto UDP (17), length 123)
    172.20.10.1.53 > 172.20.10.4.56662: [udp sum ok] 5785 NXDomain* q: PTR? 1.0.19.172.in-addr.arpa. 0/1/0 ns: 19.172.IN-ADDR.ARPA. [1d] SOA 19.172.IN-ADDR.ARPA. . 0 28800 7200 604800 86400 (95)
16:54:34.303325 IP (tos 0x0, ttl 64, id 46188, offset 0, flags [none], proto UDP (17), length 69)
    172.20.10.4.58590 > 172.20.10.1.53: [udp sum ok] 52066+ PTR? 1.0.19.172.in-addr.arpa. (41)
16:54:34.309241 IP (tos 0x0, ttl 64, id 7329, offset 0, flags [none], proto UDP (17), length 123)
    172.20.10.1.53 > 172.20.10.4.58590: [udp sum ok] 52066 NXDomain* q: PTR? 1.0.19.172.in-addr.arpa. 0/1/0 ns: 19.172.IN-ADDR.ARPA. [1d] SOA 19.172.IN-ADDR.ARPA. . 0 28800 7200 604800 86400 (95)
16:54:39.309028 IP (tos 0x0, ttl 64, id 52570, offset 0, flags [none], proto UDP (17), length 69)
    172.20.10.4.63544 > 172.20.10.1.53: [udp sum ok] 12851+ PTR? 1.0.19.172.in-addr.arpa. (41)
16:54:39.320135 IP (tos 0x0, ttl 64, id 32359, offset 0, flags [none], proto UDP (17), length 123)
    172.20.10.1.53 > 172.20.10.4.63544: [udp sum ok] 12851 NXDomain* q: PTR? 1.0.19.172.in-addr.arpa. 0/1/0 ns: 19.172.IN-ADDR.ARPA. [1d] SOA 19.172.IN-ADDR.ARPA. . 0 28800 7200 604800 86400 (95)
16:54:39.392915 IP (tos 0x0, ttl 64, id 8082, offset 0, flags [none], proto UDP (17), length 69)
    172.20.10.4.59994 > 172.20.10.1.53: [udp sum ok] 22334+ PTR? 1.0.19.172.in-addr.arpa. (41)
16:54:39.416821 IP (tos 0x0, ttl 64, id 51863, offset 0, flags [none], proto UDP (17), length 123)
    172.20.10.1.53 > 172.20.10.4.59994: [udp sum ok] 22334 NXDomain* q: PTR? 1.0.19.172.in-addr.arpa. 0/1/0 ns: 19.172.IN-ADDR.ARPA. [1d] SOA 19.172.IN-ADDR.ARPA. . 0 28800 7200 604800 86400 (95)

and this is the output when on our office wifi:

16:54:13.870062 IP (tos 0x0, ttl 64, id 17811, offset 0, flags [none], proto UDP (17), length 69)
    192.168.0.32.56316 > 192.168.0.1.53: [udp sum ok] 21469+ PTR? 1.0.19.172.in-addr.arpa. (41)
16:54:13.870723 IP (tos 0x0, ttl 64, id 53920, offset 0, flags [none], proto UDP (17), length 69)
    192.168.0.32.52082 > 192.168.0.1.53: [udp sum ok] 50601+ PTR? 1.0.19.172.in-addr.arpa. (41)
16:54:13.872706 IP (tos 0x0, ttl 64, id 50395, offset 0, flags [none], proto UDP (17), length 69)
    192.168.0.32.56176 > 192.168.0.1.53: [udp sum ok] 45180+ PTR? 1.0.19.172.in-addr.arpa. (41)

So it looks like something with reverse DNS lookup but I'm unsure of how to troubleshoot. Disabling wifi completely guarantees this slowness as well. Disabling and re-enabling doesn't help.

[ScottKelly]

Of course you find your own solution right after you post the question. It looks like an installed package in our dev environment updated DNS settings in OSX which cause the problem. Once I reset the OSX DNS to use the defaults in /etc/resolv.conf, everything works.

[afxjzs]

Could this be related to the issue with Bonjour 'owning' .local and an IPV6 bug?
details: https://news.ycombinator.com/item?id=11567442

[Typositoire]

Dunno if this helps but I used to have this issue and I fixed it by changing my DNS servers to 8.8.8.8 and 8.8.4.4.

Also this issue only happened on my Home network. At work I did not have this issue.

[cabhi]

I am also facing the same issue even after trying @jewilmeer's fix.
docker-compose up works fine at my office network but it takes around ~7 mins at my home network.
same behavior with other docker-compose commands like stop, pull, ps etc.

docker --version
Docker version 1.12.0-rc2, build 906eacd, experimental

docker-compose --version
docker-compose version 1.8.0-rc1, build 9bf6bc6

docker-machine --version
docker-machine version 0.8.0-rc1, build fffa6c9

[jpoutrin]

Not sure why but I had to remove any local domain element to make it work.

/etc/hosts:
127.0.0.1 localunixsocket

[moloch--]

I have a very similar issue, but I think it may be related to my use of DNSCrypt?

[phazei]

I'd run the line to monitor dns that was in the original post on the fix:

sudo tcpdump -A -s0 -nni en0 port 53

That showed me that what I needed to add to my hostfile was:

127.0.0.1 localunixsocket.localdomain

seems something changed from .local to .localdomain?

[gardner]

I have since done a fresh install of 10.12 Sierra. I reinstalled docker and could not reproduce the issue.

[Rakhmanov]

I ran in to this issue today, my first day on Mac.
Docker compose was just stalled, literally after inserting the line in to the /etc/hosts it started working as expected.
I was using WiFi, everyone in the office with same version on Ethernet never even heard of this issue.
Docker: Version 17.09.0-ce-mac35 (19611)
Mac 10.13.1 (17B48)

[ryan2x]

Same bug here. I have to add in the three lines to /etc/hosts to solve this problem.
127.0.0.1 localunixsocket
127.0.0.1 localunixsocket.lan
127.0.0.1 localunixsocket.local

[tylermercier]

Same bug. This worked for me.

$ sudo tcpdump -vvv -s 0 -l -n port 53
...
13:46:10.203734 IP (tos 0x0, ttl 255, id 7224, offset 0, flags [none], proto UDP (17), length 81, bad cksum 0 (->7b21)!)
    10.64.14.244.54683 > 10.0.0.10.53: [bad udp cksum 0x2491 -> 0xf39b!] 30038+ A? localunixsocket.*.svc.cluster.local. (53)

$ sudo vim /etc/hosts

# hacks for docker
127.0.0.1 localunixsocket.*.svc.cluster.local

[TristanMarion]

I added 127.0.0.1 localunixsocket in /etc/hosts and it worked for me, thanks !
(but it is still a wtf bug)

[alberto56]

Still a problem on the latest version. The above fixes don't seem to do anything for me, at some point it just gets so slow that it hangs. The workaround for me is to restart Docker for mac every so often.

[FelikZ]

Confirmed that 127.0.0.1 localunixsocket in /etc/hosts dramatically speeds things up, please fix!

[norbertsienkiewicz]

adding 127.0.0.1 localunixsocket in /etc/hosts helps. I'm using docker-compose version 1.18.0, build 8dd22a9

[benlinton]

The solution @norbertsienkiewicz recommended worked perfectly for me. It dropped my docker-compose up time from over 10 minutes down to less than a minute (version 1.18.0).

[jstnlef]

I'm actually more curious as to why this started happening in the first place. It seems a bit silly to me for me to have to modify my hosts file as a workaround to a problem that was recently introduced and have it declared to be the "solution".

[mnottale]

Here is the backtrace that causes the spurious DNS lookup:

  File "/Library/Python/2.7/site-packages/docker-3.1.0-py2.7.egg/docker/api/daemon.py", line 88, in info
    return self._result(self._get(self._url("/info")), True)
  File "/Library/Python/2.7/site-packages/docker-3.1.0-py2.7.egg/docker/utils/decorators.py", line 46, in inner
    return f(self, *args, **kwargs)
  File "/Library/Python/2.7/site-packages/docker-3.1.0-py2.7.egg/docker/api/client.py", line 193, in _get
    return self.get(url, **self._set_request_timeout(kwargs))
  File "/Library/Python/2.7/site-packages/requests-2.18.4-py2.7.egg/requests/sessions.py", line 521, in get
    return self.request('GET', url, **kwargs)
  File "/Library/Python/2.7/site-packages/requests-2.18.4-py2.7.egg/requests/sessions.py", line 499, in request
    prep.url, proxies, stream, verify, cert
  File "/Library/Python/2.7/site-packages/requests-2.18.4-py2.7.egg/requests/sessions.py", line 672, in merge_environment_settings
    env_proxies = get_environ_proxies(url, no_proxy=no_proxy)
  File "/Library/Python/2.7/site-packages/requests-2.18.4-py2.7.egg/requests/utils.py", line 692, in get_environ_proxies
    if should_bypass_proxies(url, no_proxy=no_proxy):
  File "/Library/Python/2.7/site-packages/requests-2.18.4-py2.7.egg/requests/utils.py", line 676, in should_bypass_proxies
    bypass = proxy_bypass(netloc)
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib.py", line 1487, in proxy_bypass
    return proxy_bypass_macosx_sysconf(host)
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib.py", line 1453, in proxy_bypass_macosx_sysconf
    hostIP = socket.gethostbyname(hostonly)

[bdrhoa]

I just had this problem after switching from wireless to wired. Back to wireless and all is right with the world.

[cphares]

Are the changes to /etc/host to the host machine or the docker container??

[abaxanean]

Which hosts file to modify ?

1. macOS hosts file ?
2. The Linux VM serving as docker host ?
3. The container itself ?

[buddhiv]

Not sure why but I had to remove any local domain element to make it work.

/etc/hosts:
127.0.0.1 localunixsocket

Genius!!!

[lmeyerov]

^^^ In the host (OS X). followed by sudo killall -HUP mDNSResponder to restart

we went 30% faster this way in catalina, but still way slower than we expect . seems especially helpful for yarn / webpack / npm for steps like slow linking and file gen for many small files.

this should have gone away w/ docker/docker-py#1928 so no idea

Edit: we're also adding:

# Prevent os x binary notarization, which may be impacting docker/python
127.0.0.1 api.apple-cloudkit.com