New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

docker-compose slow on docker for mac os beta #3419

Closed
Erwyn opened this Issue May 4, 2016 · 107 comments

Comments

Projects
None yet
@Erwyn

Erwyn commented May 4, 2016

docker-compose is slow with docker for mac os beta on my home network. Here is my workaround for now:

  • docker-compose up (take ages)
  • shut down wifi
  • docker-compose up (really fast)
  • re-enable wifi

I do not reproduce the issue on another network than mine, my work network for instance do not make it slow. I already had a potentially related issue with the docker client itself which couldn't pull any image (going to bizarre local ips instead of the docker hub registry) but it has been fixed since one of the latest docker for mac os beta update.

The issue is not reproduced against the docker-toolbox, only the "native" docker for mac.

My version of docker-compose is : docker-compose version 1.7.0, build 0d7bf73
My version of docker for mac is: Version 1.11.1-beta10 (build: 6662)

The docker-compose file I'm trying to run is:

#docker-compose.yml
sync-engine:
  build: nylas-sync-engine
  ports:
    - 5555:5555
  links:
    - mysql:mysql
    - redis:redis
  hostname: sync-engine
  log_opt:
    max-size: "10m"
    max-file: "10"

mysql:
  image: mysql
  environment:
    - MYSQL_ROOT_PASSWORD=whateverpassword
  volumes:
    - nylas_mysql:/var/lib/mysql
  log_opt:
    max-size: "10m"
    max-file: "10"

redis:
  image: redis
  volumes:
    - nylas_redis:/data
  log_opt:
    max-size: "10m"
    max-file: "10"
@thaJeztah

This comment has been minimized.

Member

thaJeztah commented May 4, 2016

ping @FrenchBen 😄

@smyth64

This comment has been minimized.

smyth64 commented May 4, 2016

+1

@smyth64

This comment has been minimized.

smyth64 commented May 4, 2016

got the same issue :D

@thaJeztah

This comment has been minimized.

Member

thaJeztah commented May 4, 2016

@smith64fx problem also goes away if you turn off your WiFi?

@smyth64

This comment has been minimized.

smyth64 commented May 5, 2016

@stijn Yes, when i turn off wifi everything works like charm

Von meinem iPhone gesendet

Am 05.05.2016 um 00:26 schrieb Sebastiaan van Stijn notifications@github.com:

@smith64fx problem also goes away if you turn off your WiFi?


You are receiving this because you were mentioned.
Reply to this email directly or view it on GitHub

@Erwyn

This comment has been minimized.

Erwyn commented May 5, 2016

@smith64fx I see from your signature that you are likely from Germany (or Austria/switzerland). Do you mind me asking what is your Internet provider? I'm wondering if we have the same, and if it could be coming from the box which does not look like a very good piece of hardware/software and would not act as thought by docker.

(I am with Vodafone and I have their easybox-whatever)

@holstvoogd

This comment has been minimized.

holstvoogd commented May 6, 2016

Same issue on a wired network (corporate network), as soon as I un plug it everything is very fast. So I'm positive it is not related to you specific provider.

I've looked at the verbose output, there is no obvious error (nor anywhere in the system logs). Did notice this though:

With no networking, these lines are grouped together:

docker attach <- (u'd90162cbfd7b312c28488a209440641b2d9c9f99e8eb59082f20ddf7d84e7b7e', stderr=True, stream=True, stdout=True)
docker attach -> <generator object _multiplexed_response_stream_helper at 0x1045f20a0>
docker start <- (u'd90162cbfd7b312c28488a209440641b2d9c9f99e8eb59082f20ddf7d84e7b7e')

Without networking, I get ~100-200 lines of 'compose.parallel.feed_queue: Pending: set([])' between the attach <- and where it returns with attach -> ...

Before this point there is a lot more happening with network as well, but mostly just more calls to inspect image etc it seems.

I've attached the output of compose --verbose up for bot situations. The compose file is just two containers straight from the hub.

with-networking.txt
without-networking.txt

@Erwyn

This comment has been minimized.

Erwyn commented May 6, 2016

@holstvoogd oh, ok for the provider. Thanks for the info, I was a bit worried :)

@FrenchBen

This comment has been minimized.

Contributor

FrenchBen commented May 6, 2016

@Erwyn @smith64fx To confirm, you're always connected (hard-wired) and at the same time use the same network with wifi?

@smyth64

This comment has been minimized.

smyth64 commented May 7, 2016

@FrenchBen No it's only in the wifi network in my home. At my office it's super fast. But the thing is, everything else at home runs fast, except docker-compose ^_^"

@Erwyn

This comment has been minimized.

Erwyn commented May 9, 2016

@FrenchBen same as @smith64fx. It's only wifi at home so there is no wired connection involved. And as I can see @holstvoogd seems to have the same issue with a wired connection.

@eugenesia

This comment has been minimized.

eugenesia commented May 16, 2016

I am noticing the same issue with Docker for Mac Beta. docker-compose up is slow when Wifi is enabled, fast when Wifi is disabled.

  • Docker Compose version: docker-compose version 1.7.1, build 0a9ab35
  • Docker version: Docker version 1.11.1, build 5604cbe
  • OS version: OS X El Capitan 10.11.4
@KryDos

This comment has been minimized.

KryDos commented May 25, 2016

Hi, Is there any news about this?

I have same issue. Compose/Docker/OSX versions are same as @eugenesia has.
I use WiFi at home and at office and at my home it works incredibly slow. At office it works as expected (fast).

I thought maybe it is something related to DNS server of my ISP (home and office internet providers are different) and I tried to use some public DNS including Google's one but it didn't help.

If I turn WiFi off then docker-compose works really fast

@FrenchBen

This comment has been minimized.

Contributor

FrenchBen commented May 25, 2016

@KryDos A new release should be coming out this week with some speed improvements

@runand

This comment has been minimized.

runand commented May 26, 2016

I have the same issue, after updating to docker for mac 1.11.1-beta13, the problem persists. The workaround by switching the network(s) off and on again does not work anymore, the services start fast when switching the network off, but when switching the network on again, the services is no longer accessible and the docker daemon is not responding

docker ps
Error response from daemon: Bad response from Docker engine
  • Docker Compose version: docker-compose version 1.7.1, build 0a9ab35
  • Docker version: Docker version 1.11.1-beta13, build 7975
  • OS version: OS X El Capitan 10.11.5
@jewilmeer

This comment has been minimized.

jewilmeer commented May 26, 2016

I've had the same issues, and found a post (sorry, lost the ref) mentioning docker-compose is trying to resolve localunixsocket.local. You can get insight into the dns lookup by running sudo tcpdump -A -s0 -nni en0 port 53

For now I've pointed localunixsocket.local to localhost in my /etc/hosts. Now everything is speedy again.

127.0.0.1 localunixsocket.local
@thaJeztah

This comment has been minimized.

Member

thaJeztah commented May 26, 2016

Thanks @jewilmeer, that looks useful

@smyth64

This comment has been minimized.

smyth64 commented May 26, 2016

I switched back using virtualbox with docker-machine. Problem doesn't exist and it's up to 10x faster than Docker Mac Beta!

@thaJeztah

This comment has been minimized.

Member

thaJeztah commented May 26, 2016

@smith64fx please keep your comments constructive; it's a beta, it's not a finished product yet, so bugs and performance issues are expected. It's exactly these kind of issues that need reporting (and testing) to resolve them.

@seeekr

This comment has been minimized.

seeekr commented May 26, 2016

super awesome comment, @jewilmeer! For me I had to add a few more addresses to /etc/hosts which I discovered using your tcpdump command:

# (yours)
127.0.0.1 localunixsocket.local
# additional ones -- be sure to replace bracketed thing with the output of `hostname`
127.0.0.1 localunixsocket.home <my hostname>.home

After those additions -- speedy! Actually, amazingly speedy, seems a good bunch faster than when using Docker Toolbox! woop woop :) (Though that may be a highly subjective observation!)

@justincormack

This comment has been minimized.

Member

justincormack commented May 26, 2016

That is pretty odd, but seems to point at what the underlying issue is...

@ScottKelly

This comment has been minimized.

ScottKelly commented Jun 5, 2016

I'm currently facing this same issue and I can't resolve the problem.
I've tried the previous suggestions for editing /etc/hosts. On our office network, docker is extremely slow. On home networks or using tethering to a cell phone removes all slow downs and everything is snappy.

We're using docker-compose with a web container linked to four service containers (postgres, redis, rabbitmq, elasticsearch). Connecting to any of the four service containers directly from OSX is fine. It's only slow when trying to connect from the web container to any of the service containers.

Running tcpdump -vvv -s 0 -l -n port 53 gives the following output when tethered to a cell phone

tcpdump: data link type PKTAP
tcpdump: listening on pktap, link-type PKTAP (Packet Tap), capture size 262144 bytes
16:54:34.236026 IP (tos 0x0, ttl 64, id 25341, offset 0, flags [none], proto UDP (17), length 69)
    172.20.10.4.56662 > 172.20.10.1.53: [udp sum ok] 5785+ PTR? 1.0.19.172.in-addr.arpa. (41)
16:54:34.239617 IP (tos 0x0, ttl 64, id 29137, offset 0, flags [none], proto UDP (17), length 123)
    172.20.10.1.53 > 172.20.10.4.56662: [udp sum ok] 5785 NXDomain* q: PTR? 1.0.19.172.in-addr.arpa. 0/1/0 ns: 19.172.IN-ADDR.ARPA. [1d] SOA 19.172.IN-ADDR.ARPA. . 0 28800 7200 604800 86400 (95)
16:54:34.303325 IP (tos 0x0, ttl 64, id 46188, offset 0, flags [none], proto UDP (17), length 69)
    172.20.10.4.58590 > 172.20.10.1.53: [udp sum ok] 52066+ PTR? 1.0.19.172.in-addr.arpa. (41)
16:54:34.309241 IP (tos 0x0, ttl 64, id 7329, offset 0, flags [none], proto UDP (17), length 123)
    172.20.10.1.53 > 172.20.10.4.58590: [udp sum ok] 52066 NXDomain* q: PTR? 1.0.19.172.in-addr.arpa. 0/1/0 ns: 19.172.IN-ADDR.ARPA. [1d] SOA 19.172.IN-ADDR.ARPA. . 0 28800 7200 604800 86400 (95)
16:54:39.309028 IP (tos 0x0, ttl 64, id 52570, offset 0, flags [none], proto UDP (17), length 69)
    172.20.10.4.63544 > 172.20.10.1.53: [udp sum ok] 12851+ PTR? 1.0.19.172.in-addr.arpa. (41)
16:54:39.320135 IP (tos 0x0, ttl 64, id 32359, offset 0, flags [none], proto UDP (17), length 123)
    172.20.10.1.53 > 172.20.10.4.63544: [udp sum ok] 12851 NXDomain* q: PTR? 1.0.19.172.in-addr.arpa. 0/1/0 ns: 19.172.IN-ADDR.ARPA. [1d] SOA 19.172.IN-ADDR.ARPA. . 0 28800 7200 604800 86400 (95)
16:54:39.392915 IP (tos 0x0, ttl 64, id 8082, offset 0, flags [none], proto UDP (17), length 69)
    172.20.10.4.59994 > 172.20.10.1.53: [udp sum ok] 22334+ PTR? 1.0.19.172.in-addr.arpa. (41)
16:54:39.416821 IP (tos 0x0, ttl 64, id 51863, offset 0, flags [none], proto UDP (17), length 123)
    172.20.10.1.53 > 172.20.10.4.59994: [udp sum ok] 22334 NXDomain* q: PTR? 1.0.19.172.in-addr.arpa. 0/1/0 ns: 19.172.IN-ADDR.ARPA. [1d] SOA 19.172.IN-ADDR.ARPA. . 0 28800 7200 604800 86400 (95)

and this is the output when on our office wifi:

16:54:13.870062 IP (tos 0x0, ttl 64, id 17811, offset 0, flags [none], proto UDP (17), length 69)
    192.168.0.32.56316 > 192.168.0.1.53: [udp sum ok] 21469+ PTR? 1.0.19.172.in-addr.arpa. (41)
16:54:13.870723 IP (tos 0x0, ttl 64, id 53920, offset 0, flags [none], proto UDP (17), length 69)
    192.168.0.32.52082 > 192.168.0.1.53: [udp sum ok] 50601+ PTR? 1.0.19.172.in-addr.arpa. (41)
16:54:13.872706 IP (tos 0x0, ttl 64, id 50395, offset 0, flags [none], proto UDP (17), length 69)
    192.168.0.32.56176 > 192.168.0.1.53: [udp sum ok] 45180+ PTR? 1.0.19.172.in-addr.arpa. (41)

So it looks like something with reverse DNS lookup but I'm unsure of how to troubleshoot. Disabling wifi completely guarantees this slowness as well. Disabling and re-enabling doesn't help.

@ScottKelly

This comment has been minimized.

ScottKelly commented Jun 5, 2016

Of course you find your own solution right after you post the question. It looks like an installed package in our dev environment updated DNS settings in OSX which cause the problem. Once I reset the OSX DNS to use the defaults in /etc/resolv.conf, everything works.

@afxjzs

This comment has been minimized.

afxjzs commented Jun 18, 2016

Could this be related to the issue with Bonjour 'owning' .local and an IPV6 bug?
details: https://news.ycombinator.com/item?id=11567442

@Typositoire

This comment has been minimized.

Typositoire commented Jun 28, 2016

Dunno if this helps but I used to have this issue and I fixed it by changing my DNS servers to 8.8.8.8 and 8.8.4.4.

Also this issue only happened on my Home network. At work I did not have this issue.

@cabhi

This comment has been minimized.

cabhi commented Jul 4, 2016

I am also facing the same issue even after trying @jewilmeer's fix.
docker-compose up works fine at my office network but it takes around ~7 mins at my home network.
same behavior with other docker-compose commands like stop, pull, ps etc.

docker --version
Docker version 1.12.0-rc2, build 906eacd, experimental

docker-compose --version
docker-compose version 1.8.0-rc1, build 9bf6bc6

docker-machine --version
docker-machine version 0.8.0-rc1, build fffa6c9

@jpoutrin

This comment has been minimized.

jpoutrin commented Jul 6, 2016

Not sure why but I had to remove any local domain element to make it work.

/etc/hosts:
127.0.0.1 localunixsocket

@moloch--

This comment has been minimized.

moloch-- commented Jul 9, 2016

I have a very similar issue, but I think it may be related to my use of DNSCrypt?

@zackify

This comment has been minimized.

zackify commented Oct 5, 2017

Any word on a real solution to this other than having to tell everyone to add a host rule or turn off proxies?

@PascalTurbo

This comment has been minimized.

PascalTurbo commented Oct 6, 2017

+1

2 similar comments
@elalemanyo

This comment has been minimized.

elalemanyo commented Oct 6, 2017

+1

@giovannetti-eric

This comment has been minimized.

giovannetti-eric commented Oct 9, 2017

+1

@algal

This comment has been minimized.

algal commented Nov 2, 2017

What fixed it for me was going to my System Preferences / Network / Advanced / DNS / Search Domains, and removing the entry ".local." which I had put there. This caused the macOS to populate the Search Domains only with the default value, "localdomain". And then docker-compose became responsive again.

docker itself was responsive all the time.

I don't know, but I would guess that perhaps docker is correctly finding an on-system resource using an IP address or a stable local name, while docker-compose is unsafely relying on localdomain always being defined as one of the search domains. But I dunno!

@phazei

This comment has been minimized.

phazei commented Nov 17, 2017

I'd run the line to monitor dns that was in the original post on the fix:

sudo tcpdump -A -s0 -nni en0 port 53

That showed me that what I needed to add to my hostfile was:

127.0.0.1 localunixsocket.localdomain

seems something changed from .local to .localdomain?

@gardner

This comment has been minimized.

gardner commented Nov 17, 2017

I have since done a fresh install of 10.12 Sierra. I reinstalled docker and could not reproduce the issue.

@Rakhmanov

This comment has been minimized.

Rakhmanov commented Nov 20, 2017

I ran in to this issue today, my first day on Mac.
Docker compose was just stalled, literally after inserting the line in to the /etc/hosts it started working as expected.
I was using WiFi, everyone in the office with same version on Ethernet never even heard of this issue.
Docker: Version 17.09.0-ce-mac35 (19611)
Mac 10.13.1 (17B48)

@ryan2x

This comment has been minimized.

ryan2x commented Nov 22, 2017

Same bug here. I have to add in the three lines to /etc/hosts to solve this problem.
127.0.0.1 localunixsocket
127.0.0.1 localunixsocket.lan
127.0.0.1 localunixsocket.local

@tylermercier

This comment has been minimized.

tylermercier commented Nov 23, 2017

Same bug. This worked for me.

$ sudo tcpdump -vvv -s 0 -l -n port 53
...
13:46:10.203734 IP (tos 0x0, ttl 255, id 7224, offset 0, flags [none], proto UDP (17), length 81, bad cksum 0 (->7b21)!)
    10.64.14.244.54683 > 10.0.0.10.53: [bad udp cksum 0x2491 -> 0xf39b!] 30038+ A? localunixsocket.*.svc.cluster.local. (53)

$ sudo vim /etc/hosts

# hacks for docker
127.0.0.1 localunixsocket.*.svc.cluster.local
@TristanMarion

This comment has been minimized.

TristanMarion commented Dec 6, 2017

I added 127.0.0.1 localunixsocket in /etc/hosts and it worked for me, thanks !
(but it is still a wtf bug)

@alberto56

This comment has been minimized.

alberto56 commented Dec 19, 2017

Still a problem on the latest version. The above fixes don't seem to do anything for me, at some point it just gets so slow that it hangs. The workaround for me is to restart Docker for mac every so often.

@FelikZ

This comment has been minimized.

FelikZ commented Dec 20, 2017

Confirmed that 127.0.0.1 localunixsocket in /etc/hosts dramatically speeds things up, please fix!

@norbertsienkiewicz

This comment has been minimized.

norbertsienkiewicz commented Jan 22, 2018

adding 127.0.0.1 localunixsocket in /etc/hosts helps. I'm using docker-compose version 1.18.0, build 8dd22a9

@benlinton

This comment has been minimized.

benlinton commented Feb 15, 2018

The solution @norbertsienkiewicz recommended worked perfectly for me. It dropped my docker-compose up time from over 10 minutes down to less than a minute (version 1.18.0).

@jstnlef

This comment has been minimized.

jstnlef commented Feb 21, 2018

I'm actually more curious as to why this started happening in the first place. It seems a bit silly to me for me to have to modify my hosts file as a workaround to a problem that was recently introduced and have it declared to be the "solution".

@mnottale

This comment has been minimized.

Contributor

mnottale commented Feb 26, 2018

Here is the backtrace that causes the spurious DNS lookup:

  File "/Library/Python/2.7/site-packages/docker-3.1.0-py2.7.egg/docker/api/daemon.py", line 88, in info
    return self._result(self._get(self._url("/info")), True)
  File "/Library/Python/2.7/site-packages/docker-3.1.0-py2.7.egg/docker/utils/decorators.py", line 46, in inner
    return f(self, *args, **kwargs)
  File "/Library/Python/2.7/site-packages/docker-3.1.0-py2.7.egg/docker/api/client.py", line 193, in _get
    return self.get(url, **self._set_request_timeout(kwargs))
  File "/Library/Python/2.7/site-packages/requests-2.18.4-py2.7.egg/requests/sessions.py", line 521, in get
    return self.request('GET', url, **kwargs)
  File "/Library/Python/2.7/site-packages/requests-2.18.4-py2.7.egg/requests/sessions.py", line 499, in request
    prep.url, proxies, stream, verify, cert
  File "/Library/Python/2.7/site-packages/requests-2.18.4-py2.7.egg/requests/sessions.py", line 672, in merge_environment_settings
    env_proxies = get_environ_proxies(url, no_proxy=no_proxy)
  File "/Library/Python/2.7/site-packages/requests-2.18.4-py2.7.egg/requests/utils.py", line 692, in get_environ_proxies
    if should_bypass_proxies(url, no_proxy=no_proxy):
  File "/Library/Python/2.7/site-packages/requests-2.18.4-py2.7.egg/requests/utils.py", line 676, in should_bypass_proxies
    bypass = proxy_bypass(netloc)
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib.py", line 1487, in proxy_bypass
    return proxy_bypass_macosx_sysconf(host)
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib.py", line 1453, in proxy_bypass_macosx_sysconf
    hostIP = socket.gethostbyname(hostonly)
@bdrhoa

This comment has been minimized.

bdrhoa commented May 22, 2018

I just had this problem after switching from wireless to wired. Back to wireless and all is right with the world.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment