New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

"driver failed programming external connectivity on endpoint" (1.7.0-rc1) #3277

Closed
jamshid opened this Issue Apr 6, 2016 · 183 comments

Comments

Projects
None yet
@jamshid

jamshid commented Apr 6, 2016

I'm pretty sure this was working on docker-compose 1.7.0-rc1 before I installed the latest Docker for Mac beta, which upgraded to docker 1.11.0-rc3. I'm trying to get the socat service to run on port 172.17.0.1:8123, so that it's available to docker builds. My real yml uses an environment variable for the IP address, but it happens even when hardcoded.

ERROR: for socat_httpcache  driver failed programming external connectivity on endpoint test_socat_httpcache_1 (5d973ed559d63a5561b715248f797a336915a44960b5e32e622ac8349b16e5d2): Error starting userland proxy: failed to bind port: Unix.Unix_error(Unix.EADDRNOTAVAIL, "bind", "")
version: '2'
services:
  httpcache:
    restart: always
    image: clue/polipo
    command: proxyAddress=0.0.0.0 allowedClients=0.0.0.0/0 disableIndexing=false disableServersList=false
    mem_limit: 500m
    memswap_limit: 500m
    volumes:
      - /var/cache/polipo

  socat_httpcache:
    restart: always
    hostname: POLIPO1
    image: bobrik/socat
    mem_limit: 50m
    command: TCP-LISTEN:8123,fork,reuseaddr TCP:httpcache:8123
    depends_on:
      - httpcache
    ports:
      - "172.17.0.1:8123:8123"

verbose.txt

@aboch

This comment has been minimized.

aboch commented Apr 6, 2016

@jamshid
Can you post the o/p of sudo netstat -nlp | grep 8123. Thanks.

@jamshid

This comment has been minimized.

jamshid commented Apr 6, 2016

My mac says -p requires an argument, so I think this is what you want:

$ sudo netstat -nl -p tcp  | grep 8123
(nothing)
$ sudo netstat -nl -p tcp  | head    # to confirm mac output is what you expect
Active Internet connections
Proto Recv-Q Send-Q  Local Address          Foreign Address        (state)    
tcp4       0      0  192.168.2.4.52918      216.58.194.34.80       ESTABLISHED
tcp4       0      0  192.168.2.4.52917      23.199.236.147.80      ESTABLISHED
tcp4       0      0  192.168.2.4.52916      192.229.163.25.443     ESTABLISHED
tcp4       0      0  192.168.2.4.52915      192.229.163.175.80     ESTABLISHED
...
$ docker ps -a | grep 8123
7baf5b2ec0eb        clue/polipo               "polipo proxyAddress="   26 minutes ago      Up 12 minutes                  8123/tcp            test_httpcache_1
3fea15e3d5b4        clue/polipo               "polipo proxyAddress="   33 minutes ago      Up 19 minutes                  8123/tcp            buildenv_httpcache_1
@bezelga

This comment has been minimized.

bezelga commented Apr 29, 2016

same thing happening to me.

@leeahoward

This comment has been minimized.

leeahoward commented May 3, 2016

I am getting the same issue when restarting using docker-compose up. In my case it's an nginx proxy service that fails to start. Must be related to the Docker for Mac beta. It started after I installed the beta and hasn't happened after switching back over to Virtualbox.

docker-compose version 1.7.0, build 0d7bf73
Docker version 1.11.0, build 4dc5990

@geniousphp

This comment has been minimized.

geniousphp commented May 4, 2016

I encouner the same error when I try to start my stack using docker-compose. All the containers start correctly except one (consul). I'm getting this error

ERROR: for consul driver failed programming external connectivity on endpoint devas_consul_1 (6ceba32c51cb70b354f 2f4b68d173ca97a65d6a0bfd308d3137e2a7069abd1b6): Error starting userland proxy: Failed to bind: EADDRNOTAVAIL

I'm sure it has something to do with my docker version beta for mac.

docker-compose -v docker-compose version 1.7.0, build 0d7bf73

I'm using the last docker beta version for mac

`
docker version
Client:
Version: 1.11.1
API version: 1.23
Go version: go1.5.4
Git commit: 5604cbe
Built: Wed Apr 27 00:34:20 2016
OS/Arch: darwin/amd64

Server:
Version: 1.11.1
API version: 1.23
Go version: go1.5.4
Git commit: 8b63c77
Built: Fri Apr 29 14:13:26 2016
OS/Arch: linux/amd64
`

@zhangandyx

This comment has been minimized.

zhangandyx commented May 5, 2016

Following the above message, I get Error starting userland proxy: Failed to bind: EADDRINUSE.
Confused as to where the conflict is where none existed before

@mattgi

This comment has been minimized.

mattgi commented May 7, 2016

How does one "release" the port that is in use so that compose can bring up the container again? First time it happened, I waited for a minute or so and it worked on second attempt. Second time, its been about an hour. Docker compose kill, down, rm --all, etc. and something still has a hold of it.

@mattgi

This comment has been minimized.

mattgi commented May 7, 2016

@geniousphp

This comment has been minimized.

geniousphp commented May 13, 2016

I figure out what the problem is about, the container (consul) causing the issue has this binding port 172.17.0.1:53:8600/udp. I replaced it with 53:8600/udp and it's working now. I didn't understand yet why the first binding doesn't work. BTW, How can I know the bridge ip address?

@rsilveira65

This comment has been minimized.

rsilveira65 commented May 30, 2016

Hello, I had the same problem with Docker version 1.11.1 on Mac OSX.
ERROR: for nginx driver failed programming external connectivity on endpoint managercentralapp_nginx_1 (e11d19ee8f486a2e12c9c87802dca7ea5f541f026645f69e338e94732ac1eb56): Error starting userland proxy: Failed to bind: EADDRINUSE

@madjam002

This comment has been minimized.

madjam002 commented Jun 20, 2016

I too am getting this after upgrading to Docker 1.12-rc2, not sure if this is the right place to post this.

Client:
 Version:      1.12.0-rc2
 API version:  1.24
 Go version:   go1.6.2
 Git commit:   906eacd
 Built:        Fri Jun 17 20:35:33 2016
 OS/Arch:      darwin/amd64
 Experimental: true

Server:
 Version:      1.12.0-rc2
 API version:  1.24
 Go version:   go1.6.2
 Git commit:   906eacd-unsupported
 Built:        Fri Jun 17 21:12:28 2016
 OS/Arch:      linux/amd64
 Experimental: true
@madjam002

This comment has been minimized.

madjam002 commented Jun 21, 2016

Full error:

Error response from daemon: driver failed programming external connectivity on endpoint xxx_xxx_1 (xxx): Bind for 0.0.0.0:2224 failed: port is already allocated

netstat:

tcp6       0      0 :::2224                 :::*                    LISTEN      9083/docker-proxy
@deg

This comment has been minimized.

deg commented Jun 22, 2016

I'm seeing this sporadically. Retrying the command after a minute usually "fixes" the problem.
This is after moving to Docker for Mac yesterday.

$ docker version
Client:
 Version:      1.12.0-rc2
 API version:  1.24
 Go version:   go1.6.2
 Git commit:   906eacd
 Built:        Fri Jun 17 20:35:33 2016
 OS/Arch:      darwin/amd64
 Experimental: true

Server:
 Version:      1.12.0-rc2
 API version:  1.24
 Go version:   go1.6.2
 Git commit:   a7119de
 Built:        Fri Jun 17 22:09:20 2016
 OS/Arch:      linux/amd64
 Experimental: true

@kausik93

This comment has been minimized.

kausik93 commented Jun 29, 2016

I've also been getting the same error as you guys. I believed it was because nginx was binding it ports that were already taken up but I don't think so now. Did anyone find any solutions to this? :)

@briankelleher

This comment has been minimized.

briankelleher commented Jul 6, 2016

+1 Same error on mac

@joemewes

This comment has been minimized.

joemewes commented Jul 6, 2016

I had this happening after a move to Docker for Mac, but after stopping and removing All containers after the move, restarting docker, recreating container etc etc. It was fixed.

I had this again today after an update of Docker for Mac , but repeating the above scenario fixed. I assume the port (and "other" things) I'm assigning to particular containers is not being released until completely cleaned up.

eg:

ERROR: for nginx driver failed programming external connectivity on endpoint dev_nginx (4c340c55f7402c7b265803ea64541ebdf8613f4b43dfa816001d27bdc4f513b3): Error iptables FORWARD: Another app is currently holding the xtables lock; waiting (1s) for it to exit... ERROR: Encountered errors while bringing up the project.

@krim

This comment has been minimized.

krim commented Jul 7, 2016

+1 have the same problem on mac

@leifdejong

This comment has been minimized.

leifdejong commented Jul 7, 2016

@TuningGuide MacBook Pro (15-inch, Late 2011), 2.2 GHz Intel Core i7, 8GB 1333 MHz DDR3
Running Node 4.4.7 extended to support Ember CLI. Setup is below:

FROM node:4.4.7

# expose ember-cli ports
EXPOSE 4200 35729 49152

# install ember-cli dependencies
RUN npm install -q -g ember-cli
RUN npm install -q -g bower
RUN npm install -q -g phantomjs-prebuilt

# install watchman
# http://ember-cli.com/user-guide/#watchman
RUN \
  git clone https://github.com/facebook/watchman.git &&\
  cd watchman &&\
  git checkout v3.5.0 &&\
  ./autogen.sh &&\
  ./configure &&\
  make &&\
  make install

Ran with mounted volume of the ember-cli files using 'ember serve' as the command.

@luki3k5

This comment has been minimized.

luki3k5 commented Jul 8, 2016

+1 same problem on the mac with docker for mac

@TuningGuide

This comment has been minimized.

TuningGuide commented Jul 8, 2016

Please stop sending +1, as it spams all subscriber inboxes. Either use the emoticons (smiley on the right of each comment) or at least give your full setup, version numbers and so on. Thx

@miraage

This comment has been minimized.

miraage commented Jul 8, 2016

➜  ~ docker info
Containers: 2
 Running: 0
 Paused: 0
 Stopped: 2
Images: 2
Server Version: 1.12.0-rc3
Storage Driver: aufs
 Root Dir: /var/lib/docker/aufs
 Backing Filesystem: extfs
 Dirs: 21
 Dirperm1 Supported: true
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
 Volume: local
 Network: null host bridge overlay
Swarm: inactive
Runtimes: runc
Default Runtime: runc
Security Options: seccomp
Kernel Version: 4.4.14-moby
Operating System: Alpine Linux v3.4
OSType: linux
Architecture: x86_64
CPUs: 4
Total Memory: 1.954 GiB
Name: moby
ID: GZAT:26TX:PYCM:KKNA:JSWY:AMEL:RQYI:O23B:QGSQ:WBVB:3Q7E:XDPJ
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): true
 File Descriptors: 18
 Goroutines: 30
 System Time: 2016-07-08T17:41:26.03558348Z
 EventsListeners: 1
No Proxy: *.local, 169.254/16
Registry: https://index.docker.io/v1/
Experimental: true
Insecure Registries:
 127.0.0.0/8
➜  ~ cat docker-compose-local.yml
version: '2'
services:
  cache:
    image: redis
    command: redis-server --appendonly yes
    ports:
      - "6379:6379"
  db:
    image: postgres:9.4
    ports:
      - "5432:5432"
    volumes:
      - ./tmp/dump.gz:/tmp/dump.gz
      - ./tmp/dump.sh:/docker-entrypoint-initdb.d/dump.sh
    environment:
      POSTGRES_DB: dbname
      POSTGRES_USER: dbuser
      POSTGRES_PASSWORD: dbpass

Got next errors:

ERROR: for cache  driver failed programming external connectivity on endpoint carsaver_cache_1 (253054ef6864bdcb5377bd1690173fa352021d505dc05c3aafecf66b02c0b73d): Error starting userland proxy: write /port/tcp:0.0.0.0:6379:tcp:172.18.0.2:6379/ctl: errno 526

ERROR: for db  driver failed programming external connectivity on endpoint carsaver_db_1 (8a5990bd06a50d87c7db96de84367bb01b71c971d5a4b9e578fea7d75d893d59): Error starting userland proxy: write /port/tcp:0.0.0.0:5432:tcp:172.18.0.3:5432/ctl: errno 526
ERROR: Encountered errors while bringing up the project.
@miraage

This comment has been minimized.

miraage commented Jul 8, 2016

Found solution for people with same error as I shown above.
I was previously using Docker Toolbox, but I did not delete Virtualbox. It was using my ports so I can not start. After complete removing VirtualBox, docker-compose worked awesome.

@aanand @dnephin please, add to Getting Started a warning like explained above.

@FrenchBen

This comment has been minimized.

Contributor

FrenchBen commented Jul 8, 2016

@miraage happy to hear you solved your issue.
I didn't have your volume data+dump.sh but did a quick compose without issues:

Creating driverfailed_cache_1
Creating driverfailed_db_1
Attaching to driverfailed_cache_1, driverfailed_db_1

@jamshid Why are you specifying an IP for socat (not familiar with it)? How are you using it?
Doing a docker-compose up without the IP I don't have any issues:

Creating socat_httpcache_1
Creating socat_socat_httpcache_1
Attaching to socat_httpcache_1, socat_socat_httpcache_1
docker ps
CONTAINER ID        IMAGE               COMMAND                  CREATED             STATUS              PORTS                    NAMES
f2e3b3767982        bobrik/socat        "socat TCP-LISTEN:812"   2 seconds ago       Up 1 seconds        0.0.0.0:8123->8123/tcp   socat_socat_httpcache_1
43dbd35629e0        clue/polipo         "polipo proxyAddress="   3 seconds ago       Up 2 seconds        8123/tcp                 socat_httpcache_1

If your goal is for your http_cache to talk to the socat container, why not put them on the same network and be done with it?

version: '2'
services:
  httpcache:
    restart: always
    image: clue/polipo
    command: proxyAddress=0.0.0.0 allowedClients=0.0.0.0/0 disableIndexing=false disableServersList=false
    mem_limit: 500m
    memswap_limit: 500m
    volumes:
      - /var/cache/polipo
    networks:
      - socatnet

  socat_httpcache:
    restart: always
    hostname: POLIPO1
    image: bobrik/socat
    mem_limit: 50m
    command: TCP-LISTEN:8123,fork,reuseaddr TCP:httpcache:8123
    depends_on:
      - httpcache
    networks:
      - socatnet
networks:
  socatnet:
@marcelmfs

This comment has been minimized.

marcelmfs commented Jul 11, 2016

Well, I'm using docker engine 1.11.2 in centos7 in a Continuous Build/Integration flow to automate tests and as far as I can tell, eventually this error pops up even though no container is running (while re-deploying a specific version of the whole service stack). Also it seems related to the fact that sometimes docker engine gets an error on trying to cleanup after the removal of running containers, and some iptables rules might be kept allocating network ports for some reason, maybe related to moby/moby#23371 (comment) - but I'm still not sure about that.

djs55 added a commit to djs55/vpnkit that referenced this issue Jul 11, 2016

hostnet: convert Uwt errors to Unix_errors, pretty-print "expected" c…
…ases

Previously we lacked a function to convert from Uwt.error to Unix.error.
Happily uwt.0.0.3 contains such a useful error converting function, so
we use it.

When a port forward is requested, use `Lwt.catch` to handle the exception
at the top-level and pretty-print the error message for the expected cases

- EADDRINUSE: the port was already 'allocated'
- EADDRNOTAVAIL: the interface IP was not found (probably it exists in a
  VM but not on the host)
- EPERM: insufficient privileges to bind a privileged port

The pretty-printing is intended to match the output of `docker` on
regular Linux.

This is related to [docker/compose#3277]

Signed-off-by: David Scott <dave.scott@docker.com>

djs55 added a commit to djs55/vpnkit that referenced this issue Jul 12, 2016

hostnet: convert Uwt errors to Unix_errors, pretty-print "expected" c…
…ases

Previously we lacked a function to convert from Uwt.error to Unix.error.
Happily uwt.0.0.3 contains such a useful error converting function, so
we use it.

When a port forward is requested, use `Lwt.catch` to handle the exception
at the top-level and pretty-print the error message for the expected cases

- EADDRINUSE: the port was already 'allocated'
- EADDRNOTAVAIL: the interface IP was not found (probably it exists in a
  VM but not on the host)
- EPERM: insufficient privileges to bind a privileged port

The pretty-printing is intended to match the output of `docker` on
regular Linux.

This is related to [docker/compose#3277]

Signed-off-by: David Scott <dave.scott@docker.com>
@igorrKurr

This comment has been minimized.

igorrKurr commented Jul 12, 2016

I've had pretty same issue, with Docker for Mac, running postgresql container. So, for me it was caused by postgres running on same port on host system as port specified in my compose config file. Pretty obvious thing, but could be helpful for someone I guess.
Thanks for cool tool. 👍

@marcosalberto

This comment has been minimized.

marcosalberto commented Jul 12, 2016

I had a similar problem Docker for Mac running nginx container, my port 80 was being used by apache (native). I used telnet and netstat to see the running service.

@surajadh

This comment has been minimized.

surajadh commented Jul 12, 2016

For me it was squid, I stopped squid and didn't see that anymore. So I think it may have to do with whatever is taking up the port and if we force docker to map that port.

@wanderingstan

This comment has been minimized.

wanderingstan commented Jul 3, 2018

Restarting docker (via "Restart" command or manually quitting and re-launching) did NOT work for me.

@Kryshtopenko

This comment has been minimized.

Kryshtopenko commented Aug 6, 2018

Reset Docker to factory defaults

It helped me.

@mbXian

This comment has been minimized.

mbXian commented Aug 8, 2018

it help me by reset Docker.

@gudge25

This comment has been minimized.

gudge25 commented Aug 8, 2018

@Ray-Eldath

This comment has been minimized.

Ray-Eldath commented Sep 2, 2018

Restart Docker works for me. (Docker for Windows, Docker version 18.06.1-ce, build e68fc7a)

@VestOfHolding

This comment has been minimized.

VestOfHolding commented Sep 5, 2018

To add to Ray-Eldath's comment: I was confused because restarting my computer twice didn't work (same version: 18.06.1-ce for Windows), and surely restarting your computer is enough to count as restarting Docker too, right? So I guess restarting didn't fix it, right? Well, no, lol. I actually go to Docker and click "Restart" and now it works.

EDIT: Oh, interesting, now I know why this happened (probably). Restarting my computer actually created this problem I think. Check this out:

https://stackoverflow.com/questions/40668908/running-docker-for-windows-error-when-exposing-ports

Relevant quote:

The last Windows 10 update (Fall Creators Update, 2017) has a new "feature". It automatically starts any applications that were running when you last shutdown.

This reconstitutes Docker for Windows in a bad state. That made it appear those ports were in use by something else - it was the ghost of itself. This explained why those ports were still in use even though I stopped/started my containers and even reboot!

Given that, I'm still suspect of the 18.06.1-ce Windows update, since this wasn't a problem for me until I updated to that.

@folterj

This comment has been minimized.

folterj commented Sep 7, 2018

I have Windows 10 Pro both on a laptop and desktop with the exact same issue. I'm currently running version: 18.06.1-ce, but at least the last 2 previous versions had the same behavior. This is a consistent and easily reproducible issue, occurring on a very common configuration.

Are we all doing something wrong, or is this something that requires fixing?

@djpysu

This comment has been minimized.

djpysu commented Sep 10, 2018

You can check also with ACTIVITY MONITOR on OSX. I saw this problem when another user is logged in and is using Docker. Kill the processes or logout and it will work.

@lbcoker

This comment has been minimized.

lbcoker commented Sep 13, 2018

I had this happen on docker for windows with nginx. I restarted docker and do not see it now. Of course it will probably be there if I remove then create a new container for nginx.

@J0-nas

This comment has been minimized.

J0-nas commented Sep 14, 2018

I had the same problem (starting a nginx reverse proxy on port 80) on Windows 10, maybe caused by a previous virtual box installation.

Resetting docker to factory defaults (rightclick icon in notification area -> settings -> reset) fixed it for me.

@ballmdr

This comment has been minimized.

ballmdr commented Sep 19, 2018

restart docker work for me. on windows 10

@BuzaL

This comment has been minimized.

BuzaL commented Sep 19, 2018

thx @ballmdr , that`s works for me either.

@shivdeepnv

This comment has been minimized.

shivdeepnv commented Sep 27, 2018

I was getting the same issue. Port 80 was being used by IIS.

  1. Hit Run, type - 'inetmgr' which opens up IIS
  2. On Actions (Top Right) - Click 'Stop'
  3. Do Docker run again
@makproductions

This comment has been minimized.

makproductions commented Oct 2, 2018

I was getting the same issue on MAC moJave after updating docker-compose. Simply restarting docker fixes the issues for me.

@diviak

This comment has been minimized.

diviak commented Oct 3, 2018

Same here, I have macOS Mojave and restarting fixed issues.

@makproductions thanks.

@prabhuignoto

This comment has been minimized.

prabhuignoto commented Oct 6, 2018

I am getting this issue pretty regularly(on windows ) and restarting Docker solves the problem.

@napogf

This comment has been minimized.

napogf commented Oct 7, 2018

Ubuntu 18.04, restarting docker solve the problem

@damtur

This comment has been minimized.

damtur commented Oct 12, 2018

The problem still exists on Windows in 18.06.1-ce-win73 (19507) Plase re-open this.

@omrishalev22

This comment has been minimized.

omrishalev22 commented Oct 12, 2018

Usually happens since postgresql is running and using port 5432 try stopping the service and running the docker-compose up --build so it will do the initialization..

sudo service postgresql stop
docker-compose up --build

should work

@vivekruhela

This comment has been minimized.

vivekruhela commented Oct 15, 2018

madjam002

1.docker rm -f all containers.
2.docker network rm network interfaces.
3.sudo rm /var/lib/docker/network/files/local-kv.db

I tried this now I am stuck with another problem as docker: Error response from daemon: failed to update store for object type *libnetwork.endpointCnt: Key not found in store.. How to solve this problem, can you suggest me.... Thanks.

@pd93

This comment has been minimized.

pd93 commented Oct 15, 2018

I'm aware this thread still generates a lot of discussion, but I'm going to close it as it has nothing to do with Compose per se.

This is a docker issue - not a docker-compose issue. This issue is closed, so isn't going to gain any traction. There are several issues already open in the main docker repo. Comments should probably go there:

See docker/for-win#2722
and docker/for-win#1967

@shin- Should this be locked?

@bdaniel7

This comment has been minimized.

bdaniel7 commented Oct 25, 2018

I get the same error in the following situation:
Docker 18.06.1, Docker Compose 1.22.0, Ubuntu 18.04

Starting d1df6864f98d2599_api_1 for d1df6864f98d2599_api_1 Cannot start service api: driver failed programming external connectivity on endpoint d1df6864f98d2599_api_1 (6b3f4a9c68d6858ace2b39a0296ed41620ff3007b14c2379b539397267228f7d): Bind for 0.0.0.0:5000 failed: port is already allocated

When I start the containers using docker-compose from command line, the container is called src_api_1, because I'm running the command from a src folder.

When I run docker-compose from TeamCity, the container is called d1df6864f98d2599_api_1 and I get that error.

The command is docker-compose -f src/docker-compose-production.yml up -d api.

@ronakganatra9

This comment has been minimized.

ronakganatra9 commented Oct 27, 2018

I just changed my port number in .yaml file and it started working for me.

@rsunmoto

This comment has been minimized.

rsunmoto commented Nov 7, 2018

In my case, it was because windows login password was updated and apparently the new password was not updated to docker. I had to go to docker for windows -> settings -> shared drive, uncheck shared drives, click apply, and then check again and apply, it will then prompt for password. After I enter the password, it's is working again.

@tylerssn

This comment has been minimized.

tylerssn commented Nov 8, 2018

This is continuing to plague my setup. Restarting docker does not free up the port on OSX. Restarting the host machine does free up the port.

@jtveiteras

This comment has been minimized.

jtveiteras commented Nov 13, 2018

I got the same error message, but my problem was caused by working on two different pages at the same time.

When switching from one too the other I forgot to use docker-compose down,

@PremiumLearnings

This comment has been minimized.

PremiumLearnings commented Nov 14, 2018

I was also facing the same problem but after a long time a meet with the perfect solution

  1. type docker container ls -a (check the container which is using that port number )
  2. docker container rm -f (Deallocate the port by removing that container)
  3. Assing your new container with your port number
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment