Docker Compose doesn't work when SSH connection used to remote Docker Engine

[raesene]

Description of the issue

Just trying out the new SSH connection introduced in Docker 18.09 and I noticed an error when attempting to do docker-compose up whilst targeting a remote Docker Engine instance.

Errors message below appears to indicate that Compose isn't aware of the SSH protocol for this purpose

docker.errors.DockerException: Invalid bind address protocol: ssh://xfoxy.secinternal.local
[486] Failed to execute script docker-compose

Context information (for bug reports)

Output of docker-compose version

docker-compose version 1.23.1, build b02f1306
docker-py version: 3.5.0
CPython version: 3.6.7
OpenSSL version: OpenSSL 1.1.0f  25 May 2017

Output of docker version

Client:
 Version:           18.09.0
 API version:       1.39
 Go version:        go1.10.4
 Git commit:        4d60db4
 Built:             Wed Nov  7 00:49:01 2018
 OS/Arch:           linux/amd64
 Experimental:      false

Server: Docker Engine - Community
 Engine:
  Version:          18.09.0
  API version:      1.39 (minimum version 1.12)
  Go version:       go1.10.4
  Git commit:       4d60db4
  Built:            Wed Nov  7 00:16:44 2018
  OS/Arch:          linux/amd64
  Experimental:     false

Output of docker-compose config
(Make sure to add the relevant -f and other flags)

networks:
  testnet: {}
services:
  dradis:
    image: raesene/dradis
    networks:
      testnet: null
    ports:
    - 3000/tcp
    volumes:
    - data:/data:rw
  sectest:
    image: raesene/sectest
    networks:
      testnet: null
    ports:
    - 22/tcp
    volumes:
    - data:/data:rw
version: '3.0'
volumes:
  data: {}

Steps to reproduce the issue

1. Configure a Docker client (18.09) to connect to a remote Docker engine instance via SSH
2. Run docker-compose up in a directory with a docker-compose.yml file.
3. Error occors.

Observed result

Error occurs

Expected result

Docker compose contacts the remote docker engine instance to create the containers.

Stacktrace / full error message

Traceback (most recent call last):
  File "bin/docker-compose", line 6, in <module>
  File "compose/cli/main.py", line 71, in main
  File "compose/cli/main.py", line 124, in perform_command
  File "compose/cli/command.py", line 42, in project_from_options
  File "compose/cli/command.py", line 123, in get_project
  File "compose/cli/command.py", line 94, in get_client
  File "compose/cli/docker_client.py", line 127, in docker_client
  File "site-packages/docker/api/client.py", line 118, in __init__
  File "site-packages/docker/utils/utils.py", line 256, in parse_host
docker.errors.DockerException: Invalid bind address protocol: ssh://xfoxy.secinternal.local
[486] Failed to execute script docker-compose

Additional information

Client is WSL (Ubuntu 18.04) Server is Ubuntu 18.04 running Docker 18.09.

[shin-]

Support for the SSH protocol will be added in the next version of Compose. docker/docker-py#2159

[raesene]

Cool Thanks for the info. :)

[mmiglier]

Tested this with docker-compose version 1.24.0-rc1 but it doesn't look like it is reading my ~/.ssh/config. DOCKER_HOST=ssh://my.docker.host docker ps works fine while DOCKER_HOST=ssh://my.docker.host docker-compose ps gives me paramiko.ssh_exception.AuthenticationException: Authentication failed.

[y0zg]

Tested this with docker-compose version 1.24.0-rc1 but it doesn't look like it is reading my ~/.ssh/config. DOCKER_HOST=ssh://my.docker.host docker ps works fine while DOCKER_HOST=ssh://my.docker.host docker-compose ps gives me paramiko.ssh_exception.AuthenticationException: Authentication failed.

settings sshd config to MaxSessions 30 helped as workaround

[tskipa]

Any update on this?

[ssbarnea]

Apparently switching to edge version on mac (which uses 1.24) makes it work but is really inconvenient as edge comes with its own set of issues.

[fzamperin]

Confirm that using docker-compose -H ssh://user@my.host up gives me:

/var/folders/vn/5g8_6g0s63s0zzz8kcdnxx2c0000gn/T/_MEIub7S75/paramiko/ecdsakey.py:164: CryptographyDeprecationWarning: Support for unsafe construction of public numbers from encoded data will be removed in a future version. Please use EllipticCurvePublicKey.from_encoded_point
Traceback (most recent call last):
File "docker-compose", line 6, in
File "compose/cli/main.py", line 71, in main
File "compose/cli/main.py", line 124, in perform_command
File "compose/cli/command.py", line 42, in project_from_options
File "compose/cli/command.py", line 123, in get_project
File "compose/cli/command.py", line 94, in get_client
File "compose/cli/docker_client.py", line 127, in docker_client
File "site-packages/docker/api/client.py", line 166, in init
File "site-packages/docker/transport/sshconn.py", line 84, in init
File "site-packages/docker/transport/sshconn.py", line 94, in _connect
File "site-packages/paramiko/client.py", line 343, in connect
File "site-packages/paramiko/util.py", line 280, in retry_on_signal
File "site-packages/paramiko/client.py", line 343, in
TimeoutError: [Errno 60] Operation timed out
[21295] Failed to execute script docker-compose

But with docker command docker -H ssh://user@docker.host ps works normally, using docker-compose version 1.24.1, build 4667896 on Mac OS and Docker version 19.03.1, build 74b1e89

[cuttingd]

Any news on this? I'm running this: docker-compose -H ssh://user@host down and I get the following:

Traceback (most recent call last):
File "docker-compose", line 6, in
File "compose\cli\main.py", line 71, in main
File "compose\cli\main.py", line 124, in perform_command
File "compose\cli\command.py", line 42, in project_from_options
File "compose\cli\command.py", line 123, in get_project
File "compose\cli\command.py", line 94, in get_client
File "compose\cli\docker_client.py", line 127, in docker_client
File "site-packages\docker\api\client.py", line 166, in init
File "site-packages\docker\transport\sshconn.py", line 84, in init
File "site-packages\docker\transport\sshconn.py", line 94, in _connect
File "site-packages\paramiko\client.py", line 437, in connect
File "site-packages\paramiko\client.py", line 676, in _auth
File "site-packages\paramiko\agent.py", line 379, in init
File "site-packages\paramiko\agent.py", line 65, in _connect
File "site-packages\paramiko\agent.py", line 83, in _send_message
File "site-packages\paramiko\agent.py", line 91, in _read_all
paramiko.ssh_exception.SSHException: lost ssh-agent
[16740] Failed to execute script docker-compose

[spirit986]

Can confirm I am having the same issue as @raesene

[jayfk]

I believe that most of the SSH related problems stem from the fact that docker-py (which is used by docker-compose) uses the paramiko ssh library instead of defaulting to the ssh binary like the docker cli.

There's an issue in docker-py discussing this: docker/docker-py#2289

[dockermaestro]

Any update?

[zmunro]

Why is this issue closed? I am still unable to set a docker-compose context to a docker engine running on a remote server over ssh.

[bvinayprasad]

I'm seeing this error when running docker-compose with -H option
docker.errors.DockerException: Invalid bind address protocol: ssh://alpha@192.171.123.1

[amovfx]

Same problems here

[mmoya]

In the meantime, this workaround works for me:

1. Launch socat in a terminal:

   ssh -t -L 8888:127.0.0.1:8888 remote-docker-host 'socat TCP4-LISTEN:8888,fork,bind=127.0.0.1 UNIX-CONNECT:/var/run/docker.sock'
   

2. Point compose to local port 8888

   docker-compose -H tcp://:8888 up
   

[cde0]

Why this issue is close i don't think is fix.

[KatSick]

Yep. Looks like docker-compose does not use ~/.ssh/config when using DOCKER_HOST variable, while docker does

[KatSick]

This workaround helps #6513 (comment)

[hazcod]

Seeing this problem as wel with a super simple deploy over SSH.

[eliyahus]

Same issue. docker commands work fine with the SSH connections, while docker-compose commands don't.

Traceback (most recent call last):
File "docker-compose", line 6, in
File "compose\cli\main.py", line 72, in main
File "compose\cli\main.py", line 125, in perform_command
File "compose\cli\command.py", line 76, in project_from_options
File "compose\cli\command.py", line 142, in get_project
File "compose\cli\docker_client.py", line 47, in get_client
File "compose\cli\docker_client.py", line 174, in docker_client
File "site-packages\docker\api\client.py", line 166, in init
File "site-packages\docker\transport\sshconn.py", line 111, in init
File "site-packages\docker\transport\sshconn.py", line 119, in _connect
File "site-packages\paramiko\client.py", line 446, in connect
File "site-packages\paramiko\client.py", line 765, in _auth
paramiko.ssh_exception.SSHException: No authentication methods available
[14836] Failed to execute script docker-compose

Update: eventually, I solved this one using this tip: #6655 (comment)

[hazcod]

Interestingly things work fine with docker stack deploy -c compose.yml since that doesn't use the Python library, altough that requires a Swarm or Kubernetes host.

[snwfdhmp]

I'm having the same issue, cannot deploy over ssh

[mmenapace]

same problem here.
On local context everything works like a charm.
When I try to launch the docker-compose on a remote host via ssh the docker engine on the stops and restarts the services but doesn't update it.
command "trace"

• created context "remote" with docker endpoint ssh://[user]@[local ip]
• docker context use remote
• [rebuild solution] (gradle clean build) -> docker file for bootJar copy
• docker-compose up
  The service on remote starts but with an old version of the solution

[julianwagle]

still an issue:
(base) home@J django-on-docker % DOCKER_HOST=ssh://name@domain.com docker-compose -f docker-compose.staging.yml up -d --build
Traceback (most recent call last):
File "docker-compose", line 6, in
File "compose/cli/main.py", line 72, in main
File "compose/cli/main.py", line 125, in perform_command
File "compose/cli/command.py", line 76, in project_from_options
File "compose/cli/command.py", line 142, in get_project
File "compose/cli/docker_client.py", line 47, in get_client
File "compose/cli/docker_client.py", line 174, in docker_client
File "site-packages/docker/api/client.py", line 166, in init
File "site-packages/docker/transport/sshconn.py", line 111, in init
File "site-packages/docker/transport/sshconn.py", line 119, in _connect
File "site-packages/paramiko/client.py", line 349, in connect
File "site-packages/paramiko/util.py", line 283, in retry_on_signal
File "site-packages/paramiko/client.py", line 349, in
TimeoutError: [Errno 60] Operation timed out
[61197] Failed to execute script docker-compose

[carlosonunez]

I dug into this a bit today. Compose via SSH only works if you:

• Connect with a password (user:password@host), or
• Connect with a private key that is in your ssh-agent

Compose does not support the IdentityFile parameter in your .ssh/config. This is an easy fix that I made locally; I can submit a PR if there's enough interest.

[azsdaja]

@carlosonunez — sounds great, personally I'm much interested in a fix.

[snwfdhmp]

@carlosonunez thanks a lot ! Unfortunately that's a pain when using multiple SSH remote hosts.

[bdoublet91]

Any update ? Docker-compose works through docker context and ssh-agent but not with ssh_config

[konzinov]

Maybe this can help

[bdoublet91]

docker-compose v1.28.2 fix ssh remote context for me

docker-compose version
docker-compose version 1.28.2, build unknown
docker-py version: 4.4.1
CPython version: 3.6.9
OpenSSL version: OpenSSL 1.1.1  11 Sep 2018

docker context ls
int-swarmanager *                                             ssh://toto@int_swarmanager1

cat /etc/ssh/ssh_config

Host int_swarmanager1
    Hostname ****
    IdentityFile /etc/ssh/toto_rsa
    IdentitiesOnly yes
    StrictHostKeyChecking no

Before version 1.28.2, I had to create docker context with ip no hostname and use ssh-agent to store the ssh key for the connection. Now docker-compose support IdentityFile in ssh_config :)

Just a little bit long with the remote ssh when executing docker-compose, like 5 seconds, can see that it's doing when activated verbose.

[brunomarquete]

docker-compose v1.28.2 fix ssh remote context for me

docker-compose version
docker-compose version 1.28.2, build unknown
docker-py version: 4.4.1
CPython version: 3.6.9
OpenSSL version: OpenSSL 1.1.1  11 Sep 2018

docker context ls
int-swarmanager *                                             ssh://toto@int_swarmanager1

cat /etc/ssh/ssh_config

Host int_swarmanager1
    Hostname ****
    IdentityFile /etc/ssh/toto_rsa
    IdentitiesOnly yes
    StrictHostKeyChecking no

Before version 1.28.2, I had to create docker context with ip no hostname and use ssh-agent to store the ssh key for the connection. Now docker-compose support IdentityFile in ssh_config :)

Just a little bit long with the remote ssh when executing docker-compose, like 5 seconds, can see that it's doing when activated verbose.

You saved me!

This solves the problem. Thank you very much!
I've been trying to solve it for days.

The problem only occurs in version 1.28.3 of the docker-compose, released on February 17, 2021. This explains why the problem came up suddenly.

The following installation is the solution:

• pip install docker-compose==1.28.2

Or if you have not previously installed rust:

• pip install docker-compose==1.28.2 cryptography==3.0

I noticed that using rust takes longer.

Thank you again!

[crshumate]

I noticed this issue using docker-compose v 1.28.5 in Travis-CI.

I downgraded to 1.27.4 which is what we use in Travis-CI for another project and it worked as expected.

[thomsa]

I spent two days on this but... I got it working on GitLab CI

eval $(ssh-agent -s)`
echo "$SSH_PRIVATE_KEY" >> ~/.ssh/id_ci
chmod 400 ~/.ssh/id_ci
ssh-add -k ~/.ssh/id_ci  
export COMPOSE_PARAMIKO_SSH=1`

Run the ssh-agent, add the SSH private key from a file to the agent with the -k flag.
And most importantly COMPOSE_PARAMIKO_SSH=1 this will use SSH client from compose version 1.27.x (i hope i remember correctly) and voila, it works.

compose.cli.docker_client.get_client: docker-compose version 1.29.1, build c34c88b2
docker-py version: 5.0.0
CPython version: 3.7.10
OpenSSL version: OpenSSL 1.1.0l  10 Sep 2019
compose.cli.docker_client.get_client: Docker base_url: http+docker://ssh

[kaangoksal]

I was using the snap version of docker, where the docker compose version is

$ docker-compose version

docker-compose version 1.25.5, build unknown
docker-py version: 4.4.4
CPython version: 3.6.9
OpenSSL version: OpenSSL 1.1.1  11 Sep 2018

this config is not working with docker-compose -H "ssh://ubuntu@10.0.0.194" ps

what I did to solve this issue was

pip3 install docker-compose

this installs the latest pip version of docker-compose, I think docker-compose is just bunch of py scripts... however it doesn't populate into your path since there is a snap version. In order to mitigate that you can just execute it from its path as follows:

/home/yourUsername/.local/bin/docker-compose -H "ssh://ubuntu@10.0.0.194" ps

I think snapcraft team will fix this soon, I saw the pull request

PR

[Ugoivy]

I have the same issue using Java and Groovy, I can't execute docker-compose commands. However, I can in Go using golang.org/x/crypto/ssh without problem and without changing my ssh configuration.

[gfiedler]

This works for me:

sudo pip3 install docker-compose==1.29.2
export COMPOSE_PARAMIKO_SSH=0