Custom SSH Credentials

[dnut]

I can't rely on an ssh agent to provide a private key. Sometimes I need to provide a password, sometimes I need to manually load the key file. But the docker-py code only connects with this in SSHHTTPAdapter:

self.ssh_client.connect(
    parsed.hostname, parsed.port, parsed.username,
)

There is no flexibility here. How am I supposed to connect with a custom key or password? I thought I might hack my way in by reassigning APIClient._custom_adapter to my own subclass of SSHHTTPAdapter, but then I realized the APIClient.__init__ is a huge mess that does way too much. That method would always raise an exception so I would also have to totally reimplement that method in a subclass. This is too much maintenance overhead for my deployment script that I would like to keep as simple as possible.

It should be exposed in SSHHTTPAdapter. I might even recommend doing both of these:

1. Add optional key and password parameters to SSHHTTPAdapter so you don't have to write your own subclass of it just to use custom credentials
2. Add optional ssh_client parameter to SSHHTTPAdapter to be used instead of instantiating one in __init__.

It should also be exposed somehow in APIClient.__init__, for example:

1. Expose password and private key with...
   a. Explicit parameters (a bit messy considering this is not an SSH specific class)
   b. General purpose configuration object (dict or better yet custom configuration class)
2. Enable programmer to provide their own SSHClient to APIClient. This is not ideal for the same reason as 1a.
3. Enable programmer to provide their own instance of a BaseHTTPAdapter to be used as self._custom_adapter. I think this is the best solution but there are some open questions. Like what's the best way to handle all the usages of base_url when it is no longer required to establish a connection.

I'm happy to take the lead on this and submit a PR, but I would like to get some feedback first.

[shin-]

Hi!

You can already provide a custom password through the base_url parameter, e.g. ssh://user:mypassword@host.com:22. For more advanced uses, you should be able to write your own Adapter subclass and pass it to the APIClient through mount() (see the requests doc on that topic). For example,

ssh_adapter = MySSHAdapter(url, key_path)
client = APIClient(base_url='ssh://bogus:22')
client.mount('http+docker://ssh', ssh_adapter)

I think eventually we may want to have the option of passing a private key to the Client instantiation in some form (TBD - maybe something similar to the TLSConfig), but anything beyond that, like first-class adapter modularity, would probably be beyond the scope of what we're trying to do with the library.

Hope that helps!

[dnut]

Hey, thanks for your reply, but there are some issues with your recommendations.

first-class adapter modularity

What does this mean?

Password in URL

You can already provide a custom password through the base_url parameter, e.g. ssh://user:mypassword@host.com:22

This is not right. I get the following exception:

  File "docker/client.py", line 40, in __init__
    self.api = APIClient(*args, **kwargs)
  File "docker/api/client.py", line 133, in __init__
    base_url, IS_WINDOWS_PLATFORM, tls=bool(tls)
  File "docker/utils/utils.py", line 265, in parse_host
    'Invalid bind address format: {}'.format(addr)
docker.errors.DockerException: Invalid bind address format: ssh://user:mypassword@host.com:22

Also as you can see in the source I quoted in my original comment, there is no way that the password would be used for the ssh connection, even if the url were valid.

Setting a Custom Adapter

So regardless of whether you want to use a password or a custom key file, you would definitely need to provide a custom Adapter class. And your recommended code to do so does not work because it will try to connect to a host called "bogus" on line two. If you leave out base_url, it changes it to http+unix and tries to connect to the local docker daemon. If there is no local docker daemon, it fails here.

There's no way to instantiate an APIClient without it trying to connect to a docker daemon. This is what I was talking about with __init__ doing too much and was the whole reason why I made this ticket.

A better design would be that APIClient does not even try to connect unless you explicitly call a method that needs a connection. There needs to be a way to provide a custom adapter or at least ssh credentials. APIClient.__init__ should really be broken up into several methods and SSHHTTPAdapter needs some changes too.

Does SSH work at all?

I actually tried using a hostname that my ssh agent can provide a key for and I got a valid SSH connection. But then I saw this error whenever I tried to do anything. It looks like it's treating the ssh connection like it can make GET requests directly over that connection which doesn't seem right.

  File "docker/api/container.py", line 210, in containers
    res = self._result(self._get(u, params=params), True)
  File "docker/utils/decorators.py", line 46, in inner
    return f(self, *args, **kwargs)
  File "docker/api/client.py", line 235, in _get
    return self.get(url, **self._set_request_timeout(kwargs))
  File "requests/sessions.py", line 546, in get
    return self.request('GET', url, **kwargs)
  File "requests/sessions.py", line 533, in request
    resp = self.send(prep, **send_kwargs)
  File "requests/sessions.py", line 646, in send
    r = adapter.send(request, **kwargs)
  File "requests/adapters.py", line 498, in send
    raise ConnectionError(err, request=request)
requests.exceptions.ConnectionError: ('Connection aborted.', BadStatusLine('No status line received - the server has closed the connection',))

[cmcga1125]

@dnut FYI I just got SSH to work to answer your question. It's not documented and i had to read through some of the source code but here are a few things i learned.

1. If you're wanting to use an ssh key, it has to be in the ~/.ssh/id_rsa location - or - /root/.ssh/id_rsa (if alpine)
2. If you don't have a known_hosts file you'll get a warning but the connection actually connects.
3. the syntax for the base_url is like mentioned above - which as a great help :)

[dnut]

@cmcga1125 I can get SSH to work on my desktop, but I need this to run on our build servers with custom credentials.

If you're wanting to use an ssh key, it has to be in the ~/.ssh/id_rsa location - or - /root/.ssh/id_rsa (if alpine)

This is a problem for me. I need to be able to specify any key file.

[cmcga1125]

@dnut - i'm running in a docker container - which allows me to map in the file, would that work?

[byt3bl33d3r]

Bump. I just went down the same rabbit hole as @dnut . Being able to specify the path to a SSH keyfile and/or the key material directly via an argument is very much needed for any automation when using this feature.

[rtubio]

Got the same issue here. I am using this sdk but, I had to shift to the CLI option since it was not clear to me whether authenticating with SSH credentials was an option for the sdk.

• In an ideal world, I would expect an ssh parameter to exist as a keyword to the build method; mimicking the usage of the docker command line utility to reduce the learning curve.

The answer provided by @cmcga1125 helps, but a slightly more elaborated step-by-step would really be super nice! :)

• Would it be possible at least to document the process to use an external SSH key?

[smyja]

This article says in the first paragraph, that password authentication is not possible when using a docker host for SSH. So a key gen must be used. Which makes The situation worse,the keygen is not on my pc. Its on my VPS, do I have to download it to my local pc?

[curlup]

Upvoting this. Need it in airflow and other stuff.

[testdoc2233]

Got the ssh config to work using the custom sshAdapter mentioned in one of the posts:
client = APIClient('ssh://bogus:22', use_ssh_client=True, version='1.41')
ssh_adapter = MySSHHTTPAdapter('ssh://anyuser@xx.xx.xx.xx:22', key_path='/path/to/id_rsa')
client.mount('http+docker://ssh', ssh_adapter)

This can also be modified to use password auth easily.

[devmaxde]

Hey,
I wrote a working Adapter based on this thread for you. Thank me later :)

import os
from docker import APIClient
from docker.transport import SSHHTTPAdapter

class MySSHHTTPAdapter(SSHHTTPAdapter):
    def _connect(self):
        if self.ssh_client:
            self.ssh_params["key_filename"] = os.environ.get("SSH_KEY_FILENAME")
            self.ssh_params["passphrase"] = os.environ.get("SSH_PASSPHRASE")
            self.ssh_client.connect(**self.ssh_params)


client = APIClient('ssh://ip:22', use_ssh_client=True, version='1.41')
ssh_adapter = MySSHHTTPAdapter('ssh://user@ip:22')
client.mount('http+docker://ssh', ssh_adapter)
print(client.version())

[devmaxde]

The Version="1.41" wont be necessary, if you implement it directly. Docker is trying to get the Version directly after configurating the SSHHTTPAdapter. Because we don't need to override it later anymore, we also don't need to disable the version checking :)

[synap5e]

Here's my monkeypatch to make this work with modern keys. I did this because I want to use the DockerClient rather than the ApiClient.

SSHHTTPAdapter_create_paramiko_client = SSHHTTPAdapter._create_paramiko_client
def SSHHTTPAdapter_patched_create_paramiko_client(self, *args, **kwargs):
    SSHHTTPAdapter_create_paramiko_client(self, *args, **kwargs)
    self.ssh_params["key_filename"] = str(Path("~/.ssh/id_ed25519").expanduser())
SSHHTTPAdapter._create_paramiko_client = SSHHTTPAdapter_patched_create_paramiko_client

I'm very surprised I need to do it this way, is there an option I'm missing?