Debugging pylxd.exceptions.ClientConnectionFailed?

[adosztal]

I'm trying to connect to an LXD cluster using the steps from this guide but I get a pylxd.exceptions.ClientConnectionFailed error:

>>> from pylxd import Client
>>> client = Client(endpoint='http://192.168.123.121:8443', cert=('lxd.crt', 'lxd.key'))
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/local/lib/python2.7/dist-packages/pylxd/client.py", line 290, in __init__
    raise exceptions.ClientConnectionFailed()
pylxd.exceptions.ClientConnectionFailed

The hosts are on the same subnet (.120 is my host, .121 is the LXD cluster member), ping works, port 8443 is open, the cert files are in place, I can get an answer back with wget:

host_120:~$ ping -q -c 5 192.168.123.121
PING 192.168.123.121 (192.168.123.121) 56(84) bytes of data.

--- 192.168.123.121 ping statistics ---
5 packets transmitted, 5 received, 0% packet loss, time 4081ms
rtt min/avg/max/mdev = 0.435/0.691/0.881/0.179 ms

host_121:~$ netstat -an | egrep "8443.*LISTEN"
tcp        0      0 192.168.123.121:8443    0.0.0.0:*               LISTEN     

host_120:~$ ls lxd.*
lxd.crt  lxd.csr  lxd.key

host_120:~$ wget http://192.168.123.121:8443/1.0
--2018-07-08 06:10:27--  http://192.168.123.121:8443/
Connecting to 192.168.123.121:8443... connected.
HTTP request sent, awaiting response... 200 No headers, assuming HTTP/0.9
Length: unspecified
Saving to: '1.0'

1.0                                 [ <=>                                                                        ]       7  --.-KB/s    in 0s      

2018-07-08 06:10:27 (339 KB/s) - '1.0' saved [7]

The saved file contains "1503 0100 0202 0a" in hex. I did a capture too: pylxd.pcap.zip

Versions:

• OS: Ubuntu 18.04 (Bionic) server
• Python: 2.7.15rc1,
• pylxd: 2.2.7
• LXD: 3.0.1

Is it something with pylxd or the LXD cluster? Is there a way I could get a detailed error message, not just the exception?

[ajkavanagh]

It could probably do with a better error message, but I'm pretty sure you have to use https://... rather than http://.. as your connection address, as LXD only supports SSL rather than unsecured transports. Ping back if that's not your issue, though! If so, we might add some parsing/checking to ensure that pyxld is passed an https://....

[adosztal]

I receive a valid response with wget (wget --no-check-certificate https://192.168.123.121:8443/1.0) but I get the same exception with pylxd.
Maybe something with the cert/key files? I used these commands to generate them:

openssl req -newkey rsa:2048 -nodes -keyout lxd.key -out lxd.csr
openssl x509 -signkey lxd.key -in lxd.csr -req -days 3650 -out lxd.crt

[ajkavanagh]

Did you try in pylxd with https. I'm sorry, from you comment, it's not clear to me.

[adosztal]

Sorry. Yes, I did:

>>> from pylxd import Client
>>> client = Client(endpoint='https://192.168.123.121:8443', cert=('lxd.crt', 'lxd.key'))
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/local/lib/python2.7/dist-packages/pylxd/client.py", line 290, in __init__
    raise exceptions.ClientConnectionFailed()
pylxd.exceptions.ClientConnectionFailed

Capture: pylxd_https.pcap.zip

[ajkavanagh]

Okay, let's eliminate pylxd from the party for a moment. If you add the https://192.168.123.121:8443 as a remote to the lxc client, does it work?

The syntax is:

lxc remote add <remote> <IP|FQDN|URL> [--accept-certificate] [--password=PASSWORD]

e.g. lxd remote add mycluster https://192.168.123.121:8443 --password=<your password>

And then see if you can do things with it: lxc list mycluster:

Also, which versions of lxd and pylxd are you using?

[adosztal]

That works.

host_120:~$ lxc remote add mycluster https://192.168.123.121:8443 --password=xxx
Generating a client certificate. This may take a minute...
Certificate fingerprint: xxxxxx
ok (y/n)? y
Client certificate stored at server:  mycluster

host_120:~$ lxc list mycluster:
+------+---------+------------------------+------+------------+-----------+----------+
| NAME |  STATE  |          IPV4          | IPV6 |    TYPE    | SNAPSHOTS | LOCATION |
+------+---------+------------------------+------+------------+-----------+----------+
| con1 | RUNNING | 192.168.123.219 (eth0) |      | PERSISTENT | 0         | lxd1     |
|      |         | 172.26.1.131 (eth1)    |      |            |           |          |
|      |         | 10.10.10.2 (eth2)      |      |            |           |          |
+------+---------+------------------------+------+------------+-----------+----------+
| con2 | STOPPED |                        |      | PERSISTENT | 0         | lxd2     |
+------+---------+------------------------+------+------------+-----------+----------+

Versions:

• OS: Ubuntu 18.04 (Bionic) server
• Python: 2.7.15rc1,
• pylxd: 2.2.7
• LXD: 3.0.1

[ajkavanagh]

Okay, that's good.

So, that certificate that got generated is at $HOME/.config/lxc/client.key and $HOME/.config/lxc/client.crt.

You should now be able to connect in pylxd by just doing:

client = Client(endpoint='http://192.168.123.121:8443')

The problem, I suspect, is that the original lxd.key / lxd.crt weren't being found. So if the above does work then we probably need some checks in pylxd about finding certs, etc. as the code in question doesn't seem to do that sanity check.

[adosztal]

Same issue, I checked both with http and https:

>>> from pylxd import Client
>>> client = Client(endpoint='http://192.168.123.121:8443')
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/adosztal/.local/lib/python2.7/site-packages/pylxd/client.py", line 290, in __init__
    raise exceptions.ClientConnectionFailed()
pylxd.exceptions.ClientConnectionFailed

>>> client = Client(endpoint='https://192.168.123.121:8443')
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/adosztal/.local/lib/python2.7/site-packages/pylxd/client.py", line 290, in __init__
    raise exceptions.ClientConnectionFailed()
pylxd.exceptions.ClientConnectionFailed

I tried using absolute path, same result:

>>> client = Client(endpoint='https://192.168.123.121:8443', cert=('/home/adosztal/.config/lxc/client.crt', '/home/adosztal/.config/lxc/client.key'))
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/adosztal/.local/lib/python2.7/site-packages/pylxd/client.py", line 290, in __init__
    raise exceptions.ClientConnectionFailed()
pylxd.exceptions.ClientConnectionFailed

[ajkavanagh]

Oh, that's a bit odd. Okay, let's dig a bit deeper. If you do:

import logging
logging.basicConfig(level=logging.DEBUG)

before you do the client connect, we might get some more useful debugging information.

[adosztal]

The only additional info is that urllib3 starts the https connection:

>>> import logging
>>> logging.basicConfig(level=logging.DEBUG)
>>> from pylxd import Client
>>> client = Client(endpoint='https://192.168.123.121:8443', cert=('/home/adosztal/.config/lxc/client.crt', '/home/adosztal/.config/lxc/client.key'))
DEBUG:urllib3.connectionpool:Starting new HTTPS connection (1): 192.168.123.121:8443
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/adosztal/.local/lib/python2.7/site-packages/pylxd/client.py", line 290, in __init__
    raise exceptions.ClientConnectionFailed()
pylxd.exceptions.ClientConnectionFailed

[adosztal]

I can see this message in syslog:
Jul 8 09:44:00 lxd1 lxd[802]: 2018/07/08 09:44:00 http: TLS handshake error from 192.168.123.120:49040: remote error: tls: unknown certificate authority

Strange because I was using the cert generated by lxc remote add

[ajkavanagh]

Okay, how good are you with Python? Because the only thing I can suggest now, is to go into "/home/adosztal/.local/lib/python2.7/site-packages/pylxd/client.py" at and add a print() into the code before the exception is raised (at line 190) and print out the response. This would let us see what is coming back from the server "raw" and then work out what to look at next.

[ajkavanagh]

Ah, re: the remote error issue. Sorry, if you add , verify=False to the connection command, it will not verify the certificate and the connection should be made. You'll get an Insecure warning though.

[adosztal]

That indeed helped, it works now. Thanks a lot!
Maybe I'll submit a PR to include this tip in doc/source/authentication.rst; what do you think?

[ajkavanagh]

@adosztal excellent! yes, we definitely need to beef up the documentation around how this works, so please do feel free to submit a PR. For example you can pass the ca bundle (as a string) to the verify parameter, but that's only documented on the docstring. It would be good to have some examples/gotchas around this area. For a very long time, pylxd was only used with the unix socket, so some of this isn't as well tested in the real world.

[vanushwashere]

Hi, we are randomly getting Terminated websocket exception when trying to do several container.exec() one by one, while doing several other task on another container in the background using celery.
pylxd connects via TCP to remote machine.
Can you help with debug ?

[ajkavanagh]

@vanushwashere can you please provide details of versions of pylxd, lxd and OS? Thanks.

[vanushwashere]

Also I'm getting Got error getting next reader websocket: close 1000 , &{%!s(*os.file=&{21 slave <nil>})} in debug :/

[vanushwashere]

OS: Ubuntu Server 16.04
pylxd version: 2.2.7
lxd versino: 2.0.11

Also I found this issue over the internet:
Issue on lxd
I think this one is related

[ajkavanagh]

@vanushwashere could you re-try with master branch please? There's been a few commits in that area and it might have been fixed.

[vanushwashere]

@ajkavanagh I did that, no problems so far 👍
When do you plan roll out next release ?

[ajkavanagh]

@vanushwashere I'm kind of waiting on a few PRs to land; just trying to get their tests squared away. However, due to the time its unfortunately taking, I'm thinking of releasing next week as this issue is obviously causing issues.

[vanushwashere]

That'll be great 😎

[ajkavanagh]

I'm closing this as a new version has been released and I think it solves the error. Please could you comment if it hasn't and I'll re-open the issue. Thanks.

[Commander-Firestrike]

I am having the exact same issue as was originally stated in this thread:
>>> client = Client(endpoint="https://[ip]:8443", cert=("lxd.crt", "lxd.key"), verify=False)
is the only thing that works, but it gives InsecureRequestWarning for almost every action.

[vanushwashere]

Hi @Commander-Firestrike , we disabled warnings like this

import urllib3
urllib3.disable_warnings(urllib3.exceptions.InsecureRequestWarning)