New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

docker-compose fails with ERROR: SSL error: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:661) #38

Closed
colindembovsky opened this Issue Mar 3, 2017 · 17 comments

Comments

Projects
None yet
5 participants
@colindembovsky

colindembovsky commented Mar 3, 2017

(Logging here as per @RoopeshNair on Microsoft/VSTS-Docker-Preview#14)

I have created a release with a several docker tasks. I have an endpoint to a docker host machine. I am able to run docker commands successfully (like docker run) against the docker host. However, the docker-compose up commands fail with ERROR: SSL error: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:661). (In the task I'm using the "Run Services" action).

I can repro this on my client machine by running docker-compose ps against the same host - I get the cert error. However, if I specify the tls settings explicitly like this:

docker-compose -H $dockerHost --tls --tlscacert ~/.docker/ca.pem --tlscert ~/.docker/cert.pem --tlskey ~/.docker/key.pem ps

then the command succeeds. Is there some issue with the environment variables that the task is using to run docker-compose up?

@logankoester

This comment has been minimized.

Show comment
Hide comment
@logankoester

logankoester Mar 4, 2017

I'm seeing the same thing.

logankoester commented Mar 4, 2017

I'm seeing the same thing.

@jitekuma

This comment has been minimized.

Show comment
Hide comment
@jitekuma

jitekuma Mar 6, 2017

It seems that the certificates used in Docker Container Host Endpoint and the one saved in ~/.docker/ folder are different.

The certificate stored in Docker Container Host Endpoint is saved in ~/.dockercerts/ by task and used for execution. Can you verify if these certificates are same.

~/.dockercerts/ gets deleted once the task execution completes. You can take the certificate while task is executing to validate.

jitekuma commented Mar 6, 2017

It seems that the certificates used in Docker Container Host Endpoint and the one saved in ~/.docker/ folder are different.

The certificate stored in Docker Container Host Endpoint is saved in ~/.dockercerts/ by task and used for execution. Can you verify if these certificates are same.

~/.dockercerts/ gets deleted once the task execution completes. You can take the certificate while task is executing to validate.

@colindembovsky

This comment has been minimized.

Show comment
Hide comment
@colindembovsky

colindembovsky Mar 6, 2017

Hi @jitekuma - I only have one set of certs. And I can't see how they can be different because docker commands work using the endpoint. It's just the docker-compose command that fails. I suspect that the openssl on my Windows Bash and in the VSTS agent container is old - I'm going to try to update it to see if that resolves the issue.

colindembovsky commented Mar 6, 2017

Hi @jitekuma - I only have one set of certs. And I can't see how they can be different because docker commands work using the endpoint. It's just the docker-compose command that fails. I suspect that the openssl on my Windows Bash and in the VSTS agent container is old - I'm going to try to update it to see if that resolves the issue.

@SemionPar

This comment has been minimized.

Show comment
Hide comment
@SemionPar

SemionPar Mar 7, 2017

I am having the same issue.

DOCKER_TLS_VERIFY = "1"
DOCKER_HOST = "tcp://some.host:2376"
DOCKER_CERT_PATH = "/home/xyz/.docker"

docker --tlsverify ps executes just fine, while docker-compose --tlsverify up -d --force-recreate gives me an error:
SSL error: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:661)

SemionPar commented Mar 7, 2017

I am having the same issue.

DOCKER_TLS_VERIFY = "1"
DOCKER_HOST = "tcp://some.host:2376"
DOCKER_CERT_PATH = "/home/xyz/.docker"

docker --tlsverify ps executes just fine, while docker-compose --tlsverify up -d --force-recreate gives me an error:
SSL error: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:661)

@RoopeshNair

This comment has been minimized.

Show comment
Hide comment
@RoopeshNair

RoopeshNair Mar 7, 2017

Member
Member

RoopeshNair commented Mar 7, 2017

@colindembovsky

This comment has been minimized.

Show comment
Hide comment
@colindembovsky

colindembovsky Mar 7, 2017

I've tried to update python and install docker-compose via pip (since some other threads seem to resolve this issue that way) but that doesn't seem to work for me.

colindembovsky commented Mar 7, 2017

I've tried to update python and install docker-compose via pip (since some other threads seem to resolve this issue that way) but that doesn't seem to work for me.

@jitekuma

This comment has been minimized.

Show comment
Hide comment
@jitekuma

jitekuma Mar 8, 2017

@colindembovsky You dont need to pass --tls or --tlsverify option in the docker-config path as the task already sets DOCKER_TSL_VERIFY environment varaible. I debugged docker-compose and docker-py library and verified that if you pass any flag --tls or --tlsverify flag it tries to create tslConfig object out of options and not from environment and hence either ca_cert object or verify is none in TFSConfig file. You can use the task without using these flags.
Check the variable getting set here https://github.com/Microsoft/vsts-docker/blob/cdccda9323abe8e861d0891f03922d0e3f8fd8d1/src/tasks/dockerCompose/dockerComposeConnection.ts
Let me know if you need more details.

jitekuma commented Mar 8, 2017

@colindembovsky You dont need to pass --tls or --tlsverify option in the docker-config path as the task already sets DOCKER_TSL_VERIFY environment varaible. I debugged docker-compose and docker-py library and verified that if you pass any flag --tls or --tlsverify flag it tries to create tslConfig object out of options and not from environment and hence either ca_cert object or verify is none in TFSConfig file. You can use the task without using these flags.
Check the variable getting set here https://github.com/Microsoft/vsts-docker/blob/cdccda9323abe8e861d0891f03922d0e3f8fd8d1/src/tasks/dockerCompose/dockerComposeConnection.ts
Let me know if you need more details.

@colindembovsky

This comment has been minimized.

Show comment
Hide comment
@colindembovsky

colindembovsky Mar 8, 2017

@jitekuma I'm not setting the env variables during the build/release - I just noted that I can repro the task behavior if I do so locally.

I'm still not sure if this is an issue with the docker task or the Azure cli that created the certificates in the first place. Bottom line: if I pass the certs in I get a successful connection - when I use env variables, I get failures. Here's what happens when I run some docker commands from my machine:

Docker commands succeed when passing --tls:

colin@colinsurface31 ~> set vmName "cd-dockerhost"
colin@colinsurface31 ~> set location "westus"
colin@colinsurface31 ~> set dockerHost "tcp://$vmName.$location.cloudapp.azure.com:2376"
colin@colinsurface31 ~> docker -H $dockerHost --tls ps
CONTAINER ID        IMAGE                                             COMMAND             CREATED             STATUS              PORTS               NAMES
90497ba28b92        microsoft/vsts-agent:ubuntu-16.04-docker-1.11.2   "./start.sh"        4 days ago          Up 4 days                               jovial_nightingale

Docker commands fail when using env vars:

colin@colinsurface31 ~> set -x DOCKER_HOST $dockerHost
colin@colinsurface31 ~> set -x DOCKER_TLS_VERIFY 1
colin@colinsurface31 ~> docker ps
FATA[0000] An error occurred trying to connect: Get https://cd-dockerhost.westus.cloudapp.azure.com:2376/v1.18/containers/json: x509: certificate is valid for *, not cd-dockerhost.westus.cloudapp.azure.com

Docker-compose succeeds when passing in certs:

colin@colinsurface31 ~> docker-compose -H $dockerHost --tls --tlscacert ~/.docker/ca.pem --tlscert ~/.docker/cert.pem --tlskey ~/.docker/key.pem ps
/usr/local/lib/python2.7/dist-packages/requests/packages/urllib3/connectionpool.py:838: InsecureRequestWarning: Unverified HTTPS request is being made. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/security.html
  InsecureRequestWarning)
/usr/local/lib/python2.7/dist-packages/requests/packages/urllib3/connectionpool.py:838: InsecureRequestWarning: Unverified HTTPS request is being made. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/security.html
  InsecureRequestWarning)
Name   Command   State   Ports
------------------------------

Docker-compose fails when using env vars:

colin@colinsurface31 ~> set -x DOCKER_TLS_VERIFY 1
colin@colinsurface31 ~> set -x DOCKER_HOST $dockerHost
colin@colinsurface31 ~> docker-compose ps
ERROR: SSL error: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:590)

colindembovsky commented Mar 8, 2017

@jitekuma I'm not setting the env variables during the build/release - I just noted that I can repro the task behavior if I do so locally.

I'm still not sure if this is an issue with the docker task or the Azure cli that created the certificates in the first place. Bottom line: if I pass the certs in I get a successful connection - when I use env variables, I get failures. Here's what happens when I run some docker commands from my machine:

Docker commands succeed when passing --tls:

colin@colinsurface31 ~> set vmName "cd-dockerhost"
colin@colinsurface31 ~> set location "westus"
colin@colinsurface31 ~> set dockerHost "tcp://$vmName.$location.cloudapp.azure.com:2376"
colin@colinsurface31 ~> docker -H $dockerHost --tls ps
CONTAINER ID        IMAGE                                             COMMAND             CREATED             STATUS              PORTS               NAMES
90497ba28b92        microsoft/vsts-agent:ubuntu-16.04-docker-1.11.2   "./start.sh"        4 days ago          Up 4 days                               jovial_nightingale

Docker commands fail when using env vars:

colin@colinsurface31 ~> set -x DOCKER_HOST $dockerHost
colin@colinsurface31 ~> set -x DOCKER_TLS_VERIFY 1
colin@colinsurface31 ~> docker ps
FATA[0000] An error occurred trying to connect: Get https://cd-dockerhost.westus.cloudapp.azure.com:2376/v1.18/containers/json: x509: certificate is valid for *, not cd-dockerhost.westus.cloudapp.azure.com

Docker-compose succeeds when passing in certs:

colin@colinsurface31 ~> docker-compose -H $dockerHost --tls --tlscacert ~/.docker/ca.pem --tlscert ~/.docker/cert.pem --tlskey ~/.docker/key.pem ps
/usr/local/lib/python2.7/dist-packages/requests/packages/urllib3/connectionpool.py:838: InsecureRequestWarning: Unverified HTTPS request is being made. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/security.html
  InsecureRequestWarning)
/usr/local/lib/python2.7/dist-packages/requests/packages/urllib3/connectionpool.py:838: InsecureRequestWarning: Unverified HTTPS request is being made. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/security.html
  InsecureRequestWarning)
Name   Command   State   Ports
------------------------------

Docker-compose fails when using env vars:

colin@colinsurface31 ~> set -x DOCKER_TLS_VERIFY 1
colin@colinsurface31 ~> set -x DOCKER_HOST $dockerHost
colin@colinsurface31 ~> docker-compose ps
ERROR: SSL error: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:590)
@jitekuma

This comment has been minimized.

Show comment
Hide comment
@jitekuma

jitekuma Mar 8, 2017

@colindembovsky You also need to set DOCKER_CERT_PATH environment variable.

DOCKER_TLS_VERIFY = "1"
DOCKER_HOST = $dockerHost
DOCKER_CERT_PATH = "~/.docker/"

then simply do docker-compose ps this will work

or specify all the tls parameter like

docker-compose -H $dockerHost --tls --tlscacert ~/.docker/ca.pem --tlscert ~/.docker/cert.pem --tlskey ~/.docker/key.pem ps

jitekuma commented Mar 8, 2017

@colindembovsky You also need to set DOCKER_CERT_PATH environment variable.

DOCKER_TLS_VERIFY = "1"
DOCKER_HOST = $dockerHost
DOCKER_CERT_PATH = "~/.docker/"

then simply do docker-compose ps this will work

or specify all the tls parameter like

docker-compose -H $dockerHost --tls --tlscacert ~/.docker/ca.pem --tlscert ~/.docker/cert.pem --tlskey ~/.docker/key.pem ps

@colindembovsky

This comment has been minimized.

Show comment
Hide comment
@colindembovsky

colindembovsky Mar 8, 2017

@jitekuma - the default is correct in my case (~/.docker). But even if I set that I get the same behavior:

colin@colinsurface31 ~> set vmName "cd-dockerhost"
colin@colinsurface31 ~> set location "westus"
colin@colinsurface31 ~> set dockerHost "tcp://$vmName.$location.cloudapp.azure.com:2376"
colin@colinsurface31 ~> set -x DOCKER_HOST $dockerHost
colin@colinsurface31 ~> set -x DOCKER_TLS_VERIFY 1
colin@colinsurface31 ~> set -x DOCKER_CERT_PATH ~/.docker
colin@colinsurface31 ~> docker ps
FATA[0000] An error occurred trying to connect: Get https://cd-dockerhost.westus.cloudapp.azure.com:2376/v1.18/containers/json: x509: certificate is valid for *, not cd-dockerhost.westus.cloudapp.azure.com
colin@colinsurface31 ~> docker-compose ps
ERROR: SSL error: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:590)

I think my point is that I think the task should be passing the varialbes into the command rather than relying on the env vars - especially given that passing the certs in works reliably while using the env settings appears to be flaky for some reason.

colindembovsky commented Mar 8, 2017

@jitekuma - the default is correct in my case (~/.docker). But even if I set that I get the same behavior:

colin@colinsurface31 ~> set vmName "cd-dockerhost"
colin@colinsurface31 ~> set location "westus"
colin@colinsurface31 ~> set dockerHost "tcp://$vmName.$location.cloudapp.azure.com:2376"
colin@colinsurface31 ~> set -x DOCKER_HOST $dockerHost
colin@colinsurface31 ~> set -x DOCKER_TLS_VERIFY 1
colin@colinsurface31 ~> set -x DOCKER_CERT_PATH ~/.docker
colin@colinsurface31 ~> docker ps
FATA[0000] An error occurred trying to connect: Get https://cd-dockerhost.westus.cloudapp.azure.com:2376/v1.18/containers/json: x509: certificate is valid for *, not cd-dockerhost.westus.cloudapp.azure.com
colin@colinsurface31 ~> docker-compose ps
ERROR: SSL error: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:590)

I think my point is that I think the task should be passing the varialbes into the command rather than relying on the env vars - especially given that passing the certs in works reliably while using the env settings appears to be flaky for some reason.

@jitekuma

This comment has been minimized.

Show comment
Hide comment
@jitekuma

jitekuma Mar 8, 2017

@colindembovsky now i think this is a certificate issue can you put --tlsverify option in docker-compose

docker-compose -H $dockerHost --tls --tlsverify --tlscacert ~/.docker/ca.pem --tlscert ~/.docker/cert.pem --tlskey ~/.docker/key.pem ps

I think this should also fail. Let me know the output.

jitekuma commented Mar 8, 2017

@colindembovsky now i think this is a certificate issue can you put --tlsverify option in docker-compose

docker-compose -H $dockerHost --tls --tlsverify --tlscacert ~/.docker/ca.pem --tlscert ~/.docker/cert.pem --tlskey ~/.docker/key.pem ps

I think this should also fail. Let me know the output.

@colindembovsky

This comment has been minimized.

Show comment
Hide comment
@colindembovsky

colindembovsky Mar 8, 2017

@jitekuma Yes! It failed indeed (same as the error with docker):

colin@colinsurface31 ~> docker-compose -H $dockerHost --tls --tlscacert ~/.docker/ca.pem --tlscert ~/.docker/cert.pem --tlskey ~/.docker/key.pem --tlsverify ps
ERROR: SSL error: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:590)

The thing is that I didn't create the certs by hand - I used azure vm docker create which created the certs. Is there an issue with that process??

colindembovsky commented Mar 8, 2017

@jitekuma Yes! It failed indeed (same as the error with docker):

colin@colinsurface31 ~> docker-compose -H $dockerHost --tls --tlscacert ~/.docker/ca.pem --tlscert ~/.docker/cert.pem --tlskey ~/.docker/key.pem --tlsverify ps
ERROR: SSL error: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:590)

The thing is that I didn't create the certs by hand - I used azure vm docker create which created the certs. Is there an issue with that process??

@jitekuma

This comment has been minimized.

Show comment
Hide comment
@jitekuma

jitekuma Mar 8, 2017

No. I dont think so. Might be due to some command ran during development messed up with you certificate.

jitekuma commented Mar 8, 2017

No. I dont think so. Might be due to some command ran during development messed up with you certificate.

@colindembovsky

This comment has been minimized.

Show comment
Hide comment
@colindembovsky

colindembovsky Mar 8, 2017

Hmm I didn't do anything after creating the dockerHost. I think this might be the issue:

An error occurred trying to connect: Get https://cd-dockerhost.westus.cloudapp.azure.com:2376/v1.18/containers/json: x509: certificate is valid for *, not cd-dockerhost.westus.cloudapp.azure.com

I'm going to delete this host, clean the certs folder and run the azure vm docker create command again to make sure everything is squeaky clean. Then I'll see if I get the same behavior.

colindembovsky commented Mar 8, 2017

Hmm I didn't do anything after creating the dockerHost. I think this might be the issue:

An error occurred trying to connect: Get https://cd-dockerhost.westus.cloudapp.azure.com:2376/v1.18/containers/json: x509: certificate is valid for *, not cd-dockerhost.westus.cloudapp.azure.com

I'm going to delete this host, clean the certs folder and run the azure vm docker create command again to make sure everything is squeaky clean. Then I'll see if I get the same behavior.

@jitekuma

This comment has been minimized.

Show comment
Hide comment
@jitekuma

jitekuma Mar 8, 2017

@SemionPar in docker-compose command you can not mix environment variable and command option. You can specify setting in env variable and then just use docker-compose ps. The connection will be secured with TLS protocol if DOCKER_TLS_VERIFY variable is set.

jitekuma commented Mar 8, 2017

@SemionPar in docker-compose command you can not mix environment variable and command option. You can specify setting in env variable and then just use docker-compose ps. The connection will be secured with TLS protocol if DOCKER_TLS_VERIFY variable is set.

@jitekuma

This comment has been minimized.

Show comment
Hide comment
@jitekuma

jitekuma Mar 8, 2017

@colindembovsky also please check the path where certificates are created in the log, usually its different than ~/docker .

jitekuma commented Mar 8, 2017

@colindembovsky also please check the path where certificates are created in the log, usually its different than ~/docker .

@colindembovsky

This comment has been minimized.

Show comment
Hide comment
@colindembovsky

colindembovsky Mar 9, 2017

I figured out what the problem was - I used azure vm docker create to create the dockerhost. By default the CN for the generated certificates is "*". I just set it to hostname.location.cloudapp.azure.com using --docker-cert-cn "cd-dockerhost.westus.cloudapp.azure.com" when calling create and now I can connect without cert errors.

colindembovsky commented Mar 9, 2017

I figured out what the problem was - I used azure vm docker create to create the dockerhost. By default the CN for the generated certificates is "*". I just set it to hostname.location.cloudapp.azure.com using --docker-cert-cn "cd-dockerhost.westus.cloudapp.azure.com" when calling create and now I can connect without cert errors.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment