SSL verification on python2.7 failing #36

sebgoa · 2016-11-23T10:03:49Z

I saw a commit flying by that said this was fixed but I get:

>>> import os
>>> config.load_kube_config(os.environ["HOME"] + '/.kube/config')
>>> v1=client.CoreV1Api()
>>> ret = v1.list_pod_for_all_namespaces(watch=False)
2016-11-23 10:26:29,195 ERROR Certificate did not match expected hostname: 192.168.99.100. Certificate: {'subjectAltName': (('DNS', 'kubernetes.default.svc.cluster.local'), ('DNS', 'kubernetes.default.svc'), ('DNS', 'kubernetes.default'), ('DNS', 'kubernetes'), ('IP Address', '192.168.99.100'), ('IP Address', '10.0.0.1')), 'notBefore': u'Nov 16 09:38:49 2016 GMT', 'serialNumber': u'02', 'notAfter': 'Nov 16 09:38:49 2017 GMT', 'version': 3L, 'subject': ((('commonName', u'minikube'),),), 'issuer': ((('commonName', u'minikubeCA'),),)}
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "kubernetes/client/apis/core_v1_api.py", line 14377, in list_pod_for_all_namespaces
    (data) = self.list_pod_for_all_namespaces_with_http_info(**kwargs)
  File "kubernetes/client/apis/core_v1_api.py", line 14475, in list_pod_for_all_namespaces_with_http_info
    collection_formats=collection_formats)
  File "kubernetes/client/api_client.py", line 326, in call_api
    _return_http_data_only, collection_formats, _preload_content, _request_timeout)
  File "kubernetes/client/api_client.py", line 150, in __call_api
    _request_timeout=_request_timeout)
  File "kubernetes/client/api_client.py", line 349, in request
    headers=headers)
  File "kubernetes/client/rest.py", line 222, in GET
    query_params=query_params)
  File "kubernetes/client/rest.py", line 199, in request
    raise ApiException(status=0, reason=msg)
kubernetes.client.rest.ApiException: (0)
Reason: SSLError
hostname '192.168.99.100' doesn't match either of 'kubernetes.default.svc.cluster.local', 'kubernetes.default.svc', 'kubernetes.default', 'kubernetes', '192.168.99.100', '10.0.0.1'

The text was updated successfully, but these errors were encountered:

sebgoa · 2016-11-23T16:05:29Z

it does work with python3.5

mbohlool · 2016-11-23T16:23:10Z

are you using latest urllib3? can you run setup.py install and try this again or try to manually update urllib3.

pip show urllib3
Name: urllib3
Version: 1.19.1
Summary: HTTP library with thread-safe connection pooling, file post, and more.

Also what is your python version?

$ python --version
Python 2.7.12

mbohlool · 2016-11-23T21:38:36Z

I figured this out. We had a patch that we removed because urllib3 is already doing the patch, however it only does that if package "ipaddress" is installed. I added that as a dependency so we should be good.

mbohlool · 2016-11-23T21:46:49Z

More concrete reason here: https://github.com/shazow/urllib3/blob/master/urllib3/packages/ssl_match_hostname/_implementation.py#L9

sebgoa · 2016-11-24T18:02:32Z

ok confirmed that it solved the issue.

madhavmalhotra3089 · 2017-05-28T06:07:41Z

Still not working for python 2.x windows 10

mbohlool · 2017-05-28T16:09:59Z

https://github.com/kubernetes-incubator/client-python/blob/master/README.md#hostname-doesnt-match

madhavmalhotra3089 · 2017-05-28T20:12:40Z

Hey Thanks

On Sun, May 28, 2017 at 9:10 AM Mehdy Bohlool ***@***.***> wrote: https://github.com/kubernetes-incubator/client-python/blob/master/README.md#hostname-doesnt-match — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#36 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AQel4SxEv6OCtfDYGMCYATi7E3UscAISks5r-ZxYgaJpZM4K6ZZl> .

-- Madhav Malhotra Graduate Student University of Victoria Canada +1-778-922-5630

tahir24434 · 2017-07-12T18:20:14Z

Hi,
I followed the instructions described in this thread and I am still hitting the issue.

Exception when calling CoreV1Api->connect_post_namespaced_pod_exec: (0)
Reason: hostname '1.0.0.14' doesn't match either of 'minion-1-0-0-14', 'kubernetes', 'kubernetes.default', 'kubernetes.default.svc', 'kubernetes.default.svc.cluster.local'

I have following versions of packages

python --version
Python 2.7.6

pip show ipaddress
Name: ipaddress
Version: 1.0.18
Summary: IPv4/IPv6 manipulation library
Home-page: https://github.com/phihag/ipaddress
Author: Philipp Hagemeister
Author-email: phihag@phihag.de
License: Python Software Foundation License
Location: /usr/local/lib/python2.7/dist-packages
Requires:

pip show urllib3
Name: urllib3
Version: 1.20
Summary: HTTP library with thread-safe connection pooling, file post, and more.
Home-page: https://urllib3.readthedocs.io/
Author: Andrey Petrov
Author-email: andrey.petrov@shazow.net
License: MIT
Location: /usr/local/lib/python2.7/dist-packages
Requires:

Any ideas please?
Regards

mbohlool · 2017-07-12T20:13:45Z

What is your kubernetes package version? what framework this is on? (os, anaconda?, etc.). Also please confirm that kubectl works.

flavianmissi · 2017-07-28T12:18:33Z

Same problem here:

$ python --version
Python 2.7.13

$ pip show ipaddress
Name: ipaddress
Version: 1.0.18
Summary: IPv4/IPv6 manipulation library
Home-page: https://github.com/phihag/ipaddress
Author: Philipp Hagemeister
Author-email: phihag@phihag.de
License: Python Software Foundation License
Location: $HOME/my-venv/lib/python2.7/site-packages
Requires:

$ pip show urllib3
Name: urllib3
Version: 1.22
Summary: HTTP library with thread-safe connection pooling, file post, and more.
Home-page: https://urllib3.readthedocs.io/
Author: Andrey Petrov
Author-email: andrey.petrov@shazow.net
License: MIT
Location: $HOME/my-venv/lib/python2.7/site-packages
Requires:

$ pip show kubernetes
Name: kubernetes
Version: 2.0.0
Summary: Kubernetes python client
Home-page: https://github.com/kubernetes-incubator/client-python
Author: Kubernetes
Author-email: UNKNOWN
License: Apache License Version 2.0
Location: /Users/flavia/go/src/mindoktor.io/md-venv/lib/python2.7/site-packages
Requires: websocket-client, ipaddress, setuptools, oauth2client, six, urllib3, pyyaml, python-dateutil, certifi

The actual error looks a little different, though the underlying problem seems exactly the same

c.connect_get_namespaced_pod_exec("podname", "namespace", command="/bin/bash -c 'ls'")
[...]
ApiException: (0)
Reason: hostname 'MY.IPADDR' doesn't match either of 'kubernetes', 'kubernetes.default', 'kubernetes.default.svc', 'kubernetes.default.svc.cluster.local'

Additionally, kubectl works just fine.

mbohlool · 2017-07-28T21:23:09Z

You are calling exec. We use websocket library for exec. They introduced some bug and we limited the version in the dependency file (requirements.txt). If you search issues for websocket or just make sure you are using compatible websocket version, that should fix your problem.

flavianmissi · 2017-07-31T07:17:18Z

Thanks for the info! I have noticed that.
With websocket-client==0.44.0 I get

ApiException: (0)
Reason: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:661)

The error I posted previously I got using websocket-client==0.40.0 (sorry I missed posting the info for it). Looking at some more examples I found that I needed to stop hostname assertion (doh):

from kubernetes.client import configuration
configuration.assert_hostname = True

After doing that I was able to move forward a bit, but not without another failure. Here's my full code:

from kubernetes.client import configuration
from kubernetes import config
from kubernetes.client.apis import core_v1_api

configuration.assert_hostname = False
config.load_kube_config()
c = core_v1_api.CoreV1Api()
c.connect_get_namespaced_pod_exec("mypod", "namespace", command="/bin/bash -c 'ls'")
# stacktrace...
# ApiException: (0)
# Reason: Handshake status 404

I feel like I'm missing something, appreciate the help!

flavianmissi · 2017-07-31T13:41:38Z

BTW I have looked at #144 and updated my snippet (as seen above), and I get the exact same issue there... I have tested on my local machine (OS X, OpenSSL 1.0.2l) and on a VM (Debian OpenSSL 1.0.1t ) both share same requirements versions.

Have also done some debugging and the final url used in the websocket call is wss://my.master.ip/api/v1/namespaces/default/pods/mypod/exec. I've also made sure that mypod actually exists... although I know that 404 is a client error I'm clueless to why this is happening.

mbohlool · 2017-07-31T13:57:56Z

Interesting. Try kubectl (to call exec on your pod) with -v9 flag to see if it hits the same url. Also what is your cluster running on? GKE? On Prem? AWS? ...

flavianmissi · 2017-07-31T14:17:47Z

My cluster is on GKE, master version is 1.5.7, node version 1.4.7.

$ kubectl exec mypod ls -v 9
[...]
https://same.master.ip/api/v1/namespaces/default/pods/mypod/exec

(I've omited query strings on both urls)
So exactly same url, only different protocols.

mbohlool · 2017-07-31T15:05:21Z

We had a bug in gke config loader. Can you try latest beta release 3.0.0b1 to see if that fixes your problem?

…

On Jul 31, 2017 7:17 AM, "Flavia Missi" ***@***.***> wrote: My cluster is on GKE, master version is 1.5.7, node version 1.4.7. $ kubectl exec mypod ls -v 9 [...]https://same.master.ip/api/v1/namespaces/default/pods/mypod/exec (I've omited query strings on both urls) So exactly same url, only different protocols. — You are receiving this because you modified the open/close state. Reply to this email directly, view it on GitHub <#36 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ABic4HKzsku-uV9MFXEitOk6h2xPK2mAks5sTeIMgaJpZM4K6ZZl> .

flavianmissi · 2017-07-31T18:17:40Z

Tried updating it, didn't work...

But I found the issue somewhere else, the problem is on websocket_call:

query_params = [(key, value) for key, value in query_params if
                    key != 'command']

query_params value before the above line is [('command', 'ls')], after it, query_params is empty, and that's where my problem starts. If I don't explicitly give other arguments to connect_get_namespaced_pod_exec (like stdout or stderr), the url building will be inconsistent, my resulting url is /api/v1/namespaces/default/pods/mypod/exec&command=ls, note the lack of ? in the beginning of the query string. So giving connect_get_namespaced_pod_exec actually builds a correctly formed url, e.g: /api/v1/namespaces/default/pods/mypod/exec?stderr=True&command=ls. BAM, it works 🎆

I do believe this is a bug on your side though, since only name and namespace are the only required parameters, but using them alone will break. Shall I open an issue or will you?

mbohlool · 2017-08-01T11:10:29Z

Nice that you figured it out. Both/either issue and PR are welcome.

prashantabkari · 2017-10-10T02:31:36Z

Hi - I have hit the same issue with the following log
" raise MaxRetryError(_pool, url, error or ResponseError(cause))
urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='192.168.1.108', port=6443): Max retries exceeded with url: /api/v1/pods?watch=False (Caused by SSLError(CertificateError("hostname '192.168.1.108' doesn't match either of 'master', 'kubernetes', 'kubernetes.default', 'kubernetes.default.svc', 'kubernetes.default.svc.cluster.local'",),))"

Following are my configurations
IPaddress
Metadata-Version: 2.0
Name: ipaddress
Version: 1.0.18

urllib3
Metadata-Version: 2.0
Name: urllib3
Version: 1.22

Metadata-Version: 1.1
Name: websocket-client
Version: 0.40.0

python version
Python 2.7.5

K8 cluster
kubectl cluster-info gives the correct info of master and kubedns running server

Is the above version of urllib3 supported?

UPDATE:
Following are the workaround suggested in https://github.com/kelproject/pykube/issues/29

CHange the kubeconfig file or
install python3.5

But looking for solutions with Python 2.7

czz253861763 · 2019-06-11T03:04:24Z

from kubernetes.client import configuration
config.load_kube_config()
configuration.assert_hostname = False

this works fine for me

nettyxiong · 2019-12-26T14:54:50Z

from kubernetes import client
from kubernetes import config
from kubernetes.client.api import core_v1_api

config.load_kube_config('/root/.kube/config')
configuration = client.Configuration()
configuration.assert_hostname = False
configuration.verify_ssl = True
client.Configuration.set_default(configuration)

v1 = core_v1_api.CoreV1Api()
print("Listing pods with their IPs:")

ret = v1.list_pod_for_all_namespaces(watch=False)
for i in ret.items:
    print("%s\t%s\t%s" % (i.status.pod_ip, i.metadata.namespace, i.metadata.name))

this works for me

Keep the watch action working forever

sebgoa mentioned this issue Nov 23, 2016

1.0.0-alpha Release #33

Closed

4 tasks

mbohlool closed this as completed Nov 23, 2016

mbohlool reopened this Nov 23, 2016

mbohlool closed this as completed Nov 23, 2016

mbohlool mentioned this issue Nov 23, 2016

Add ipaddress as a dependency. #43

Merged

flavianmissi mentioned this issue Aug 3, 2017

Executing commands on a pod fails when only command kwarg is provided #312

Closed

splix mentioned this issue Sep 26, 2017

bin/status.py: api exception ethereumproject/ecip1017test#3

Closed

nseyvet mentioned this issue Nov 14, 2017

Kubernetes error with "kubernetes_get_host.py" in monasca-agent monasca/monasca-docker#327

Open

roycaihw mentioned this issue Nov 22, 2021

[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed when try to connect to aks #1622

Closed

yliaog pushed a commit to yliaog/client-python that referenced this issue Jan 8, 2022

Merge pull request kubernetes-client#36 from lichen2013/keep_watch

6f02e73

Keep the watch action working forever

iridian-ks mentioned this issue Mar 31, 2022

[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get issuer certificate | Rancher EKS Cluster #1767

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

SSL verification on python2.7 failing #36

SSL verification on python2.7 failing #36

sebgoa commented Nov 23, 2016

sebgoa commented Nov 23, 2016

mbohlool commented Nov 23, 2016 •

edited

Loading

mbohlool commented Nov 23, 2016

mbohlool commented Nov 23, 2016

sebgoa commented Nov 24, 2016

madhavmalhotra3089 commented May 28, 2017

mbohlool commented May 28, 2017

madhavmalhotra3089 commented May 28, 2017 via email

tahir24434 commented Jul 12, 2017

mbohlool commented Jul 12, 2017 •

edited

Loading

flavianmissi commented Jul 28, 2017 •

edited

Loading

mbohlool commented Jul 28, 2017

flavianmissi commented Jul 31, 2017 •

edited

Loading

flavianmissi commented Jul 31, 2017

mbohlool commented Jul 31, 2017

flavianmissi commented Jul 31, 2017

mbohlool commented Jul 31, 2017 via email

flavianmissi commented Jul 31, 2017 •

edited

Loading

mbohlool commented Aug 1, 2017

prashantabkari commented Oct 10, 2017 •

edited

Loading

czz253861763 commented Jun 11, 2019

nettyxiong commented Dec 26, 2019

SSL verification on python2.7 failing #36

SSL verification on python2.7 failing #36

Comments

sebgoa commented Nov 23, 2016

sebgoa commented Nov 23, 2016

mbohlool commented Nov 23, 2016 • edited Loading

mbohlool commented Nov 23, 2016

mbohlool commented Nov 23, 2016

sebgoa commented Nov 24, 2016

madhavmalhotra3089 commented May 28, 2017

mbohlool commented May 28, 2017

madhavmalhotra3089 commented May 28, 2017 via email

tahir24434 commented Jul 12, 2017

mbohlool commented Jul 12, 2017 • edited Loading

flavianmissi commented Jul 28, 2017 • edited Loading

mbohlool commented Jul 28, 2017

flavianmissi commented Jul 31, 2017 • edited Loading

flavianmissi commented Jul 31, 2017

mbohlool commented Jul 31, 2017

flavianmissi commented Jul 31, 2017

mbohlool commented Jul 31, 2017 via email

flavianmissi commented Jul 31, 2017 • edited Loading

mbohlool commented Aug 1, 2017

prashantabkari commented Oct 10, 2017 • edited Loading

czz253861763 commented Jun 11, 2019

nettyxiong commented Dec 26, 2019

mbohlool commented Nov 23, 2016 •

edited

Loading

mbohlool commented Jul 12, 2017 •

edited

Loading

flavianmissi commented Jul 28, 2017 •

edited

Loading

flavianmissi commented Jul 31, 2017 •

edited

Loading

flavianmissi commented Jul 31, 2017 •

edited

Loading

prashantabkari commented Oct 10, 2017 •

edited

Loading