Accessing the API Server from a pod using service account causes an error

[akshayl]

K8S Version: 1.2
Code:

api = HTTPClient(KubeConfig.from_service_account())
pods = Pod.objects(api).filter(namespace="gondor-system")
ready_pods = filter(operator.attrgetter("ready"), pods)

Error message:

SSLError: hostname '10.3.240.1' doesn't match either of 'kubernetes', 'kubernetes.default', 'kubernetes.default.svc', 'kubernetes.default.svc.cluster.local'

[brosner]

This is happening likely due to running on Python < 3.5. IP hostnames are not supported until Python 3.5. I am open to ideas on how to fix it as we are not generating the config. I am not a huge fan of version checking and replacement as we can't guanrantee hostnames.

[brosner]

One thing that occurs to me is that I added PYKUBE_KUBERNETES_SERVICE_HOST environment variable support to override KUBERNETES_SERVICE_HOST provided to the pod. This should give you the ability to do:

PYKUBE_KUBERNETES_SERVICE_HOST=kubernetes

which will work around the certificate checking against the IP that fails against Python < 3.5.

[jrydberg]

I'm getting this straight of the box with a kube configuration file created by gcloud.

Is there no way to get around this on python 2.7 system?

[brosner]

@jrydberg unfortunately there is nothing I am aware of that will fix it. The underlying issue is within Python. It may be possible to override the certificate checking mechanism to add IP hostname support to Python < 3.5, but I would hate to this at the risk of comprising security already being managed by Python.

[zcarlson-signifai]

It turns out that pykube uses requests, which pulls in urllib3, which attempts to pull in match_hostname first from Python native SSL, then from backports, falling back on its own implementation if neither are available. However, the latest backports's match_hostname seems to be more complete than the native SSL match_hostname, allowing for IP addresses. You can monkey-patch this in your end script with the following, so long as "IP Address:" actually does show up in subjectAltNames:

import backports.ssl_match_hostname
import pykube

# Monkey-patch match_hostname with backports's match_hostname, allowing for IP addresses
# XXX: the exception that this might raise is backports.ssl_match_hostname.CertificateError
pykube.http.requests.packages.urllib3.connection.match_hostname = backports.ssl_match_hostname.match_hostname

[zcarlson-signifai]

Of course it should be noted that the 'real fix' is for the packaged version of urllib3 with requests to use backports.ssl_match_hostname over the native, if available. But in the short term...

[paultiplady]

Feels like resorting to monkey-patching is not a user-friendly solution to this problem, nor is "use Python 3.5", so I've opened an issue at urllib3 to try to resolve the underlying issue.

Think there's probably a very simple change to that library that would get round this issue, but I might be missing something. Let's see what the urllib guys have to say (they are usually super-helpful).

[Lukasa]

Hey folks! Just a FYI: I'm looking at working on this next week. It's slightly more complex than we'd like because the new match_hostname uses the ipaddress module, which we will also need to backport, but it's definitely on our radar.

[zcarlson-signifai]

@Lukasa You would? pip install ipaddress worked for me on python 2.7...

[paultiplady]

@zcarlson-signifai I think the property they are trying to preserve is that pip install requests should work out of the box (including the new IP SAN handling), without the user having to know to pip install ipaddress as well.

[paultiplady]

FTR the issue I raised at urllib3 was a dupe of this one: urllib3/urllib3#258

Linking here so that we can more easily see when it's resolved.

[JohnSimowitz]

monkey-patching with backports.match_hostname did not work for me... but fortunately there is a simpler approach. If you only need to communicate with the kubernetes master you can define your own match_hostname function.

def _MatchHostname(_, hostname):
  if hostname != os.environ['KUBERNETES_SERVICE_HOST']:
    raise ssl.CertificateError('Hostname [%s] not equal to [%s].',
        hostname, os.environ['KUBERNETES_SERVICE_HOST'])

Now monkey patch with this and it appears to work fine.

[victorgp]

So it seems that this got fixed in urllib3, thanks @Lukasa ! now we just need to wait for a new version of 'requests'

[yuefengz]

I am still having the issue, with a local config file, with python 2.7, with urllib3 installed from its source. Is this because of my python version?

[Lukasa]

@YuefengZhou You need to install some extra third-party dependencies. Did you install urllib3[secure]?

[yuefengz]

@Lukasa Yes. I have installed it and and I have just run pip2 install --upgrade urllib3[secure] again but still failed.

[victorgp]

@Lukasa how can just by installing third party libraries this work? if requests have 'urllib3' embedded in its code, although we installed the right urllib3 version, requests will use its embedded one.

[Lukasa]

@victorgp Depends on the source. For pip installed versions of Requests, simply updating urllib3 doesn't work.

[yejw5]

Thanks @zcarlson-signifai and pykube.http.requests.packages.urllib3.connection.match_hostname = backports.ssl_match_hostname.match_hostname worked for me.

But now the package has changed and we could not find requests in pykube/http.py. Is there any way for us to fix it when we still have hostname '10.3.240.1' doesn't match either issue in Python 2.7?

[mcapuccini]

If you modify your kubeconfig file, and put the API server hostname instead of the IP address it is going to work

[yejw5]

@mcapuccini Many thanks! This works for me perfectly.

[philographer]

@mcapuccini how can i change the server hostname??

[mcapuccini]

@yoohoogun114 if you open the kubeconfig file, it should be something like::

server: X.X.X.X

You need to change X.X.X.X to a hostname (if you are using minikube, don't forget to add the hostname in /etc/hosts)

[victorgp]

That solution doesn't work if you are using GCP.

The SSL certs of the k8s masters only have SAN for kubernetes.default and kubernetes.default.svc.cluster.local therefore you cannot set the domain you want.

A workaround is to set in your kubeconfig file the server name as kubernetes.default.svc.cluster.local and fake the DNS in /etc/hosts making that domain point to the GCP cluster ip

[paultiplady]

With the latest version of Requests, I believe this issue is resolved. I've put up a PR with a change that requires the fixed requests version (#95), if anyone wants to give that a try that would be helpful.

Or you should be able to just do pip install --upgrade requests>=2.12.0 to get the appropriate version, though you'll get a spurious warning without the PR mentioned above.

[yejw5]

After added certificate to kubernetes, the issue exception: hostname 'X.X.X.X' doesn't match either of 'k8s-master', 'kubernetes' ... appears again. I tried the ways of upgrading requests and urllib3, installing ipaddress metioned above. But only modifying kuberconfig and /etc/hosts as @victorgp said work fine for me. It seems tricky. Is there other ways to solve this problem?

[trulyshelton]

I managed to make it work by updating the backports.ssl_match_hostname module.
Anyone who can't solve the issue through the above solutions may try add backports.ssl-match-hostname>=3.5.0.1 to the requirements.

The reason seems to be that an outdated version of backports.ssl_match_hostname would not check against IP.