Constantly get error message: Mesh-wide mTLS status feature disabled

[jhasselgren]

Describe the bug
After the summer we suddenly get the following error message in Kiali in all of our environments (dev, test, prod):

Mesh-wide mTLS status feature disabled., Info: [ the server could not find the requested resource (get meshpolicies.authentication.istio.io) ]
8 in 2 minutes
07/08/201909:46:45

After some troubleshooting done in Kiali Forum

This is our findings:
Kiali makes the following call:

http://localhost:8080/apis/authentication.istio.io/v1alpha1/namespaces/anyNamespace/meshpolicies

And gets a 404 as a reply

But if we makes the same call and ignoring the namespace part we get a reply

http://localhost:8080/apis/authentication.istio.io/v1alpha1/meshpolicies

And the reason for this looks to be that meshpolicies in APIResourceList has "namespaced": false

Versions used
Kiali: 1.1.0 & 1.2.0
Istio: 1.2.0
Kubernetes flavour and version: version: v1.11.10-eks-f12431 (AWS EKS)

Expected behavior
That Kiali should be able to retrive the default meshpolicy with no errors

[lucasponce]

@jhasselgren I suspect this is related the specific built of AWS EKS.
I'm testing Istio 1.2.1 upstream and I can run that query without a 404 error.

I need to investigate more.

[lucasponce]

I've used for my tests a kubernetes version kubernetes v1.11.0+d4cacc0
based on OpenShift 3.11 and deploying Istio 1.2.2 as described in Istio upstream https://istio.io/docs/setup/kubernetes/platform-setup/openshift/.

This is a starting point, just to spot if the potential issue was in Istio / Kiali code base of it's related to the platform as it looks like.

I'm trying to setup a AWS EKS environment to also confirm/reproduce the issue.

[lucasponce]

Ok, I can get the same 404 error in the API.
Yes, there is a difference between how k8s platform deal with cluster-wide objects, it seems on this case AWS EKS is forcing in a strict way.
Anyway, I will study a fix in Kiali codebase.

Thanks for your time reporting this issue @jhasselgren, it was a tricky one as Kiali was using a side effect that worked in all k8s we tested but not on AWS EKS.

[lucasponce]

https://issues.jboss.org/browse/KIALI-3223

[israel-hdez]

Looks like Kiali was taking advantage of something that was classified as a security vulnerability:

The CVE (apparently, still undisclosed): https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2019-11247
The k8s public CVE in GitHub: kubernetes/kubernetes#80983
The 15-days ago fix: kubernetes/kubernetes#80750

A related article: https://www.stackrox.com/post/2019/08/how-to-remediate-kubernetes-security-vulnerability-cve-2019-11247/

Indeed the CVE makes sense. Why see a cluster wide resource if you shouldn't have access? :/

So this is not specific to AKS, but a broader issue that will be eventually fixed in all k8s maintained versions and flavors (and I guess it's going to be soon). Which means that we must stop using these kind of queries to the cluster API.

[lucasponce]

@jhasselgren latest images from master should fix the issue.

Please let us know if there is any additional comment on this.

Closed by #1346

[jhasselgren]

Tried it and it works like a charm, good work with fixing the problem!