[Istio-1.3.0-rc0] Community Testing, 503 error in Istio Vault CA Integration

[lei-tang]

This bug may be related to the changes for k8s trustworthy JWT.

Bug description:
Following the instructions on https://preliminary.istio.io/docs/tasks/security/vault-ca/, at the step "kubectl exec -it $(kubectl get pod -l app=sleep -o jsonpath='{.items[0].metadata.name}') -c sleep -- curl -s -o /dev/null -w "%{http_code}" httpbin:8000/headers", get an error as follows:

~$ kubectl exec -it $(kubectl get pod -l app=sleep -o jsonpath='{.items[0].metadata.name}') -c sleep -- curl -s -o /dev/null -w "%{http_code}" httpbin
:8000/headers
503

Affected product area (please put an X in all that apply)

[ ] Configuration Infrastructure
[X ] Docs
[ ] Installation
[ ] Networking
[ ] Performance and Scalability
[ ] Policies and Telemetry
[ X] Security
[ ] Test and Release
[ ] User Experience
[ ] Developer Infrastructure

Expected behavior
No errors, return 200.

Steps to reproduce the bug
Following the instructions on https://preliminary.istio.io/docs/tasks/security/vault-ca/.

Version (include the output of istioctl version --remote and kubectl version)
~$ ./bin/istioctl version --remote
client version: 1.3.0-rc.0
citadel version: 1.3.0-rc.0
galley version: 1.3.0-rc.0
ingressgateway version: 1.3.0-rc.0
nodeagent version:
nodeagent version:
nodeagent version:
pilot version: 1.3.0-rc.0
policy version: 1.3.0-rc.0
sidecar-injector version: 1.3.0-rc.0
telemetry version: 1.3.0-rc.0

~$ kubectl version
Client Version: version.Info{Major:"1", Minor:"13", GitVersion:"v1.13.2", GitCommit:"cff46ab41ff0bb44d8584413b598ad8360ec1def", GitTreeState:"clean", BuildDate:"2019-01-13T23:16:58Z", GoVersion:"go1.11.4", Compiler:"gc", Platform:"darwin/amd64"}
Server Version: version.Info{Major:"1", Minor:"12+", GitVersion:"v1.12.8-gke.10", GitCommit:"f53039cc1e5295eed20969a4f10fb6ad99461e37", GitTreeState:"clean", BuildDate:"2019-06-19T20:48:40Z", GoVersion:"go1.10.8b4", Compiler:"gc", Platform:"linux/amd64"}

How was Istio installed?
helm install.

Environment where bug was observed (cloud vendor, OS, etc)
Linux

Additionally, please consider attaching a cluster state archive by attaching
the dump file to this issue.

[lei-tang]

@pitlv2109 Is this bug caused by the changes for k8s trustworthy JWT?

[lei-tang]

the sleep pod istio-proxy log is as follows:
sleep-log.txt

[lei-tang]

the httpbin pod istio-proxy log is as follows:
httpbin-log.txt

[pitlv2109]

Could you tell me the CSR flow with Vault? So Citadel agent gets a SDS request from the client with the JWT. What will happen next or is this the same as Citadel?

[lei-tang]

The CSR flow is Envoy -> Citadel Agent -> Vault. Let's discuss offline what are the changes for trustworthy JWT and whether they relate to the 503 error in the Vault CSR flow.

[pitlv2109]

So it's the same as Citadel. Vault integreation tests have been passing though https://prow.istio.io/?job=integ-security-k8s-postsubmit-tests-master

[lei-tang]

@pitlv2109 Can you provide some information about the changes for trustworthy JWT?

[pitlv2109]

Trustworthy JWTs are JWTs that are mounted into the sidecar containers and have limited scopes, such as aud and exp. It's available from k8s v1.12 (beta). OSS k8s must enable some api server flags.

[lei-tang]

The kubectl server version is v1.12.8-gke.10 (in the issue description). Does this version uses trustworthy JWT?

[pitlv2109]

It should be. You can check if the sidecar container has an istio-token at this path /var/run/secrets/tokens

[lei-tang]

Istio release 1.3 uses new k8s JWT (#16147), which breaks the user guide of Istio Vault CA integration for release 1.3. The PR istio/istio.io#4904 temporarily disables the user guide of Istio Vault CA integration for release 1.3.

[myidpt]

We will temporarily remove the Vault CA integration task from istio.io. It will be added back once the issues are fixed.

[duderino]

@geeknoid could you also review istio/istio.io#4904 to unblock this additional 1.3 release blocker