Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Prometheus cert provisioning is broken on istio-1.5.0-beta.5 #21843

Closed
lei-tang opened this issue Mar 5, 2020 · 3 comments
Closed

Prometheus cert provisioning is broken on istio-1.5.0-beta.5 #21843

lei-tang opened this issue Mar 5, 2020 · 3 comments
Assignees
Milestone

Comments

@lei-tang
Copy link
Contributor

@lei-tang lei-tang commented Mar 5, 2020

Bug description
Prometheus cert provisioning works on istio-1.5.0-beta.4 but is broken on istio-1.5.0-beta.5. The new commits (e.g., the ones that change Istio Agent) in istio-1.5.0-beta.5 may be related to this problem.

Affected product area (please put an X in all that apply)

[ ] Configuration Infrastructure
[ ] Docs
[ ] Installation
[ ] Networking
[ ] Performance and Scalability
[ X] Policies and Telemetry
[X ] Security
[ ] Test and Release
[ ] User Experience
[ ] Developer Infrastructure

Expected behavior
Prometheus cert provisioning works for istio-1.5.0-beta.5.

Steps to reproduce the bug
Use the steps in the guide https://github.com/istio/istio.io/pull/6554/files, which works for istio-1.5.0-beta.4 but breaks for istio-1.5.0-beta.5.

Version (include the output of istioctl version --remote and kubectl version and helm version if you used Helm)
istio-1.5.0-beta.5

How was Istio installed?
istioctl

Environment where bug was observed (cloud vendor, OS, etc)
MacOS.

Additionally, please consider attaching a cluster state archive by attaching
the dump file to this issue.

@lei-tang lei-tang added this to the 1.5 milestone Mar 5, 2020
@myidpt myidpt removed their assignment Mar 5, 2020
@lei-tang lei-tang changed the title Prometheus cert provisioning breaks for istio-1.5.0-beta.5 Prometheus cert provisioning is broken on istio-1.5.0-beta.5 Mar 5, 2020
@lei-tang

This comment has been minimized.

Copy link
Contributor Author

@lei-tang lei-tang commented Mar 5, 2020

Cause for #21843: outputKeyCertToDir used by Prometheus cert provisioning reads its value from the environmental variable OUTPUT_KEY_CERT_TO_DIRECTORY. On release-1.5 branch, a commit in istio-1.5.0-beta.5 changes outputKeyCertToDir to read its value from the environmental variable OUTPUT_CERTS, which was not set in the Prometheus deployment.

The master branch does not have the problem in #21843 because the commit in istio-1.5.0-beta.5 that changes outputKeyCertToDir to read its value from the environmental variable OUTPUT_CERTS is only merged in the release-1.5 branch but not in the master branch.

@lei-tang

This comment has been minimized.

Copy link
Contributor Author

@lei-tang lei-tang commented Mar 5, 2020

The fix for this issue: #21848.

@myidpt

This comment has been minimized.

Copy link
Contributor

@myidpt myidpt commented Mar 5, 2020

For Istio 1.5.0 users, here's how to manually fix this issue:

kubectl edit deploy prometheus -n istio-system

When the deploy config is opened, find the OUTPUT_KEY_CERT_TO_DIRECTORY in the env stanza for the proxy, and change it to OUTPUT_CERTS.

Wait for the new Prometheus instance to come up. Check the key and certificates are loaded by:

kubectl exec -it `kubectl get pod -l app=prometheus -n istio-system -o jsonpath='{.items[0].metadata.name}'` -c prometheus -n istio-system -- ls -la /etc/istio-certs/
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Linked pull requests

Successfully merging a pull request may close this issue.

None yet
4 participants
You can’t perform that action at this time.