[suggestion] specify necessary IAM roles in documentation

[oofnikj]

I had an issue where metrics weren't showing up in Monitoring and I saw the following error in the sidecar logs being printed continuously:

level=warn ts=2019-07-30T05:13:17.12753024Z caller=queue_manager.go:546 component=queue_manager msg="Unrecoverable error sending samples to remote storage" err="rpc error: code = PermissionDenied desc = Permission monitoring.timeSeries.create denied (or the resource may not exist)."

It was fixed by creating an IAM policy binding from the service account associated with the cluster to the role roles/monitoring.metricWriter.

Might be helpful to explicitly specify that this permission is required somewhere in the documentation.

[jkohen]

Hi, thanks for the suggestion and I'm glad to hear you've solved the issue already. It sounds like you are using the software with GKE. The GKE documentation states that this permission is required at https://cloud.google.com/kubernetes-engine/docs/how-to/hardening-your-cluster#use_least_privilege_sa

Let us know if we can help any other way.

[philips]

The docs here don't see to talk about the service account which is very confusing: https://cloud.google.com/monitoring/kubernetes-engine/prometheus

[hixichen]

for anyone who search and view this page:

refer: GKE workload identity

export GCP_PROJECT=my-project
export GCP_SA=gke-prometheus
export K8S_SA=prometheus
export K8S_NS=prometheus

gcloud iam service-accounts create ${GCP_SA} --display-name=${GCP_SA}

gcloud iam service-accounts add-iam-policy-binding \
  --role roles/iam.workloadIdentityUser \
  --member "serviceAccount:${GCP_PROJECT}.svc.id.goog[${K8S_NS}/${K8S_SA}]" \
  ${GCP_SA}@${GCP_PROJECT}.iam.gserviceaccount.com

gcloud projects add-iam-policy-binding ${GCP_PROJECT} \
  --member "serviceAccount:${GCP_SA}@${GCP_PROJECT}.iam.gserviceaccount.com" \
  --role roles/monitoring.metricWriter

gcloud projects add-iam-policy-binding ${GCP_PROJECT} \
  --member "serviceAccount:${GCP_SA}@${GCP_PROJECT}.iam.gserviceaccount.com" \
  --role roles/monitoring.viewer


gcloud projects add-iam-policy-binding ${GCP_PROJECT} \
  --member "serviceAccount:${GCP_SA}@${GCP_PROJECT}.iam.gserviceaccount.com" \
  --role roles/logging.logWriter


gcloud projects add-iam-policy-binding ${GCP_PROJECT} \
  --member "serviceAccount:${GCP_SA}@${GCP_PROJECT}.iam.gserviceaccount.com" \
  --role roles/stackdriver.resourceMetadata.writer


kubectl annotate serviceaccount ${K8S_SA} \
  iam.gke.io/gcp-service-account="${GCP_SA}@${GCP_PROJECT}.iam.gserviceaccount.com" \
  -n ${K8S_NS}