How to configure serviceaccountPath with istioctl and istio 1.5 for stackdriver

[huydinhle]

This is not a bug report. I can't find any documentations anywhere on how to correctly mount in serviceAccount file for istio 1.5. So I want to put an issue here to ask how.

I can see the configuration flag serviceAccountPath in the mixer configurations I don't know how the serviceAccount file will be mounted in the istiod pod.

Reference: https://discuss.istio.io/t/configure-istio-1-5-mixer-with-gcloud-stackdriver-backend/5907

mixer:
      stackdriver:
          auth:
            apiKey: ""
            appCredentials: false
            serviceAccountPath: ""
          enabled: false
          tracer:
            enabled: false
            sampleProbability: 1

[christianAckman]

i believe you can do this:

    mixer:
      adapters:
        stackdriver:
          enabled: true
          auth:
            serviceAccountPath: "/var/run/secrets/istio.io/policy/adapter/service-account.json"
          tracer:
            enabled: true
            sampleProbability: 1
          contextGraph:
            enabled: true

this is terraform, but you can translate:

resource "kubernetes_secret" "stackdriver_secret" {
  metadata {
    name = "policy-adapter-secret"
    namespace = "istio-system"
  }

  data = {
    "service-account.json" = file(var.path_to_service_account_json)
  }
  
  type = "generic"
}

taken from: banzaicloud/istio-operator#284

[douglas-reid]

If you are using mixer, you'll need to mount the file in the istio-telemetry pod.

But I strongly suggest using v2 telemetry with stackdriver integration.

As this is not a bug, i'm closing. Please use the forums for further investigation.

[samwhite]

@douglas-reid could you please link to where it is documented to configure v2 telemetry with Stackdriver and a GSA credentials.json file? I've looked seemingly everywhere and can't find an example of this working 😞

[douglas-reid]

@bianpengyuan do we have any good docs on using service account creds with Stackdriver?

[samwhite]

This link provides some more discussion in to our specific issue:
https://discuss.istio.io/t/v2-stackdriver-telemetry-and-workload-identity/6511

We can get the stackdriver envoy filter working if we manually grant access to a GSA with the right credentials, then associate that to the default KSA in the namespace i.e.:

$ gcloud iam service-accounts add-iam-policy-binding \
  --role roles/iam.workloadIdentityUser \
  --member "serviceAccount:xxxxxx.svc.id.goog[my-namespace/default]" \
  istio-stackdriver@xxxxxx.iam.gserviceaccount.com

$ kubectl annotate serviceaccount \
  --namespace my-namespace default \
   iam.gke.io/gcp-service-account=istio-stackdriver@xxxxxx.iam.gserviceaccount.com

However, there's no nice way for us to do this programatically for every namespace where we have envoy sidecars (and then keep it up-to-date as new ones get created / etc.)

I don't have a great solution in mind, thinking out loud: perhaps a way for Istio to create the istio-proxy pods and inject the credentials file (managed by the operator) somehow?

We're going to investigate an in-mesh prometheus -> stackdriver approach instead, but it would be great to use the envoy filter as it definitely works, just don't have a way for it to authenticate when the cluster is using Workload Identity

[bianpengyuan]

Currently we don't have a way to read mounted cert in stackdriver filter. It might be feasible though to expose that as an option in both stackdriver filter configuration and istioctl. Although IMHO using workload identity is strictly better than managing the cert file yourself. Not sure if there is any gitops tools out there or Google provides to ease the pain.

[samwhite]

With Mixer our solution was placing a credentials.json file in a Secret and mounting it in to the mixer pod -- I don't think something similar quite translates nicely to each envoy proxy, though, agree... 😓

That was another option we've considered:

• Have something in the cluster which monitors for namespaces and uses the IAM API to do the equivalent to that ^ gcloud command automatically
• Our teams can do the annotation of the default SA (or we could use a non-default one) themselves as we already manage all of that through GitOps/Argo

[samwhite]

It appears that the Workload Identities and even KSAs can be created through terraform, so that might be a solution:

https://registry.terraform.io/modules/terraform-google-modules/kubernetes-engine/google/7.1.0/submodules/workload-identity

Still feels quite roundabout, though, I wonder how common our use case is -- it may become more common for those wishing to go directly to Stackdriver from envoy as Google is now recommending Workload Identity over any other options.

[douglas-reid]

Mounting into each pod definitely is a bit clunky, but is at least somewhat doable. You'd have to set the appropriate env var in each pod too.

We probably need a more general approach to creds distribution for extensions in general. @mandarjog @kyessenov has there been any thoughts of how to support creds via extensions API, etc.? Is the working suggestion to bundle ?

[samwhite]

A colleague of mine just had a brilliant idea, building on the 'manage in terraform' solution, which is to use Config Connector to set up the KSA/GSAs. This keeps everything in k8s manifests, at least.

It's quite a specific solution to the "Stackdriver on GCP" usecase, so we might still want to think about creds in general, but it might be nice for this specific case.

I'll give that a go in the next couple of days and report back on how it goes.

[samwhite]

So this actually works pretty great. This is what I added to the namespace:

apiVersion: iam.cnrm.cloud.google.com/v1beta1
kind: IAMPolicy
metadata:
  name: stackdriver-gsa-binding
spec:
  resourceRef:
    apiVersion: iam.cnrm.cloud.google.com/v1beta1
    kind: IAMServiceAccount
    external: projects/my-project/serviceAccounts/istio-stackdriver@my-project.iam.gserviceaccount.com
  bindings:
    - role: roles/iam.workloadIdentityUser
      members:
        - serviceAccount:my-project.svc.id.goog[my-namespace/default]
---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: default
  annotations:
    iam.gke.io/gcp-service-account: istio-stackdriver@my-project.iam.gserviceaccount.com

istio-stackdriver was a GSA which already existed, but one could also manage that through Config Connector too if desired.

The only thing that's a little bit askew here is having to modify the default SA inside the namespace, as that's what the istio-proxy pod inherits. That's okay, but is there a neat way we can point the sidecar at a different to the default (does it just use whichever the application pod itself has configured?)

[bianpengyuan]

does it just use whichever the application pod itself has configured?

Yes

[samwhite]

Ah fantastic, thanks @bianpengyuan 🙏

We'll probably suffice with using the same KSA as the application container, though I suppose for the best security one would want a specific KSA for the stackdriver telemetry, which may differ to the KSA we use for the application pod itself. I wonder if there's a neat way to configure which KSA the istio-proxy container uses...

[bianpengyuan]

afaik The finest granularity of workload identity if per pod. You'd either need to run proxy outside application pod using Sidecar API, or adopt anthos service mesh https://cloud.google.com/anthos/service-mesh, which includes a solution to exchange anthos service mesh gcp service access token at sidecar, which would make separated access control possible.

[cagataygurturk]

We are also using Config Connector to quickly provision GSA's to every Pod. It is not very elegant though.

@bianpengyuan I wonder how ASM does this and whether it is possible to implement the same functionality to OSS Istio.

[bianpengyuan]

@cagataygurturk ASM solve this by exchanging a specialized access token with trust worthy jwt token, which has metric write permission. It is not open to OSS users.