Skip to content

Latest commit

 

History

History
183 lines (129 loc) · 7.75 KB

authentication.md

File metadata and controls

183 lines (129 loc) · 7.75 KB
Raw

Configure access to Cloud Storage buckets using GKE Workload Identity Federation

Configure access

See the GKE documentation: Access Cloud Storage buckets with the Cloud Storage FUSE CSI driver

Troubleshooting Steps

If you run into permission problems, try these troubleshooting steps.

  • Uniform bucket-level access is required for read-write workloads when using Workload Identity Federation. Make sure the bucket Permissions Access control is Uniform.
  • The Cloud Storage FUSE CSI driver does not support Pods running on the host network (hostNetwork: true) due to restrictions of Workload Identity Federation for GKE. Make sure the hostNetwork is set to false.
  • If you set runAsUser or runAsGroup in Security Context for your Pod or container, or if your container image uses a non-root user or group, you must set the uid and gid mount flags. You also need to use the file-mode and dir-mode mount flags to set the file system permissions. For example, set CSI inline volume mountOptions to "uid=1001,gid=2002,file-mode=664,dir-mode=775".
  • If you set fsGroup in Security Context for your Pod, you don't need to use the file-mode and dir-mode mount flags. These flags are automatically added by the CSI fsGroup delegation feature.
  • Double check the Workload Identity Federation setup following the below steps.

Validate Workload Identity Federation and Kubernetes ServiceAccount setup

  • Make sure the Workload Identity Federation feature is enabled on your cluster:

    gcloud container clusters describe ${CLUSTER_NAME} | grep workloadPool

    The output should be like:

    workloadPool: ${PROJECT_ID}.svc.id.goog

    If not, have Workload Identity Federation enabled.

  • Make sure the DaemonSet gke-metadata-server is running on your node pool:

    kubectl get daemonset gke-metadata-server -n kube-system

    The output should be like:

    NAME                  DESIRED   CURRENT   READY   UP-TO-DATE   AVAILABLE   NODE SELECTOR                                                             AGE
gke-metadata-server   3         3         3       3            3           beta.kubernetes.io/os=linux,iam.gke.io/gke-metadata-server-enabled=true   17d

    If not, have GKE metadata server enabled on your node pool.

  • Check whether the Kubernetes ServiceAccount was created correctly:

    kubectl get serviceaccount ${KSA_NAME} --namespace ${NAMESPACE}

    The output should be like:

    NAME          SECRETS   AGE
${KSA_NAME}   0         64m

    If not, create a namespace and Kubernetes ServiceAccount accordingly. Make sure your workload runs in the same Kubernetes namespace using the Kubernetes ServiceAccount.

  • Check whether the Cloud Storage bucket has correct IAM policy bindings:

    gcloud storage buckets get-iam-policy gs://${BUCKET_NAME}

    The output should be like:

    bindings:
- members:
    - principal://iam.googleapis.com/projects/${PROJECT_NUMBER}/locations/global/workloadIdentityPools/${PROJECT_ID}.svc.id.goog/subject/ns/${NAMESPACE}/sa/${KSA_NAME}
    role: roles/storage.objectViewer

OR

bindings:
- members:
    - principal://iam.googleapis.com/projects/${PROJECT_NUMBER}/locations/global/workloadIdentityPools/${PROJECT_ID}.svc.id.goog/subject/ns/${NAMESPACE}/sa/${KSA_NAME}
    role: roles/storage.objectUser
...

    If not, follow the GKE documentation to grant one of the IAM roles for Cloud Storage to the Kubernetes ServiceAccount.

Validate the GCP and Kubernetes ServiceAccount setup (deprecated)

Note: Workload Identity Federation for GKE simplified configuration steps in GKE documentation: Configure applications to use Workload Identity Federation for GKE. Previously, the Workload Identity configuration involves extra steps to link Kubernetes ServiceAccounts to IAM, such as GCP Service Account (GSA) creation and Kubernetes ServiceAccount (KSA) configuration. With the new Workload Identity Federation for GKE feature, these steps are no longer required. This section provides validation steps for the users who still use the old configurations.

  • Make sure the Workload Identity Federation feature is enabled on the GKE cluster and node pools

    Follow the first two steps in the previous section.

  • Check whether the GCP Service Account was created:

    gcloud iam service-accounts describe ${GSA_NAME}@${GSA_PROJECT}.iam.gserviceaccount.com

    The output should be like:

    email: ${GSA_NAME}@${GSA_PROJECT}.iam.gserviceaccount.com
name: projects/${GSA_PROJECT}/serviceAccounts/${GSA_NAME}@${GSA_PROJECT}.iam.gserviceaccount.com
projectId: ${GSA_PROJECT}
...

  • Check whether the GCP Service Account has correct IAM policy bindings:

    gcloud iam service-accounts get-iam-policy ${GSA_NAME}@${GSA_PROJECT}.iam.gserviceaccount.com

    The output should be like:

    bindings:
- members:
    - serviceAccount:${PROJECT_ID}.svc.id.goog[${NAMESPACE}/${KSA_NAME}]
    role: roles/iam.workloadIdentityUser
...

  • Check whether the Cloud Storage bucket has correct IAM policy bindings:

    gcloud storage buckets get-iam-policy gs://${BUCKET_NAME}

    The output should be like:

    bindings:
- members:
    - serviceAccount:${GSA_NAME}@${GSA_PROJECT}.iam.gserviceaccount.com
    role: roles/storage.objectViewer

OR

bindings:
- members:
    - serviceAccount:${GSA_NAME}@${GSA_PROJECT}.iam.gserviceaccount.com
    role: roles/storage.objectAdmin
...

  • Check whether the Kubernetes ServiceAccount was configured correctly:

    kubectl get serviceaccount ${KSA_NAME} --namespace ${NAMESPACE} -o yaml

    The output should be like:

    apiVersion: v1
kind: ServiceAccount
metadata:
annotations:
    iam.gke.io/gcp-service-account: ${GSA_NAME}@${GSA_PROJECT}.iam.gserviceaccount.com
name: ${KSA_NAME}
namespace: ${NAMESPACE}