Cannot create GCP KMS when using Google Workload Identity Federation

[kraxx]

I'm authenticating and authorizing a GitHub Action workflow to impersonate a GCP service account using Workload Identity Federation, which has permissions to decrypt a SOPS-encrypted file. I'm using https://github.com/google-github-actions/auth to do so, which exports the GOOGLE_APPLICATION_CREDENTIALS environment variable and sets it to a generated credentials file. However, when it comes to using sops to decrypt the file, I encounter this error:

Failed to get the data key required to decrypt the SOPS file.

Group 0: FAILED
  <*** path to GCP keyring ***>: FAILED
    - | Cannot create GCP KMS service: google: error getting
      | credentials using GOOGLE_APPLICATION_CREDENTIALS environment
      | variable: unknown credential type: "external_account"

Indeed, this credential file is of type external_account, not service_account which would normally work.

The error is generated here. Looking further into this, I see the function is deprecated, and on top of that, the version of the pkg.go.dev/google.golang.org/api package used in sops is severely outdated.

My suspicion lays in the outdated google api package which may not support workload identity federation, but of course there may be other reason(s) for why it's not supported.

[enchorb]

Yup, getting this same issue - seems if the sops team upgrades to the latest oath2 library this can be resolved?

ie: googleapis/repo-automation-bots#2752
ie: GoogleCloudPlatform/docker-credential-gcr#95

[soggycactus]

I have this issue as well - can we get an update about how to resolve this issue? I'd really rather not generate a JSON key as it's less secure than using Workload Identity

[zmitry]

any updates on this?

[sean-morris]

Same issue here as well - thanks for the help

[j-bro]

@kraxx The dependencies seem to have been updated in #1024, worth another look to see if this fixes the issue with Workload Identity Federation.

[zmitry]

on 3.7.2 still have this bug
| variable: unknown credential type: "external_account"

[kpocius]

It's because the referenced PR is marked for 3.7.3 release. You have to build it yourself if you want the fix included.

[zmitry]

@ajvb

[ajvb]

As @kpocius said, this will be resolved in the next point release.

[koki-sato]

When will 3.7.3 be released? We want to use Google Workload Identity Federation as soon as possible!

[multani]

v3.7.3 has been released, and I can confirm sops now works with the workload identity federation kind of credentials 🎉

Thanks a lot @ajvb for the release ❤️

[AndreLobato]

Hi, just to check if anyone is having any strange issues, when using Workload Identity to access GCP KMS I am getting permission errors intermittently throughout the day (using v3.7.3). So obviously works from time to time, but often doesn't as well at the same container. Anyone with similar symptoms?

[multani]

@AndreLobato I haven't see any issue in the wild, at least for now.

We are using sops I would say several times a day to decrypt secrets using Workload Identity obtained credentials, so maybe we also don't have a sufficient usage rate to trigger your issue?

[AndreLobato]

@multani Thanks for the response! I don't think rate is an issue, as in my case the usage is very low, I've set a monitor script running decrypt at every minute and on the last 6 hours there were only 2 occasions which failed, one lasted 4 min the other only 2 min. So I am suspecting some connectivity issue between GKE Metadata Server and GCP KMS or something else.

[austinflowspace]

Hey all,

@AndreLobato idk if you solved your issue but I was getting an error about GCP KMS key not being found even though I followed the steps to create the key.

My issue was that I kept getting a failure trying to decrypt,encrypt,etc even though I had previously used the key. What I found out was that I originally created a IAM key from a service account awhile ago that was stored in a different project. I saved this config in my local gcloud CLI and it was overriding the project location. (How this worked the first time... idk) after running the below steps I was able to remove this error even in the latest release of SOPS.

`Failed to get the data key required to decrypt the SOPS file.

Group 0: FAILED
projects/PROJECT_ID/locations/global/keyRings/sops/cryptoKeys/sops-key: FAILED
- | Error decrypting key: googleapi: Error 403: Cloud Key
| Management Service (KMS) API has not been used in project
| PROJECT_ID before or it is disabled. Enable it by visiting
| https://console.developers.google.com/apis/api/cloudkms.googleapis.com/overview?project=PROJECT_ID
| then retry. If you enabled this API recently, wait a few
| minutes for the action to propagate to our systems and
| retry.
| Details:
| [
| {
| "@type": "type.googleapis.com/google.rpc.Help",
| "links": [
| {
| "description": "Google developers console API
| activation",
| "url":
| "https://console.developers.google.com/apis/api/cloudkms.googleapis.com/overview?project=PROJECT_ID"
| }
| ]
| },
| {
| "@type": "type.googleapis.com/google.rpc.ErrorInfo",
| "domain": "googleapis.com",
| "metadata": {
| "consumer": "projects/PROJECT_ID",
| "service": "cloudkms.googleapis.com"
| },
| "reason": "SERVICE_DISABLED"
| }
| ]
| , accessNotConfigured`

Quick fix this run the following steps:

gcloud auth application-default login

export GOOGLE_APPLICATION_CREDENTIALS=/Users/$USER/.config/gcloud/application_default_credentials.json

echo 'export GOOGLE_APPLICATION_CREDENTIALS=/Users/$USER/.config/gcloud/application_default_credentials.json' > ~/.zshrc

Cheers! Hope this helps someone

[armand93420]

Hey all,

@AndreLobato idk if you solved your issue but I was getting an error about GCP KMS key not being found even though I followed the steps to create the key.

My issue was that I kept getting a failure trying to decrypt,encrypt,etc even though I had previously used the key. What I found out was that I originally created a IAM key from a service account awhile ago that was stored in a different project. I saved this config in my local gcloud CLI and it was overriding the project location. (How this worked the first time... idk) after running the below steps I was able to remove this error even in the latest release of SOPS.

`Failed to get the data key required to decrypt the SOPS file.

Group 0: FAILED projects/PROJECT_ID/locations/global/keyRings/sops/cryptoKeys/sops-key: FAILED - | Error decrypting key: googleapi: Error 403: Cloud Key | Management Service (KMS) API has not been used in project | PROJECT_ID before or it is disabled. Enable it by visiting | https://console.developers.google.com/apis/api/cloudkms.googleapis.com/overview?project=PROJECT_ID | then retry. If you enabled this API recently, wait a few | minutes for the action to propagate to our systems and | retry. | Details: | [ | { | "@type": "type.googleapis.com/google.rpc.Help", | "links": [ | { | "description": "Google developers console API | activation", | "url": | "https://console.developers.google.com/apis/api/cloudkms.googleapis.com/overview?project=PROJECT_ID" | } | ] | }, | { | "@type": "type.googleapis.com/google.rpc.ErrorInfo", | "domain": "googleapis.com", | "metadata": { | "consumer": "projects/PROJECT_ID", | "service": "cloudkms.googleapis.com" | }, | "reason": "SERVICE_DISABLED" | } | ] | , accessNotConfigured`

Quick fix this run the following steps:

gcloud auth application-default login

export GOOGLE_APPLICATION_CREDENTIALS=/Users/$USER/.config/gcloud/application_default_credentials.json

echo 'export GOOGLE_APPLICATION_CREDENTIALS=/Users/$USER/.config/gcloud/application_default_credentials.json' > ~/.zshrc

Cheers! Hope this helps someone

Helped me, thanks

[AndreLobato]

Hi @armand93420, I haven't solved the issue, but figured out why. Turns out the GCP network is fairly unstable from where I was testing from, so every now or then I would get network connection errors causing the failure to decrypt. Wasn't an issue to me as most of services have retries in place to deal with that. Glad to hear you solved your issue.