Provider configuration values supplied as empty strings/zero-values are no longer treated like null values (muxed provider, v4.60.2+)

[goobysnack]

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request.
• Please do not leave +1 or me too comments, they generate extra noise for issue followers and do not help prioritize the request.
• If you are interested in working on this issue or have submitted a pull request, please leave a comment.
• If an issue is assigned to the modular-magician user, it is either in the process of being autogenerated, or is planned to be autogenerated soon. If an issue is assigned to a user, that user is claiming responsibility for the issue. If an issue is assigned to hashibot, a community member has claimed the issue already.

Terraform Version

v1.1.9

Affected Resource(s)

Google Provider

### Debug Output

[DEBUG] provider.terraform-provider-google_v4.60.2_x5: Calling provider defined validator.String: @module=sdk.framework description="value must match regular expression '.*/[^/]+/$'" tf_attribute_path=redis_custom_endpoint tf_rpc=PrepareProviderConfig @caller=github.com/hashicorp/terraform-plugin-framework@v1.1.1/internal/fwserver/attribute_validation.go:704 tf_mux_provider=*proto5server.Server tf_provider_addr=registry.terraform.io/hashicorp/google tf_req_id=<removed>

[DEBUG] provider.terraform-provider-google_v4.60.2_x5: Called provider defined validator.String: @module=sdk.framework tf_provider_addr=registry.terraform.io/hashicorp/google tf_req_id=<removed>@caller=github.com/hashicorp/terraform-plugin-framework@v1.1.1/internal/fwserver/attribute_validation.go:714 description="value must match regular expression '.*/[^/]+/$'" tf_attribute_path=redis_custom_endpoint tf_mux_provider=*proto5server.Server tf_rpc=PrepareProviderConfig 

[DEBUG] provider.terraform-provider-google_v4.60.2_x5: Calling provider defined validator.String: tf_rpc=PrepareProviderConfig @caller=github.com/hashicorp/terraform-plugin-framework@v1.1.1/internal/fwserver/attribute_validation.go:704 @module=sdk.framework description="value must match regular expression '.*/[^/]+/$'" tf_attribute_path=tpu_custom_endpoint tf_req_id=<removed> tf_mux_provider=*proto5server.Server tf_provider_addr=registry.terraform.io/hashicorp/google 


Error: JSON credentials are not valid
  with provider["registry.terraform.io/hashicorp/google"],
plugin process exited: path=.terraform/providers/registry.terraform.io/hashicorp/google/4.60.2/linux_amd64/terraform-provider-google_v4.60.2_x5
Terraform invocation failed

When I constrained to version 4.59.0, the issue resolved.

Panic Output

Expected Behavior

Plan should have run

Actual Behavior

Plan failed

Steps to Reproduce

1. terraform plan

Important Factoids

References

• #0000

[goobysnack]

To be clear, when I constrained the provider version to 4.59.0, I was able to run a plan without any other changes.

[edwardmedia]

@goobysnack it is fine with me for provider.terraform-provider-google_v4.60.2_x5: Mind sharing your full debug log so I can take a closer look?

Error: JSON credentials are not valid

[goobysnack]

I wasn't sure if the tf_req_id was a secret, so I removed it. LMK if that ID is critical and I'll paste it here.

google_provider_4_60_debug.txt

[edwardmedia]

@goobysnack looking closer to below error, it complains the JSON format of your key file. Have you made any changes in your key file? Can you generate a new one to see if that solve the problem?

Error: JSON credentials are not valid

  with provider["registry.terraform.io/hashicorp/google"],
  on base-provider.tf line 11, in provider "google":
  11: provider "google" {

unexpected end of JSON input

[goobysnack]

you mean my google account key file? If so, I did both a gcloud auth login and gcloud auth application default login thinking as you said, and it still err'd. Additionally, as soon as I reverted to 4.59.0 without any other corrective action, terraform worked.

[edwardmedia]

@goobysnack when you use gcloud auth application default login, did you unset the environment variable for one like GOOGLE_CREDENTIALS?

[goobysnack]

GOOGLE_CREDENTIALS is not set.

[edwardmedia]

@goobysnack there are multiple environment variables that can be used for the key files. From that suggestion here, GOOGLE_CREDENTIALS is just one of them. I just wanted to make sure you were using gcloud auth application default login.

It seems a little weird for below error while you authenticated yourself with gcloud auth application default login. Can you confirm?

Error: JSON credentials are not valid

  with provider["registry.terraform.io/hashicorp/google"],
  on base-provider.tf line 11, in provider "google":
  11: provider "google" {

unexpected end of JSON input

To be clear, I do see there was an update in the v4.59.0 and you have confirmed v4.59.0 works for you.

[goobysnack]

yup, I ran both gcloud auth commands.

gcloud auth login
gcloud auth application-default login 

[edwardmedia]

@trodge do you have any idea what could go wrong?

[goobysnack]

FYI I did a full revoke of all credentials and retried with 4.61.0 and got the same error.

[megan07]

Hi @goobysnack! I'm sorry you're running into this issue. I'm unable to reproduce it, nor do I see any differences in the code where I would expect to see differences. Could you share your provider config block with me and let me know which environment variables you're defining please?

Thanks!

[adlotsof]

The issue is persisting for me with provider version v4.62.0. Removing the empty credentials string from the provider block fixes the problem for me.

[SarahFrench]

I'll reopen for more investigation but, just to manage expectations, I can't look into this until next week earliest. Hopefully that isn't an issue as there's the workaround you described!

[briananstett]

Just adding another example of the credentials argument in provider block breaking post 4.59. Since Firebase API doesn't support end user credentials, when trying to apply Firebase resources with end user credentials (gcloud auth application-default login) you get the following error.

╷
│ Error: Error creating Project: googleapi: Error 403: Your application has authenticated using end user credentials from the Google Cloud SDK or Google Cloud Shell which are not supported by the firebase.googleapis.com. We recommend configuring the billing/quota_project setting in gcloud or using a service account through the auth/impersonate_service_account setting. For more information about service accounts and how to use them in your application, see https://cloud.google.com/docs/authentication/. If you are getting this error with curl or similar tools, you may need to specify 'X-Goog-User-Project' HTTP header for quota and billing purposes. For more information regarding 'X-Goog-User-Project' header, please check https://cloud.google.com/apis/docs/system-parameters.
│ Details:
│ [
│   {
│     "@type": "type.googleapis.com/google.rpc.ErrorInfo",
│     "domain": "googleapis.com",
│     "metadata": {
│       "consumer": "projects/764086051850",
│       "service": "firebase.googleapis.com"
│     },
│     "reason": "SERVICE_DISABLED"
│   }
│ ]
│ 
│   with module.firebase.google_firebase_project.this,
│   on ../../main.tf line 45, in resource "google_firebase_project" "this":
│   45: resource "google_firebase_project" "this" {
│ 
╵

Some blogs suggest creating a service account in Terraform and then source the service account's key directly into google-beta provider as a work around

provider "google-beta" {
  project     = "foo"
  credentials = base64decode(google_service_account_key.terraform_service_account.private_key)
}

But this breaks after 4.59 with the following error.

│ Error: unable to parse credentials
│ 
│   with module.firebase.provider["registry.terraform.io/hashicorp/google-beta"],
│   on ../../main.tf line 6, in provider "google-beta":
│    6: provider "google-beta" {
│ 
│ unexpected end of JSON input

[SarahFrench]

Hi @briananstett thanks for this example, it helps a lot. I’m in an EMEA time zone so can’t look into this now but can do tomorrow. I wanted to briefly come online and ask some questions real quick:

If you’re able, it would be helpful to have some debug logs from when you experience the issue. 🙏

If that’s not possible, am I right thinking that to reproduce the issue in a minimal way I only need to include:

• GA provider authenticated with application default credentials from the CLI only (no auth arguments in the provider config, no auth supplied by environment variables)
• GA provider used to provision a service account + a service account key
• (in a second, separate apply step) Beta provider set up only in the way you’ve shown, using the key
• Create a plan and the error will be present

[briananstett]

Hey @SarahFrench, attached is a file with some Terraform code I used to reproduce the error. I've also attached the debug logs.

If that’s not possible, am I right thinking that to reproduce the issue in a minimal way I only need to include:

GA provider authenticated with application default credentials from the CLI only (no auth arguments in the provider config, no auth supplied by environment variables)
GA provider used to provision a service account + a service account key
(in a second, separate apply step) Beta provider set up only in the way you’ve shown, using the key
Create a plan and the error will be present

Almost.

• GA provider authenticated with application default credentials from the CLI only (no auth arguments in the provider config, no auth supplied by environment variables).
• GA provider used to provision a service account + a service account key.
• In the same apply, the Beta provider uses the newly created service account/key to authentication to create a Firebase project.

Steps to reproduce

1. Update the billing_account, org_id, and name arguments in the main.tf file for your test.
2. Get GCP credentials using gcloud auth application-default login
3. terraform init and terraform plan with provider version 4.59. You should get a "clean" plan.
4. Update the provider version to 4.62 and terraform init --upgrade
5. Run terraform plan again and should get the following error.

╷
│ Error: unable to parse credentials
│ 
│   with provider["registry.terraform.io/hashicorp/google-beta"],
│   on main.tf line 21, in provider "google-beta":
│   21: provider "google-beta" {
│ 
│ unexpected end of JSON input
╵

files.zip

[SarahFrench]

@briananstett Thanks for this, it's a big help! I've found the cause of that issue but I'm going to open a separate issue for it to help with search-ability in GitHub

Edit: here is is - #14444

[SarahFrench]

I returned to looking at the original issue of setting credentials = "" in the google provider's configuration.

Problems observed with credentials = ""

When gcloud auth application-default login has been run on my machine, no other ENVs for configuring the provider are present, and the provider is configured like this:

provider "google" {
  credentials = ""
}

v4.59.0 : I able able to create a Terraform plan OK with the above

v4.60.2 : After introduction of muxing in the provider, the same configuration results in this error:

╷
│ Error: JSON credentials are not valid
│ 
│   with provider["registry.terraform.io/hashicorp/google"],
│   on main.tf line 11, in provider "google":
│   11: provider "google" {
│ 
│ unexpected end of JSON input
╵

v4.62.1 : After updating the validator code to accept an empty string, a new error is triggered:

╷
│ Error: unable to parse credentials
│ 
│   with provider["registry.terraform.io/hashicorp/google"],
│   on main.tf line 12, in provider "google":
│   12: provider "google" {
│ 
│ unexpected end of JSON input
╵

Investigation

The new error is because the muxed provider treats "" as a fully valid credentials value that a user has supplied. This code block executes because credentials is not a null value - it has a user-supplied value. The googleoauth package correctly reports that "" isn't a parsable set of credentials when this code executes:

terraform-provider-google/google/framework_config.go

Lines 1361 to 1365 in e520fc3

	creds, err := googleoauth.CredentialsFromJSON(ctx, []byte(contents), clientScopes...)
	if err != nil {
	diags.AddError("unable to parse credentials", err.Error())
	return googleoauth.Credentials{}
	}

The reason why v4.59.0 didn't return an error when credentials = "" was because previously some of the code used expressions like != "" to check whether credentials differed from the zero value (implicitly asking if it had been set by a user). Now the plugin framework in v4.60.2+ is better at detecting when values are supplied by a user or not, so it's less possible for user-supplied values that match Go's zero values to slip through unnoticed. If anything this was a bug in the old SDK code!

Conclusion

I think that the new behaviour is more correct than the old behaviour with the SDK, though it is a breaking change. But the fix in the config is easy. I'm open to discussion in this issue if there's strong objections to leaving things as it is, and will ask my team mates for their opinions too.

[SarahFrench]

Also, if we decide not to fix this issue we may need to return to saying that "" is not a valid value

[goobysnack]

Also, if we decide not to fix this issue we may need to return to saying that "" is not a valid value

Yup. Works and understood/agree. Thanks for all your hard work on this! Cheers.

[SarahFrench]

We've decided to update future 4.x.x versions to have parity with the old versions of the provider that only used the SDK to structure the code. When 5.0.0 comes around we'll remove the ability to set provider arguments as "" and have them ignored by the code in favour of ENVs/ADC/etc.

So, I'll leave this issue open until the provider config logic has been updated to allow credentials="" (and the same for other string type arguments).

[github-actions]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.
If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.