Creating a cloud run job errors when trying to use secret it doesn't have access to but still gets created

[robdit]

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request.
• Please do not leave +1 or me too comments, they generate extra noise for issue followers and do not help prioritize the request.
• If you are interested in working on this issue or have submitted a pull request, please leave a comment.
• If an issue is assigned to the modular-magician user, it is either in the process of being autogenerated, or is planned to be autogenerated soon. If an issue is assigned to a user, that user is claiming responsibility for the issue. If an issue is assigned to hashibot, a community member has claimed the issue already.

Terraform Version

1.3.1
provider version: 4.54.0

Affected Resource(s)

• google_cloud_run_v2_job

Terraform Configuration Files

# Copy-paste your Terraform configurations here.
#
# For large Terraform configs, please use a service like Dropbox and share a link to the ZIP file.
# For security, you can also encrypt the files using our GPG public key:
#    https://www.hashicorp.com/security
#
# If reproducing the bug involves modifying the config file (e.g., apply a config,
# change a value, apply the config again, see the bug), then please include both:
# * the version of the config before the change, and
# * the version of the config after the change.

Debug Output

Panic Output

Expected Behavior

When there's an error with the configuration I expected it to not create the cloud run job.

Actual Behavior

The terraform apply fails saying:

Error waiting to create Job: Error waiting for Creating Job: Error code 13, message: spec.template.spec.containers[0].env[11].value_from.secret_key_ref.name: Permission denied on secret: <secret> for Revision service account <service_account>. The service account used must be granted the 'Secret Manager Secret Accessor' role (roles/secretmanager.secretAccessor) at the secret, project or higher level

Even though terraform exits with non-zero status the job still get created. In the job details page it shows the same error as terraform did.
If I try to do an apply again terraform fails with:
Error 409: Resource '<cloud_run_job>' already exists.

Steps to Reproduce

Create a secret
Create a cloud run job referring to that secret but with a user that doesn't have permissions to access it

Important Factoids

References

• b/272785144

Similar to this issue

• Cannot assign secret to google_cloud_run_v2_job if permissions are not granted before. #13786

[edwardmedia]

@rdittrich97 can you share your config and the debug log? Is this question related? #13830

[robdit]

bug_report.zip
here you go @edwardmedia
I don't think that issue is related

[edwardmedia]

@rdittrich97 can you post the content directly here? If that is too long, maybe post and share at https://gist.github.com/?

[robdit]

@edwardmedia
https://gist.github.com/rdittrich97/3fa2fba25cc5cb036417187125310fc2

[yanweiguo]

When creating a Cloud Run resource, Cloud Run API only does some simple verification for the create request. For example, some value should not be negative. It doesn't check the secret since it requires another API call which will significantly increase create API latency. If the request passes the verifications, a cloud run resource will be created with Ready condition set to Unknown.

Then the reconciling process will start to make the resource into the desired state as described by spec. In the case described in this issue, Cloud Run tries to access the secret then gets permission denied. Then the Ready condition is set to False.

TF waits the Ready condition to become True or False when creating a cloud run resource. When it detects a False Ready condition, it treats the creation as failure.

@rdittrich97 does this explain why the resource still get created?

[robdit]

@yanweiguo Yes, thank you for the explanation. Does that mean that the cloud run being created but not in terraform state is expected behavior?

[yanweiguo]

Does that mean that the cloud run being created but not in terraform state is expected behavior?

Cloud Run resource being created but terraform reports an error for the creation is expected behavior.

[robdit]

What should the terraform state look like in this case then? When this happens the cloud run job exists in GCP but it not in the terraform state.

[yanweiguo]

heck the secret since it requires another API call which will significantly increase create API latency. If the request passes the verifications, a cloud run resource will be crea

The issue should be fixed by GoogleCloudPlatform/magic-modules#10298

[AmilKey]

any updates?