Unexpected behaviour with google_storage_bucket_object. Cannot access content of the file even though it shows up in state and output

[periwinkleFTW]

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request.
• Please do not leave +1 or me too comments, they generate extra noise for issue followers and do not help prioritize the request.
• If you are interested in working on this issue or have submitted a pull request, please leave a comment.
• If an issue is assigned to the modular-magician user, it is either in the process of being autogenerated, or is planned to be autogenerated soon. If an issue is assigned to a user, that user is claiming responsibility for the issue. If an issue is assigned to hashibot, a community member has claimed the issue already.

Terraform Version

Affected Resource(s)

google_storage_bucket_object
local_file
google_*

Terraform Configuration Files

# Copy-paste your Terraform configurations here.
#
# For large Terraform configs, please use a service like Dropbox and share a link to the ZIP file.
# For security, you can also encrypt the files using our GPG public key:
#    https://www.hashicorp.com/security
#
# If reproducing the bug involves modifying the config file (e.g., apply a config,
# change a value, apply the config again, see the bug), then please include both:
# * the version of the config before the change, and
# * the version of the config after the change.


# I have problems with a yaml file, so you might need to upload an example yaml to the bucket

resource "google_storage_bucket" "config_bucket" {
  force_destroy = true
  location      = var.location
  name          = "config_bucket"
}

data "google_storage_bucket_object_content" "my_file" {
  bucket     = google_storage_bucket.config_bucket.name
  name       = "my_file.yaml"
}

# This one will not work and will return null. Interpolation of this resource using .content also returns null
# in other resources
output "file_object_content" {
  value = data.google_storage_bucket_object_content.my_file.content
}


# This one will work and it will output bucket, content, id, and name.
output "file_object" {
  value = data.google_storage_bucket_object_content.my_file
}

### Output
#file_object = {
#        bucket  = "config_bucket"
#       content = <<-EOT
#          ... multiline yaml stuff ....
#        EOT
#        id      = "config_bucket/my_file.yaml"
#        name    = "my_file.yaml"
#    }

Debug Output

Changes to Outputs:

• object = {
  • bucket = "api-config-files-3ccf"
  • content = null
  • name = "configs/partner-aggregator-api.yaml"
    }
• object_bucket = "andrey-dev-api-config-files-3ccf"
  ~ object_id = "api-config-files-3ccf-configs/spending-api.yaml" -> (known after apply)

Panic Output

Expected Behavior

I expect terraform to be able to interpolate the data obtained from the specified object.
Using the output with google_storage_bucket_object or inspecting state of this object (terraform state show google_storage_bucket_object.my_object) show that resource has 4 attributes (bucket, content, id, name).

Actual Behavior

Trying to access the content of the google_storage_bucket_object returns null, however other attributes like bucket, name, and id are accessible. Running output on the whole object shows content is there.

Steps to Reproduce

1. terraform apply
2. upload test yaml file to the created bucket
3. try to use the contents of the file

Important Factoids

The issue could be due to multiline nature of yaml files. Same behaviour with json
Configuration uses Terraform Cloud.

Deployed both from intel macos, does not work
Deployed from m1 macos, it worked once but then content was null afterwards

References

https://registry.terraform.io/providers/hashicorp/google/latest/docs/data-sources/storage_bucket_object_content

• #0000

[edwardmedia]

@periwinkleFTW using your code, I am receiving below content of a object.

file_object_content = <<EOT
#Comment: This is a supermarket list using YAML
#Note that - character represents the list
---
food: 
  - vegetables: tomatoes #first list item
  - fruits: #second list item
      citrics: oranges 
      tropical: bananas
      nuts: peanuts
      sweets: raisins
EOT

Do you want to share your debug log so I can take a closer look?

[periwinkleFTW]

@edwardmedia
I set the debug, but the logs aren't being saved on my local machine. Maybe it's because I am using tf cloud.

Try using the content as an input in another resource. For example I want to use it to download the yaml file with api definition to deploy API gateway (I wish i could point google_api_gateway_api_config to the bucket directly, but oh well).
First I get the object using google_storage_bucket_object_content, then I attempt to save it locally using local_file.

This is the yaml file I am using:

components:
  securitySchemes:
    token_auth:
      description: Firebase Bearer Token Authentication
      scheme: bearer
      type: http
info:
  description: "Welcome to the documentation for the test API!\nThis\
    \ project is written in Python, with the\n[Flask](https://flask.palletsprojects.com/)\
    \ web framework. This documentation\nis generated automatically from the\n[project's\
    \ source code](https://github.com/testorg/my-api) using\n\
    the [APIFairy](https://github.com/miguelgrinberg/apifairy) Flask extension.\n\
    ## Introduction\ntest API is an easy to use web API for test which\
    \ aggreagates spending data by partner\n\ntest API provides the\
    \ following features\n- Get List of Partners with test for each platform\n\
    \n\n## Configuration\nIf you are running test API yourself while\
    \ developing your front end,\nthere are a number of environment variables that\
    \ you can set to configure its\nbehavior. The variables can be defined directly\
    \ in the environment or in a\n`.env` file in the project directory. The following\
    \ table lists all the\nenvironment variables that are currently used:\n\n| Environment\
    \ Variable | Default | Description |\n| - | - | - |\n| `DISABLE_AUTH` | False\
    \ | Whether to disable authentication |\n| `USE_CORS` | `yes` | Whether to allow\
    \ cross-origin requests. If allowed, CORS support can be configured or customized\
    \ with options provided by the Flask-CORS extension. |\n\n## Authentication\n\
    The authentication flow for this API is based on firebase tokens.\n\nMost endpoints\
    \ in this API are authenticated with the access token, passed\nin the `Authorization`\
    \ header, using the `Bearer` scheme.\nAccess tokens are valid for 15 minutes (by\
    \ default) from the time they are\nissued. When the access token is expired, the\
    \ client can renew it using the\nrefresh token. \n\nAll authentication failures\
    \ are handled with a `401` status code in the\nresponse.\n\n## Errors\nAll errors\
    \ returned by this API use the following JSON structure:\n```json\n{\n    \"code\"\
    : <numeric error code>,\n    \"message\": <short error message>,\n    \"description\"\
    : <longer error description>,\n}\n```\nIn the case of schema validation errors,\
    \ an `errors` property is also returned,\ncontaining a detailed list of validation\
    \ errors found in the submitted request:\n```json\n{\n    \"code\": <error code>,\n\
    \    \"message\": <error message>,\n    \"description\": <error description>,\n\
    \    \"errors\": [ <error details>, ... ]\n}\n```"
  title: test API
  version: "1.0"
openapi: 3.0.3
paths:
  /api/summary:
    get:
      operationId: partners_summary
      parameters: []
      responses:
        "204":
          description: No Content
        "400":
          description: "Invalid Request Parameters: <message>"
      security:
        - token_auth: []
      summary: Retrieve all aggregated partners
      tags:
        - Partners
servers:
  - url: https://my-url-uc.a.run.app/
tags:
  - description:
      "This is the people module and supports all the REST actions for the

      partners data"
    name: Partners

When I run these resources:

resource "local_file" "test_file" {
  content = data.google_storage_bucket_object_content. my_file.content
  # content  = "foo"
  filename = "${path.module}/foo.bar"
}
### Error output shows that content is empty or null
# │ Error: Invalid Attribute Combination
#│ 
#│   with local_file.test_file,
#│   on main.tf line 258, in resource "local_file" "test_file":
#│  258:   content = data.google_storage_bucket_object_content.my_file.content
#│ 
#│ No attribute specified when one (and only one) of
#│ [sensitive_content,content_base64,source] is required

output "file_contents" {
  value = local_file.test_file
}
## Output
file_contents = {
        bucket  = "my-bucket"
        content = <<-EOT
            components:
  securitySchemes:
    token_auth:
      description: Firebase Bearer Token Authentication
      scheme: bearer
      type: http
info:
  description: "Welcome to the documentation for the test API!\nThis\
    \ project is written in Python, with the\n[Flask](https://flask.palletsprojects.com/)\
    \ web framework. This documentation\nis generated automatically from the\n[project's\
    \ source code](https://github.com/testorg/my-api) using\n\
    the [APIFairy](https://github.com/miguelgrinberg/apifairy) Flask extension.\n\
    ## Introduction\ntest API is an easy to use web API for test which\
    \ aggreagates spending data by partner\n\ntest API provides the\
    \ following features\n- Get List of Partners with test for each platform\n\
    \n\n## Configuration\nIf you are running test API yourself while\
    \ developing your front end,\nthere are a number of environment variables that\
    \ you can set to configure its\nbehavior. The variables can be defined directly\
    \ in the environment or in a\n`.env` file in the project directory. The following\
    \ table lists all the\nenvironment variables that are currently used:\n\n| Environment\
    \ Variable | Default | Description |\n| - | - | - |\n| `DISABLE_AUTH` | False\
    \ | Whether to disable authentication |\n| `USE_CORS` | `yes` | Whether to allow\
    \ cross-origin requests. If allowed, CORS support can be configured or customized\
    \ with options provided by the Flask-CORS extension. |\n\n## Authentication\n\
    The authentication flow for this API is based on firebase tokens.\n\nMost endpoints\
    \ in this API are authenticated with the access token, passed\nin the `Authorization`\
    \ header, using the `Bearer` scheme.\nAccess tokens are valid for 15 minutes (by\
    \ default) from the time they are\nissued. When the access token is expired, the\
    \ client can renew it using the\nrefresh token. \n\nAll authentication failures\
    \ are handled with a `401` status code in the\nresponse.\n\n## Errors\nAll errors\
    \ returned by this API use the following JSON structure:\n```json\n{\n    \"code\"\
    : <numeric error code>,\n    \"message\": <short error message>,\n    \"description\"\
    : <longer error description>,\n}\n```\nIn the case of schema validation errors,\
    \ an `errors` property is also returned,\ncontaining a detailed list of validation\
    \ errors found in the submitted request:\n```json\n{\n    \"code\": <error code>,\n\
    \    \"message\": <error message>,\n    \"description\": <error description>,\n\
    \    \"errors\": [ <error details>, ... ]\n}\n```"
  title: test API
  version: "1.0"
openapi: 3.0.3
paths:
  /api/summary:
    get:
      operationId: partners_summary
      parameters: []
      responses:
        "204":
          description: No Content
        "400":
          description: "Invalid Request Parameters: <message>"
      security:
        - token_auth: []
      summary: Retrieve all aggregated partners
      tags:
        - Partners
servers:
  - url: https://my-url-uc.a.run.app/
tags:
  - description:
      "This is the people module and supports all the REST actions for the

      partners data"
    name: Partners
        EOT
        id      = "my-bucket-configs/test_file.yaml"
        name    = "configs/test_file.yaml"
    }




output "file_contents" {
  value =  local_file.test_file.content
}
# Empty

I get an error even if you try to interpolate it. I am so confused but I do not see any other way I can deploy API gateway with the yaml file stored in a bucket =(

[periwinkleFTW]

If there is another way to deploy an API gateway using a yaml file stored in a bucket that works with Terraform Cloud and does not rely on google_storage_bucket_object_content I would be on cloud9 .

Could it be a problem with my yaml file?

[periwinkleFTW]

I tried it with your yaml: and the same behaviour. Also same behaviour with json. However, I have no problem accessing attributes like bucket or id for the object_content object. But content is empty. I wonder why

#Comment: This is a supermarket list using YAML
#Note that - character represents the list
---
food:
  - vegetables: tomatoes #first list item
  - fruits: #second list item
      citrics: oranges
      tropical: bananas
      nuts: peanuts
      sweets: raisins

terraform -v

Terraform v1.4.6
on darwin_amd64
+ provider registry.terraform.io/hashicorp/archive v2.3.0
+ provider registry.terraform.io/hashicorp/google v4.59.0
+ provider registry.terraform.io/hashicorp/google-beta v4.59.0
+ provider registry.terraform.io/hashicorp/local v2.4.0
+ provider registry.terraform.io/hashicorp/random v3.4.3
+ provider registry.terraform.io/hashicorp/time v0.9.1

Is there a way to the get the contents of a file using data.google_storage_object?
I also upgraded terraform (both local and tf cloud) to 1.4.6.
Ran gcloud components update to get the latest gcloud cli

tf resoures:

data "google_storage_bucket_object_content" "api_config_yaml_object_content" {
  depends_on = [module.api_config_generator]
  bucket     = module.storage.api_config_bucket.name
  name       = "configs/test1.yaml"
}
output "object" {
  value = data.google_storage_bucket_object_content.api_config_yaml_object_content
}
output "object_contents" {
  value = data.google_storage_bucket_object_content.api_config_yaml_object_content.content
}
output "object_bucket" {
  value = data.google_storage_bucket_object_content.api_config_yaml_object_content.bucket
}
output "object_id" {
  value = data.google_storage_bucket_object_content.api_config_yaml_object_content.id
}

Terraform plan and apply:

 # data.google_storage_bucket_object_content.api_config_yaml_object_content will be read during apply
  # (depends on a resource or a module with changes pending)
 <= data "google_storage_bucket_object_content" "api_config_yaml_object_content" {
      + bucket              = "andrey-dev-api-config-files-25a3"
      + cache_control       = (known after apply)
      + content_disposition = (known after apply)
      + content_encoding    = (known after apply)
      + content_language    = (known after apply)
      + content_type        = (known after apply)
      + crc32c              = (known after apply)
      + customer_encryption = (known after apply)
      + detect_md5hash      = (known after apply)
      + event_based_hold    = (known after apply)
      + id                  = (known after apply)
      + kms_key_name        = (known after apply)
      + md5hash             = (known after apply)
      + media_link          = (known after apply)
      + metadata            = (known after apply)
      + name                = "configs/test1.yaml"
      + output_name         = (known after apply)
      + self_link           = (known after apply)
      + source              = (known after apply)
      + storage_class       = (known after apply)
      + temporary_hold      = (known after apply)
    }



Plan: 2 to add, 1 to change, 1 to destroy.

Changes to Outputs:
  ~ object          = {
      + cache_control       = (known after apply)
      ~ content             = <<-EOT
            #Comment: This is a supermarket list using YAML
            #Note that - character represents the list
            ---
            food:
              - vegetables: tomatoes #first list item
              - fruits: #second list item
                  citrics: oranges
                  tropical: bananas
                  nuts: peanuts
                  sweets: raisins
        EOT -> null
      + content_disposition = (known after apply)
      + content_encoding    = (known after apply)
      + content_language    = (known after apply)
      + content_type        = (known after apply)
      + crc32c              = (known after apply)
      + customer_encryption = (known after apply)
      + detect_md5hash      = (known after apply)
      + event_based_hold    = (known after apply)
      ~ id                  = "andrey-dev-api-config-files-25a3-configs/test1.yaml" -> (known after apply)
      + kms_key_name        = (known after apply)
      + md5hash             = (known after apply)
      + media_link          = (known after apply)
      + metadata            = (known after apply)
        name                = "configs/test1.yaml"
      + output_name         = (known after apply)
      + self_link           = (known after apply)
      + source              = (known after apply)
      + storage_class       = (known after apply)
      + temporary_hold      = (known after apply)
        # (1 unchanged attribute hidden)
    }
  + object_bucket   = "andrey-dev-api-config-files-25a3"
  + object_id       = (known after apply)

module.api_config_generator.google_storage_bucket_object.params_py_file: Destroying... [id=andrey-dev-api-params-25a3-params/params.py]
module.firebase.google_identity_platform_project_default_config.firebase_auth: Modifying... [id=andrey-dev-project-25a3]
module.api_config_generator.google_storage_bucket_object.params_py_file: Destruction complete after 0s
module.api_config_generator.local_file.params: Creating...
module.api_config_generator.local_file.params: Creation complete after 0s [id=9b0b46a92f85438e54680a80c027d1a391e810ab]
module.api_config_generator.google_storage_bucket_object.params_py_file: Creating...
module.firebase.google_identity_platform_project_default_config.firebase_auth: Modifications complete after 0s [id=andrey-dev-project-25a3]
module.api_config_generator.google_storage_bucket_object.params_py_file: Creation complete after 1s [id=andrey-dev-api-params-25a3-params/params.py]
data.google_storage_bucket_object_content.api_config_yaml_object_content: Refreshing...
data.google_storage_bucket_object_content.api_config_yaml_object_content: Refresh complete after 0s [id=andrey-dev-api-config-files-25a3-configs/test1.yaml]

Apply complete! Resources: 2 added, 1 changed, 1 destroyed.

Outputs:
object_id = "andrey-dev-api-config-files-25a3-configs/test1.yaml"
object = {
        bucket  = "andrey-dev-api-config-files-25a3"
        content = <<-EOT
            #Comment: This is a supermarket list using YAML
            #Note that - character represents the list
            ---
            food:
              - vegetables: tomatoes #first list item
              - fruits: #second list item
                  citrics: oranges
                  tropical: bananas
                  nuts: peanuts
                  sweets: raisins
        EOT
        id      = "andrey-dev-api-config-files-25a3-configs/test1.yaml"
        name    = "configs/test1.yaml"
    }
object_bucket = "andrey-dev-api-config-files-25a3"

I hope that helps

[edwardmedia]

@periwinkleFTW using below config, I see foo.bar is successfully created and its content is the same as sample.yaml.

data "google_storage_bucket_object_content" "my_file" {
  bucket     = "issue14666"
  name       = "sample.yaml"
}

resource "local_file" "test_file" {
  content = data.google_storage_bucket_object_content.my_file.content
  # content  = "foo"
  filename = "${path.module}/foo.bar"
}

Not sure what the problem was with you

[periwinkleFTW]

@edwardmedia
I think the issue might be in my yaml files. They could be corrupted. I am reworking their generation to see if that is the issue. I have local_file and variable interpolation in my config and they work like a charm, it's only one instance of of object_content with yaml files that gives me trouble.
I will provide an update as soon as I am done with refactoring.
Thank you very much for looking into this issue and being so fast with your replies. I really appreciate it =)

[edwardmedia]

@periwinkleFTW Glad I can hep. Yes, keep us updated.

[periwinkleFTW]

@edwardmedia Everything works as expected if I run storage_object and local_file from root. It does not show content if those resources are inside the module. No idea why, but I will keep it as is.

I have another issue with google_api_gateway_api_config. I relies on the dynamically generated yaml file, the problem is that using this dynamically generated variable with interpolation in openapi_documents block for api config forces terraform to evaluate 'local_file` too early. I tried using try()/can()/coalesce() to put dummy data in there until apply goes through and populates local file with correct data. It throws an error:

│ Error: Provider produced inconsistent final plan
│ 
│ When expanding the plan for local_file.local_api_config_file to include new
│ values learned so far during apply, provider
│ "registry.terraform.io/hashicorp/local" produced an invalid new value for
│ .content: was cty.StringVal("dummy_string"), but now
│ cty.StringVal("basePath: /\ndefinitions:\n  accountsCmAccount:\n
│ properties:\n      account_id:\n        type: integer\n
...
│ 
│ This is a bug in the provider, which should be reported in the provider's
│ own issue tracker.
╵
Operation failed: failed running terraform apply (exit 1)

Here are the resources:

ata "google_storage_bucket_object_content" "api_config_yaml_object_content" {
  depends_on = [module.api_config_generator, time_sleep.wait_for_api_config_file]
  bucket     = module.storage.api_spec_bucket.name
  name       = "api_config/openapi2-run.yaml"
}

locals {
  check_content = can(data.google_storage_bucket_object_content.api_config_yaml_object_content.content) ? data.google_storage_bucket_object_content.api_config_yaml_object_content.content : "waiting_for_api_spec"
}

resource "local_file" "local_api_config_file" {
  depends_on = [module.api_config_generator, data.google_storage_bucket_object_content.api_config_yaml_object_content]
#   content    = coalesce(data.google_storage_bucket_object_content.api_config_yaml_object_content.content, "waiting_for_data")
  content    = coalesce(local.check_content, "dummy_string")
  filename   = "modules/api_gateway/openapi2-run.yaml"
}

resource "google_api_gateway_api_config" "seeder_api_gw_config" {
  depends_on    = [google_api_gateway_api.api]
  provider      = google-beta
  api           = google_api_gateway_api.api.api_id
  api_config_id = "${var.client}-${var.environment}-api-cfg-${var.random_id}"
  project       = var.project
  openapi_documents {
    document {
        path     = "modules/api_gateway/openapi2-run.yaml"
        contents = base64encode(var.local_api_config_file)
    }
  }
  lifecycle {
    create_before_destroy = true
  }
}

Depends_on does not seem to work either =( API gateway will be the death of me

[edwardmedia]

@periwinkleFTW the resource appears working as expected. You may reach out to the Terraform to see if they can help on module stuff. Closing this issue now.

[github-actions]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.
If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.