Error using existing S3 bucket in 0.59.0

[pearsonhenri]

Found a bug? Maybe our Slack Community can help.

Describe the Bug

Using 0.59.0 of this module, if I specify an existing S3 bucket (which also has an existing bucket policy) and run terraform apply, the operation fails with

Error: Error creating S3 bucket: BucketAlreadyExists: The requested bucket name is not available. The bucket namespace is shared by all users of the system. Please select a different name and try again.
	status code: 409, request id: 3PMBHS08MF98NFQW, host id: sLfaifwFHwD9biX57oJgzueRkMuCBARUTwoawf2MQaJACVEEQ9m863hkwH781UuFJ0klApMABzU=

  on .terraform/modules/inspetor.inspetor_mongo_staging.logs/main.tf line 1, in resource "aws_s3_bucket" "default":
   1: resource "aws_s3_bucket" "default" {



Error: Error putting S3 policy: InvalidParameter: 1 validation error(s) found.
- minimum field size of 1, PutBucketPolicyInput.Bucket.


  on .terraform/modules/inspetor.inspetor_mongo_staging/main.tf line 79, in resource "aws_s3_bucket_policy" "default":
  79: resource "aws_s3_bucket_policy" "default" {

I tried re-running terraform apply thereafter, but then I get a rather inscrutable error:

Error: origin.0.origin_id must not be empty, got

  on .terraform/modules/inspetor.inspetor_mongo_staging/main.tf line 194, in resource "aws_cloudfront_distribution" "default":
 194: resource "aws_cloudfront_distribution" "default" {

(I'm assuming that means an empty string is being passed for origin_id)

Expected Behavior

Well... it should work? 😅

Steps to Reproduce

Steps to reproduce the behavior:

1. Create an S3 bucket
2. Instantiate this module via

module "my_module" {
  source                        = "cloudposse/cloudfront-s3-cdn/aws"
  version                       = "0.59.0"

  origin_bucket                 = "my.bucket.name"
  override_origin_bucket_policy = false
  aliases                       = ["my.alias.net"]
  dns_alias_enabled             = true
  parent_zone_id                = var.my_zone_id

  acm_certificate_arn           = var.my_acm_arn
}

3. Run terraform apply
4. See error

Environment (please complete the following information):

terraform --version
Terraform v0.13.5
+ provider registry.terraform.io/hashicorp/archive v2.1.0
+ provider registry.terraform.io/hashicorp/aws v3.35.0
+ provider registry.terraform.io/hashicorp/local v2.1.0
+ provider registry.terraform.io/hashicorp/random v3.1.0
+ provider registry.terraform.io/hashicorp/template v2.2.0
+ provider registry.terraform.io/hashicorp/vault v2.18.0
+ provider registry.terraform.io/mongodb/mongodbatlas v0.8.2
+ provider registry.terraform.io/terraform-providers/docker v2.7.2

[jamengual]

@syphernl may be related to your PR?

[syphernl]

@jamengual @pearsonhenri ISTM that the logging bucket already exists, and that Terraform errors on that, since it refers to inspetor.inspetor_mongo_staging.logs.
This module only handles an existing origin bucket, for logging one will be generated based on the tags.

[pearsonhenri]

Thanks for the replies guys. @syphernl I don't think it's that, because the bucket currently has access logging disabled. I can enable it, but then I need to specify a logging bucket, which from what your saying sounds like Terraform won't properly handle?

[pearsonhenri]

Unless you're suggesting that Terraform is trying to create a logging bucket called inspetor.inspetor_mongo_staging.logs, but it already exists (it does not exist yet).

[pearsonhenri]

Ooph but yeah I'm kinda stumped here, I can't even interact with this module now because Terraform errors in the planning phase now with the whole

Error: origin.0.origin_id must not be empty, got

  on .terraform/modules/inspetor.inspetor_mongo_staging/main.tf line 194, in resource "aws_cloudfront_distribution" "default":
 194: resource "aws_cloudfront_distribution" "default" {

thing

[syphernl]

Unless you're suggesting that Terraform is trying to create a logging bucket called inspetor.inspetor_mongo_staging.logs, but it already exists (it does not exist yet).

The particular ID I mentioned is the module name you are using with .logs appendage, which refers to the logs "module" reference within the s3-cdn module.
As your example doesn't pass in any context (tags etc) it would likely try to generate a bucket named logs rather than my-cool-app-prod-logs.
S3 buckets need to be globally unique, if it tries to create a bucket that already exists it will fail.

There are two ways of fixing this:

• Pass along the a context and to this resource.
• Disable CloudFront access logging by setting logging_enabled to false.

[pearsonhenri]

Ah, cool--ok I tried setting logging_enabled to false and recreating the module, however my applys are still failing with

Error: origin.0.origin_id must not be empty, got

  on .terraform/modules/inspetor.inspetor_mongo_staging/main.tf line 194, in resource "aws_cloudfront_distribution" "default":
 194: resource "aws_cloudfront_distribution" "default" {

So maybe that's a separate issue?

[syphernl]

Can you verify whether it works in the previous version, v0.58.1?

[pearsonhenri]

Yeah actually when I recreated just now it was on 0.58.1 and I still hit this error

[pearsonhenri]

So just to be clear, my updated configuration looks like this:

module "my_module" {
  source                        = "cloudposse/cloudfront-s3-cdn/aws"
  version                       = "0.58.1"

  origin_bucket                 = "my.s3.bucket"
  override_origin_bucket_policy = false
  aliases                       = ["my.cname.net"]
  dns_alias_enabled             = true
  logging_enabled               = false
  parent_zone_id                = var.my_zone_id

  acm_certificate_arn           = var.my_acm_arn
}

and applying this configuration results in the mentioned error and even prevents me from being able to run terraform plan subsequently

[syphernl]

ISTM the origin_id is based on the context, something like this should probably make it work:

module "my_module" {
  source                        = "cloudposse/cloudfront-s3-cdn/aws"
  version                       = "0.58.1"
  
  origin_bucket                 = "my.s3.bucket"
  override_origin_bucket_policy = false
  aliases                       = ["my.cname.net"]
  dns_alias_enabled             = true
  logging_enabled               = false
  parent_zone_id                = var.my_zone_id

  acm_certificate_arn           = var.my_acm_arn
  
  name                          = "cdn"
  stage                         = "prod"
  namespace                     = "eg"
}

[pearsonhenri]

@syphernl nice, looks like that was indeed the problem! Would be great if you could include that in the docs--the README lists an example like this for importing an existing S3 bucket:

module "cdn" {
  source = "cloudposse/cloudfront-s3-cdn/aws"
  # Cloud Posse recommends pinning every module to a specific version
  # version     = "x.x.x"

  origin_bucket     = "eg-prod-app"
  aliases           = ["assets.cloudposse.com"]
  dns_alias_enabled = true
  parent_zone_name  = "cloudposse.com"
}

(without name/stage/namespace args), which is what got me down this wayward path.

[pearsonhenri]

Ooph, one more problem @syphernl: I removed the override_origin_bucket_policy = false setting so that the module could properly grant access for the Cloudfront distribution to the S3 bucket, but then I run into this issue:

Error: Error putting S3 policy: InvalidParameter: 1 validation error(s) found.
- minimum field size of 1, PutBucketPolicyInput.Bucket.


  on .terraform/modules/inspetor.inspetor_mongo_staging/main.tf line 79, in resource "aws_s3_bucket_policy" "default":
  79: resource "aws_s3_bucket_policy" "default" {

Seems like my provided bucket name isn't getting passed along somewhere?

[pearsonhenri]

Hi there--any update on the above? I'm manually copy-pasting a bucket policy to work around this for now

[syphernl]

@pearsonhenri Another issue has been created for that with a PR to fix it.

[richstokes]

Still running into this issue on 0.75.0. What is a known-good example config for setting up a CDN with an existing bucket?

[CodechCFA]

Oof. Over a year and a half later, that same code snippet in the readme sent me down a rabbit hole with this same error message. Adding name fixed the problem.