S3 Bucket Lifecycle always prompts a change in plan

[zerfahp]

I have a module that creates s3 buckets. The module contains 4 lifecycle policies. 1 is enabled by default and 3 disabled by default. To be extra cautious I set the expiry days very high so that if the rule is enabled by accident and no expiry is then objects are no transitioned or expired for a long time.

The module looks like this....

#create the bucket
resource "aws_s3_bucket" "private_bucket" {
  bucket = "${local.bucket_id}"
  acl    = "${var.acl}"
  policy = "${data.aws_iam_policy_document.policy.json}"
  tags   = "${var.tags}"

  logging {
    target_bucket = "${local.logging_bucket_id}"
    target_prefix = "${local.bucket_id}/"
  }

  versioning {
    enabled    = "${var.enable_versioning}"
    mfa_delete = "${var.mfa_delete}"
  }

  #lifecycle rules for non-current versions (defaults to on)
  lifecycle_rule {
    enabled = "${var.enable_lifecycle}"
    id      = "default"

    abort_incomplete_multipart_upload_days = 14

    transition {
      days          = "${var.current_ia_transition_days}"
      storage_class = "STANDARD_IA"
    }

    transition {
      days          = "${var.current_glacier_transition_days}"
      storage_class = "GLACIER"
    }

    expiration {
      expired_object_delete_marker = true
      days          = "${var.current_version_expiration_days}"
    }

    noncurrent_version_transition {
      days          = "${var.noncurrent_ia_transition_days}"
      storage_class = "STANDARD_IA"
    }

    noncurrent_version_transition {
      days          = "${var.noncurrent_glacier_transition_days}"
      storage_class = "GLACIER"
    }

    noncurrent_version_expiration {
      days = "${var.noncurrent_version_expiration_days}"
    }

  }

  #optional extra lifeycle rule 1
  lifecycle_rule {
    enabled = "${var.enable_extra_lifecycle_1}"
    id      = "${var.extra_lifecycle_1_id}"

    abort_incomplete_multipart_upload_days = 14

    prefix = "${var.extra_lifecycle_1_prefix}"

    transition {
      days          = "${var.extra_lifecycle_1_current_ia_transition_days}"
      storage_class = "STANDARD_IA"
    }

    transition {
      days          = "${var.extra_lifecycle_1_current_glacier_transition_days}"
      storage_class = "GLACIER"
    }

    expiration {
      expired_object_delete_marker = true
      days          = "${var.extra_lifecycle_1_current_version_expiration_days}"
    }

    noncurrent_version_transition {
      days          = "${var.extra_lifecycle_1_noncurrent_ia_transition_days}"
      storage_class = "STANDARD_IA"
    }

    noncurrent_version_transition {
      days          = "${var.extra_lifecycle_1_noncurrent_glacier_transition_days}"
      storage_class = "GLACIER"
    }

    noncurrent_version_expiration {
      days          = "${var.extra_lifecycle_1_noncurrent_version_expiration_days}"
    }

  }

  #optional extra lifeycle rule 2
  lifecycle_rule {
    enabled = "${var.enable_extra_lifecycle_2}"
    id      = "${var.extra_lifecycle_2_id}"

    abort_incomplete_multipart_upload_days = 14

    prefix = "${var.extra_lifecycle_2_prefix}"

    transition {
      days          = "${var.extra_lifecycle_2_current_ia_transition_days}"
      storage_class = "STANDARD_IA"
    }

    transition {
      days          = "${var.extra_lifecycle_2_current_glacier_transition_days}"
      storage_class = "GLACIER"
    }

    expiration {
      expired_object_delete_marker = true
      days          = "${var.extra_lifecycle_2_current_version_expiration_days}"
    }

    noncurrent_version_transition {
      days          = "${var.extra_lifecycle_2_noncurrent_ia_transition_days}"
      storage_class = "STANDARD_IA"
    }

    noncurrent_version_transition {
      days          = "${var.extra_lifecycle_2_noncurrent_glacier_transition_days}"
      storage_class = "GLACIER"
    }

    noncurrent_version_expiration {
      days          = "${var.extra_lifecycle_2_noncurrent_version_expiration_days}"
    }

  }

  #optional extra lifeycle rule 3
  lifecycle_rule {
    enabled = "${var.enable_extra_lifecycle_3}"
    id      = "${var.extra_lifecycle_3_id}"

    abort_incomplete_multipart_upload_days = 14

    prefix = "${var.extra_lifecycle_3_prefix}"

    transition {
      days          = "${var.extra_lifecycle_3_current_ia_transition_days}"
      storage_class = "STANDARD_IA"
    }

    transition {
      days          = "${var.extra_lifecycle_3_current_glacier_transition_days}"
      storage_class = "GLACIER"
    }

    expiration {
      expired_object_delete_marker = true
      days          = "${var.extra_lifecycle_3_current_version_expiration_days}"
    }

    noncurrent_version_transition {
      days          = "${var.extra_lifecycle_3_noncurrent_ia_transition_days}"
      storage_class = "STANDARD_IA"
    }

    noncurrent_version_transition {
      days          = "${var.extra_lifecycle_3_noncurrent_glacier_transition_days}"
      storage_class = "GLACIER"
    }

    noncurrent_version_expiration {
      days          = "${var.extra_lifecycle_3_noncurrent_version_expiration_days}"
    }

  }

  #use the kms sse key for default encryption
  server_side_encryption_configuration {
    rule {
      apply_server_side_encryption_by_default {
        kms_master_key_id = "${module.kms_sse_key.key_id}"
        sse_algorithm     = "aws:kms"
      }
    }
  }
}

The Variables for lifecycles are all set similar to this (I wont paste them all as there are loads)

variable "extra_lifecycle_1_prefix" {
  description = "Prefix to discrimate which objects the rule applies to"
  default     = "extra_1/"
}

#current version lifecycle variables (default to a very long time)
variable "extra_lifecycle_1_current_ia_transition_days" {
  description = "Number of days after which to move current objects to to IA storage tier"
  default     = "9900"
}

variable "extra_lifecycle_1_current_glacier_transition_days" {
  description = "Number of days after which to move current version objects to to glacier storage tier"
  default     = "9950"
}

variable "extra_lifecycle_1_current_version_expiration_days" {
  description = "Number of days after which to expire current versions"
  default     = "9999"
}

#non-current version lifecycle variables (enabled by default)
variable "extra_lifecycle_1_noncurrent_ia_transition_days" {
  description = "Number of days after which to move non-current objects to to IA storage tier"
  default     = "30"
}

variable "extra_lifecycle_1_noncurrent_glacier_transition_days" {
  description = "Number of days after which to move non-current objects to to glacier storage tier"
  default     = "90"
}

variable "extra_lifecycle_1_noncurrent_version_expiration_days" {
  description = "Number of days after which to expire"
  default     = "365"
}

Every time I run a plan, even though I'm not changing the buckets at all it says it's going to change the lifecycle rule.

For example:

  ~ module.tf_module_sagemaker_logs_bucket.aws_s3_bucket.private_bucket
      lifecycle_rule.0.expiration.1717063568.date:                         "" => ""
      lifecycle_rule.0.expiration.1717063568.days:                         "" => "99999"
      lifecycle_rule.0.expiration.1717063568.expired_object_delete_marker: "" => "true"
      lifecycle_rule.0.expiration.66073835.date:                           "" => ""
      lifecycle_rule.0.expiration.66073835.days:                           "99999" => "0"
      lifecycle_rule.0.expiration.66073835.expired_object_delete_marker:   "false" => "false"
      lifecycle_rule.1.expiration.1717063568.date:                         "" => ""
      lifecycle_rule.1.expiration.1717063568.days:                         "" => "99999"
      lifecycle_rule.1.expiration.1717063568.expired_object_delete_marker: "" => "true"
      lifecycle_rule.1.expiration.66073835.date:                           "" => ""
      lifecycle_rule.1.expiration.66073835.days:                           "99999" => "0"
      lifecycle_rule.1.expiration.66073835.expired_object_delete_marker:   "false" => "false"
      lifecycle_rule.2.expiration.1717063568.date:                         "" => ""
      lifecycle_rule.2.expiration.1717063568.days:                         "" => "99999"
      lifecycle_rule.2.expiration.1717063568.expired_object_delete_marker: "" => "true"
      lifecycle_rule.2.expiration.66073835.date:                           "" => ""
      lifecycle_rule.2.expiration.66073835.days:                           "99999" => "0"
      lifecycle_rule.2.expiration.66073835.expired_object_delete_marker:   "false" => "false"
      lifecycle_rule.3.expiration.1717063568.date:                         "" => ""
      lifecycle_rule.3.expiration.1717063568.days:                         "" => "99999"
      lifecycle_rule.3.expiration.1717063568.expired_object_delete_marker: "" => "true"
      lifecycle_rule.3.expiration.66073835.date:                           "" => ""
      lifecycle_rule.3.expiration.66073835.days:                           "99999" => "0"
      lifecycle_rule.3.expiration.66073835.expired_object_delete_marker:   "false" => "false"

I cannot for the life of me figure out why. I'm not sure if this is a bug or if I have some syntax wrong.

[anouvel]

Hi @zerfahp ,

I've investigated a bit about your issue, and after reading the SDK documentation, it appears that there is an incompatibility between expiration.date or expiration.day and expiration.expired_object_delete_marker (https://github.com/aws/aws-sdk-go/blob/master/service/s3/api.go#L15372-L15388):

type LifecycleExpiration struct {

    // Indicates at what date the object is to be moved or deleted. Should be in
    // GMT ISO 8601 Format.
    Date *time.Time `type:"timestamp" timestampFormat:"iso8601"`

    // Indicates the lifetime, in days, of the objects that are subject to the rule.
    // The value must be a non-zero positive integer.
    Days *int64 `type:"integer"`

    // Indicates whether Amazon S3 will remove a delete marker with no noncurrent
    // versions. If set to true, the delete marker will be expired; if set to false
    // the policy takes no action. This cannot be specified with Days or Date in
    // a Lifecycle Expiration Policy.
    ExpiredObjectDeleteMarker *bool `type:"boolean"`
    // contains filtered or unexported fields
}

So just don't use both day or date and expired_object_delete_marker (even if you set it to true or false). By the way, this behaviour should be prevented by Terraform itself!

Adrien

[zerfahp]

Thanks so much for investigating. I really appreciate it. I will try this as soon as my terraform state has been repaired from another issue :-)

[tracypholmes]

Thank you for using Terraform and for opening up this question, @zerfahp! Issues on GitHub are intended to be related to bugs or feature requests with the provider codebase.

It looks as though you've received a solution to your issue. If needed, please use https://discuss.hashicorp.com/c/terraform-providers for additional feedback, community discussions, and questions around Terraform.

If you believe that your issue was miscategorized as a question or closed in error, please create a new issue using one of the following provided templates: bug report or feature request. Please make sure to provide us with the information requested in the template as well as any additional appropriate information so we can best determine how to assist with the given issue.

[bflad]

@tracypholmes this seems like something we should at least call out in our documentation to potentially help other practitioners. 👍 e.g. in website/docs/r/s3_bucket.html.markdown

Replacing:

* `expired_object_delete_marker` (Optional) On a versioned bucket (versioning-enabled or versioning-suspended bucket), you can add this element in the lifecycle configuration to direct Amazon S3 to delete expired object delete markers.

With something like the below to note the restriction:

* `expired_object_delete_marker` (Optional) On a versioned bucket (versioning-enabled or versioning-suspended bucket), you can add this element in the lifecycle configuration to direct Amazon S3 to delete expired object delete markers. This cannot be specified with `days` or `date`.

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.

If you feel this issue should be reopened, we encourage creating a new issue linking back to this one for added context. Thanks!