elasticsearch_xpack_index_lifecycle_policy always triggers update

[glisignoli]

elasticsearch_xpack_index_lifecycle_policy always triggers an update, as the returned GET json contains extra element (in_use_by) not required in the POST json:

# elasticsearch_xpack_index_lifecycle_policy.metricbeat will be updated in-place
resource "elasticsearch_xpack_index_lifecycle_policy" "metricbeat" {
      ~ body = jsonencode(
          ~ {
              - in_use_by     = {
                  - composable_templates = [
                      - "metricbeat-7.17.3",
                      - "metricbeat-8.1.0",
                    ]
                  - data_streams         = [
                      - "metricbeat-8.1.0",
                    ]
                  - indices              = [
                      - "metricbeat-7.13.0-2022.04.03-000048",
                      - "metricbeat-7.13.3-2022.04.13-000049",
                      ....
                      ....
                      ....
                      ....
                      - "metricbeat-7.12.1-2022.05.08-000064",
                    ]
                } -> null
              - modified_date = "2022-05-18T03:30:58.272Z" -> null
              - version       = 14 -> null
                # (1 unchanged element hidden)
            }
        )
        id   = "metricbeat"
        name = "metricbeat"
    }

[phillbaker]

Hello, can you please include the following:

• terraform version
• provider version
• elasticsearch version (and opendistro version if relevant)
• redacted version of the terraform resource configuration

[glisignoli]

Terraform: v1.0.7
Provider Version: provider registry.terraform.io/phillbaker/elasticsearch v2.0.1
Elasticsearch Version: 7.15.1

resource "elasticsearch_xpack_index_lifecycle_policy" "metricbeat" {
  provider = elasticsearch.elasticsearch_prod
  name     = "metricbeat"
  body     = file("files/beats/metricbeat_prod.ilm.json")
}

(File content):

{
    "policy": {
      "phases": {
        "hot": {
          "actions": {
            "rollover": {
              "max_age": "5d",
              "max_size": "50gb"
            },
            "set_priority": {
              "priority": 100
            }
          },
          "min_age": "0ms"
        },
        "warm": {
          "min_age": "15d",
          "actions": {
            "set_priority": {
              "priority": 50
            }
          }
        },
        "cold": {
          "min_age": "40d",
          "actions": {
            "set_priority": {
              "priority": 0
            }
          }
        },
        "delete": {
          "min_age": "60d",
          "actions": {
            "delete": {
              "delete_searchable_snapshot": true
            }
          }
        }
      }
    }
  }

[christophemorio]

I get the same issue with this provider v2.0.1, and ES cluster 8.2.1

---

Found a workaround (requires terraform >= 1.2.0)

It set ignore_changes on body,
and consider input value to trigger a replacement.

Meanwhile, this does not reconcile if value have changed on Elasticsearch without modification on code side.

locals {
  indices = {
    "foo" = {"policy":{"phases":{"hot":{"actions":{"rollover":{"max_age":"10d"}}}}}}
    "bar" = {"policy":{"phases":{"hot":{"actions":{"rollover":{"max_age":"20d"}}}}}}
  }
}

resource "elasticsearch_xpack_index_lifecycle_policy" "main" {
  for_each = local.indices
  name     = each.key
  body     = jsonencode(each.value)

  lifecycle {
    ignore_changes       = [body]
    replace_triggered_by = [null_resource.index_lifecycle_policy_trigger[each.key]]
  }
}

resource "null_resource" "index_lifecycle_policy_trigger" {
  for_each = local.indices
  triggers = {
    body = jsonencode(each.value)
  }
}

EDIT: it's not working once policy is attached to an index

elasticsearch_xpack_index_lifecycle_policy.main["foo"]: Destroying... [id=foo]
╷
│ Error: elastic: Error 400 (Bad Request): Cannot delete policy [foo]. It is in use by one or more indices: [test-1] [type=illegal_argument_exception]

[phillbaker]

Hi all - a fix for this is in 44db200, trying to get the tests up to date to reflect this.

[phillbaker]

Fixed in b1d58ff

[sparrowt]

I'm using v2.0.4 of this provider, and although these non-static fields are not triggering "will be updated" they are still being flagged as "has been changed" (see example below) - is there a way for the same DiffSuppressFunc code to be applied during the calculation of the "Objects have changed outside of Terraform" section too, to ignore spurious changes to e.g. in_use_by?

Plan output:

Click to expand

Note: Objects have changed outside of Terraform

Terraform detected the following changes made outside of Terraform since the
last "terraform apply":

  # elasticsearch_xpack_index_lifecycle_policy.my-ilm-name has been changed
  ~ resource "elasticsearch_xpack_index_lifecycle_policy" "my-ilm" {
      ~ body = jsonencode(
          ~ {
              + in_use_by     = {
                  + indices = [
                      + "myindex-6",
                      + "myindex-7",
                      + "myindex-8",
                      + "myindex-9",
                    ]
                }
              + modified_date = "2022-10-05T12:25:44.689Z"
              ~ policy        = {
                  ~ phases = {
                      ~ cold   = {
                          ~ actions = {
                              ~ allocate = {
                                  + exclude            = {}
                                  + include            = {}
                                  + require            = {}
                                    # (1 unchanged element hidden)
                                }
                            }
                            # (1 unchanged element hidden)
                        }
                        # (3 unchanged elements hidden)
                    }
                }
              + version       = 6
            }
        )
        id   = "my-ilm-name"
        name = "my-ilm-name"
    }

Unless you have made equivalent changes to your configuration, or ignored the
relevant attributes using ignore_changes, the following plan may include
actions to undo or respond to these changes.

─────────────────────────────────────────────────────────────────────────────

No changes. Your infrastructure matches the configuration.

Your configuration already matches the changes detected above. If you'd like
to update the Terraform state to match, create and apply a refresh-only plan:
  terraform apply -refresh-only
Releasing state lock. This may take a few moments...

Terraform:

Click to expand

resource "elasticsearch_xpack_index_lifecycle_policy" "my-ilm" {
  name = "my-ilm-name"
  body = <<EOF
{
  "policy" : {
    "phases" : {
      "hot" : {
        "min_age" : "0ms",
        "actions" : {}
      },
      "warm": {
        "min_age": "31d",
        "actions": {}
      },
      "cold": {
        "min_age": "45d",
        "actions": {
          "allocate": { "number_of_replicas": 0 }
        }
      },
      "delete" : {
        "min_age" : "62d",
        "actions" : {
          "delete" : { "delete_searchable_snapshot": true }
        }
      }
    }
  }
}
EOF
}

I realise that terraform apply -refresh-only will get rid of these particular changes, but given in_use_by.indices is constantly changing due to rollover & ILM deletion, this is only a temporary solution.