[Bug] Cannot add/update Metric filter in elasticstack_kibana_slo resource #1465
Description
Describe the bug
Metric filter is not added or updated when applying the change to elasticstack_kibana_slo resource.
To Reproduce
Steps to reproduce the behavior:
- TF configuration used
resource "elasticstack_kibana_slo" "custom_kql_slo_elasticsearch_cluster_health" {
count = var.deploy_slo_elasticsearch_cluster_health ? 1 : 0
name = "Elasticsearch Cluster Health"
description = "Elasticsearch data availability based on cluster health status"
kql_custom_indicator {
index = ".monitoring-*"
good = "cluster_state.status: \"green\""
total = "cluster_state.status: *"
filter = ""
timestamp_field = "@timestamp"
}
time_window {
duration = "30d"
type = "rolling"
}
budgeting_method = "occurrences"
objective {
target = 0.99
}
group_by = [ "elasticsearch.cluster.name" ]
settings {
sync_delay = "5m"
frequency = "5m"
}
}
resource "elasticstack_kibana_slo" "timeslice_slo_search_latency_on_boomi_index" {
count = var.deploy_slo_boomi_timeslice_search_latency ? 1 : 0
name = "Search Latency on Boomi Index"
description = "This SLO tracks search latency on a given index pattern below a certain threshold. Please adjust the index pattern (boomi*) and search threshold (2 seconds) as required."
timeslice_metric_indicator {
index = ".monitoring-*"
timestamp_field = "@timestamp"
filter = "elasticsearch.index.name: boomi_*"
metric {
metrics {
name = "A"
aggregation = "max"
field = "index_stats.total.search.query_time_in_millis"
filter = ""
}
metrics {
name = "B"
aggregation = "min"
field = "index_stats.total.search.query_time_in_millis"
filter = "index_stats.total.search.query_total > 100"
}
metrics {
name = "C"
aggregation = "max"
field = "index_stats.total.search.query_total"
# filter = ""
}
metrics {
name = "D"
aggregation = "min"
field = "index_stats.total.search.query_total"
# filter = ""
}
equation = "((A - B)/(C - D))/1000"
comparator = "LTE"
threshold = 2
}
}
group_by = [ "elasticsearch.cluster.name" ]
time_window {
duration = "30d"
type = "rolling"
}
budgeting_method = "timeslices"
objective {
target = 0.99
timeslice_target = 0.95
timeslice_window = "5m"
}
settings {
sync_delay = "5m"
frequency = "5m"
}
}
- Run
terraform applyto create the SLOs. - Run
terraform planto confirm the required change.
~ metrics {
+ filter = "index_stats.total.search.query_total > 100"
name = "B"
# (3 unchanged attributes hidden)
}
- Run
terraform applyto confirm the successful update.
Do you want to perform these actions?
Terraform will perform the actions described above.
Only 'yes' will be accepted to approve.
Enter a value: yes
...
Apply complete! Resources: 0 added, 1 changed, 0 destroyed.
Note: TF_LOG=debug terraform apply shows the filter is not included in the REQUEST.
5. Check the SLOs in Kibana to confirm the filter is not added.
6. Run terraform plan again to find the same output as #3 above.
Expected behavior
terraform apply should add/update the filter to the SLO.
Debug output
Trace output of `TF_LOG=trace terraform apply` showing a successful update without actually updating the filter.
---[ REQUEST ]---------------------------------------
GET /s/default/api/observability/slos/cf1b34ff-ec69-4eef-bfe8-ac95caebafe7 HTTP/1.1
Host: xxx.us-east-1.aws.found.io
User-Agent: elasticstack-terraform-provider/0.12.1
Accept: application/json
Authorization: ***************************************************
kbn-xsrf: true
Accept-Encoding: gzip
-----------------------------------------------------: @module=elasticstack tf_mux_provider=tf5to6server.v5tov6Server tf_req_id=4b9430c3-0d53-62d9-876d-c1a4dcb22c9d tf_resource_type=elasticstack_kibana_slo @caller=github.com/elastic/terraform-provider-elasticstack/internal/utils/http_log.go:40 tf_provider_addr=registry.terraform.io/elastic/elasticstack tf_rpc=ApplyResourceChange timestamp="2025-11-20T15:36:55.056+1100"
2025-11-20T15:36:55.336+1100 [DEBUG] provider.terraform-provider-elasticstack_v0.12.1: Kibana API Response Details:
---[ RESPONSE ]--------------------------------------
HTTP/2.0 200 OK
Cache-Control: private, no-cache, no-store, must-revalidate
Content-Security-Policy: script-src 'report-sample' 'self'; worker-src 'report-sample' 'self' blob:; style-src 'report-sample' 'self' 'unsafe-inline'; object-src 'report-sample' 'none'; report-to violations-endpoint
Content-Security-Policy-Report-Only: form-action 'report-sample' 'self'; report-to violations-endpoint
Content-Type: application/json; charset=utf-8
Cross-Origin-Opener-Policy: same-origin
Date: Thu, 20 Nov 2025 04:36:55 GMT
Elastic-Api-Version: 2023-10-31
Kbn-License-Sig: bc9f54x
Kbn-Name: instance-0000000001
Permissions-Policy: camera=(), display-capture=(), fullscreen=(self), geolocation=(), microphone=(), web-share=();report-to=violations-endpoint
Referrer-Policy: strict-origin-when-cross-origin
Reporting-Endpoints: violations-endpoint="https://my-example-deployment-69447e.kb.us-east-1.aws.found.io/internal/security/analytics/_record_violations"
Vary: accept-encoding
X-Cloud-Request-Id: NuLCKFkcQsmqRMkeWzcS3Q
X-Content-Type-Options: nosniff
X-Found-Handling-Cluster: xxx
X-Found-Handling-Instance: instance-0000000001
{
"budgetingMethod": "timeslices",
"description": "This SLO tracks search latency on a given index pattern below a certain threshold. Please adjust the index pattern (boomi*) and search threshold (2 seconds) as required.",
"groupBy": [
"elasticsearch.cluster.name"
],
"indicator": {
"params": {
"filter": "elasticsearch.index.name: boomi_*",
"index": ".monitoring-*",
"metric": {
"comparator": "LTE",
"equation": "((A - B)/(C - D))/1000",
"metrics": [
{
"aggregation": "max",
"field": "index_stats.total.search.query_time_in_millis",
"name": "A"
},
{
"aggregation": "min",
"field": "index_stats.total.search.query_time_in_millis",
"name": "B"
},
{
"aggregation": "max",
"field": "index_stats.total.search.query_total",
"name": "C"
},
{
"aggregation": "min",
"field": "index_stats.total.search.query_total",
"name": "D"
}
],
"threshold": 2
},
"timestampField": "@timestamp"
},
"type": "sli.metric.timeslice"
},
"name": "Search Latency on Boomi Index",
"objective": {
"target": 0.99,
"timesliceTarget": 0.95,
"timesliceWindow": "5m"
},
"settings": {
"preventInitialBackfill": false,
"syncDelay": "5m",
"frequency": "5m"
},
"timeWindow": {
"duration": "30d",
"type": "rolling"
},
"id": "cf1b34ff-ec69-4eef-bfe8-ac95caebafe7",
"revision": 1,
"enabled": true,
"tags": [],
"createdAt": "2025-11-17T23:20:01.566Z",
"updatedAt": "2025-11-17T23:20:01.566Z",
"createdBy": "elastic",
"updatedBy": "elastic",
"version": 2,
"artifacts": {
"dashboards": []
},
"instanceId": "*",
"summary": {
"sliValue": 1,
"errorBudget": {
"initial": 0.01,
"consumed": 0,
"remaining": 1,
"isEstimated": false
},
"status": "HEALTHY",
"fiveMinuteBurnRate": 0,
"oneHourBurnRate": 0,
"oneDayBurnRate": 0
},
"groupings": {},
"meta": {}
}
-----------------------------------------------------: tf_provider_addr=registry.terraform.io/elastic/elasticstack tf_req_id=4b9430c3-0d53-62d9-876d-c1a4dcb22c9d tf_resource_type=elasticstack_kibana_slo tf_rpc=ApplyResourceChange @caller=github.com/elastic/terraform-provider-elasticstack/internal/utils/http_log.go:52 tf_mux_provider=tf5to6server.v5tov6Server @module=elasticstack timestamp="2025-11-20T15:36:55.335+1100"
2025-11-20T15:36:55.337+1100 [DEBUG] provider.terraform-provider-elasticstack_v0.12.1: 2025/11/20 15:36:55
2025-11-20T15:36:55.337+1100 [DEBUG] provider.terraform-provider-elasticstack_v0.12.1: HTTP/2.0 200 OK
2025-11-20T15:36:55.337+1100 [DEBUG] provider.terraform-provider-elasticstack_v0.12.1: Cache-Control: private, no-cache, no-store, must-revalidate
2025-11-20T15:36:55.337+1100 [DEBUG] provider.terraform-provider-elasticstack_v0.12.1: Content-Security-Policy: script-src 'report-sample' 'self'; worker-src 'report-sample' 'self' blob:; style-src 'report-sample' 'self' 'unsafe-inline'; object-src 'report-sample' 'none'; report-to violations-endpoint
2025-11-20T15:36:55.337+1100 [DEBUG] provider.terraform-provider-elasticstack_v0.12.1: Content-Security-Policy-Report-Only: form-action 'report-sample' 'self'; report-to violations-endpoint
2025-11-20T15:36:55.337+1100 [DEBUG] provider.terraform-provider-elasticstack_v0.12.1: Content-Type: application/json; charset=utf-8
2025-11-20T15:36:55.337+1100 [DEBUG] provider.terraform-provider-elasticstack_v0.12.1: Cross-Origin-Opener-Policy: same-origin
2025-11-20T15:36:55.337+1100 [DEBUG] provider.terraform-provider-elasticstack_v0.12.1: Date: Thu, 20 Nov 2025 04:36:55 GMT
2025-11-20T15:36:55.337+1100 [DEBUG] provider.terraform-provider-elasticstack_v0.12.1: Elastic-Api-Version: 2023-10-31
2025-11-20T15:36:55.337+1100 [DEBUG] provider.terraform-provider-elasticstack_v0.12.1: Kbn-License-Sig: bc9f54378e40a6165cbd6f90fb98693a05936f1458e19ef8c191ae1b466cec31
2025-11-20T15:36:55.337+1100 [DEBUG] provider.terraform-provider-elasticstack_v0.12.1: Kbn-Name: instance-0000000001
2025-11-20T15:36:55.337+1100 [DEBUG] provider.terraform-provider-elasticstack_v0.12.1: Permissions-Policy: camera=(), display-capture=(), fullscreen=(self), geolocation=(), microphone=(), web-share=();report-to=violations-endpoint
2025-11-20T15:36:55.337+1100 [DEBUG] provider.terraform-provider-elasticstack_v0.12.1: Referrer-Policy: strict-origin-when-cross-origin
2025-11-20T15:36:55.337+1100 [DEBUG] provider.terraform-provider-elasticstack_v0.12.1: Reporting-Endpoints: violations-endpoint="https://my-example-deployment-69447e.kb.us-east-1.aws.found.io/internal/security/analytics/_record_violations"
2025-11-20T15:36:55.337+1100 [DEBUG] provider.terraform-provider-elasticstack_v0.12.1: Vary: accept-encoding
2025-11-20T15:36:55.337+1100 [DEBUG] provider.terraform-provider-elasticstack_v0.12.1: X-Cloud-Request-Id: NuLCKFkcQsmqRMkeWzcS3Q
2025-11-20T15:36:55.337+1100 [DEBUG] provider.terraform-provider-elasticstack_v0.12.1: X-Content-Type-Options: nosniff
2025-11-20T15:36:55.337+1100 [DEBUG] provider.terraform-provider-elasticstack_v0.12.1: X-Found-Handling-Cluster: 6277c336a8f945ff800a4d1abd81dd54
2025-11-20T15:36:55.337+1100 [DEBUG] provider.terraform-provider-elasticstack_v0.12.1: X-Found-Handling-Instance: instance-0000000001
2025-11-20T15:36:55.337+1100 [DEBUG] provider.terraform-provider-elasticstack_v0.12.1
2025-11-20T15:36:55.338+1100 [DEBUG] provider.terraform-provider-elasticstack_v0.12.1: {"budgetingMethod":"timeslices","description":"This SLO tracks search latency on a given index pattern below a certain threshold. Please adjust the index pattern (boomi*) and search threshold (2 seconds) as required.","groupBy":["elasticsearch.cluster.name"],"indicator":{"params":{"filter":"elasticsearch.index.name: boomi_*","index":".monitoring-*","metric":{"comparator":"LTE","equation":"((A - B)/(C - D))/1000","metrics":[{"aggregation":"max","field":"index_stats.total.search.query_time_in_millis","name":"A"},{"aggregation":"min","field":"index_stats.total.search.query_time_in_millis","name":"B"},{"aggregation":"max","field":"index_stats.total.search.query_total","name":"C"},{"aggregation":"min","field":"index_stats.total.search.query_total","name":"D"}],"threshold":2},"timestampField":"@timestamp"},"type":"sli.metric.timeslice"},"name":"Search Latency on Boomi Index","objective":{"target":0.99,"timesliceTarget":0.95,"timesliceWindow":"5m"},"settings":{"preventInitialBackfill":false,"syncDelay":"5m","frequency":"5m"},"timeWindow":{"duration":"30d","type":"rolling"},"id":"cf1b34ff-ec69-4eef-bfe8-ac95caebafe7","revision":1,"enabled":true,"tags":[],"createdAt":"2025-11-17T23:20:01.566Z","updatedAt":"2025-11-17T23:20:01.566Z","createdBy":"elastic","updatedBy":"elastic","version":2,"artifacts":{"dashboards":[]},"instanceId":"*","summary":{"sliValue":1,"errorBudget":{"initial":0.01,"consumed":0,"remaining":1,"isEstimated":false},"status":"HEALTHY","fiveMinuteBurnRate":0,"oneHourBurnRate":0,"oneDayBurnRate":0},"groupings":{},"meta":{}}
2025-11-20T15:36:55.338+1100 [TRACE] provider.terraform-provider-elasticstack_v0.12.1: Called downstream: tf_rpc=ApplyResourceChange @caller=github.com/hashicorp/terraform-plugin-sdk/v2@v2.38.1/helper/schema/resource.go:991 @module=sdk.helper_schema tf_mux_provider=tf5to6server.v5tov6Server tf_provider_addr=registry.terraform.io/elastic/elasticstack tf_req_id=4b9430c3-0d53-62d9-876d-c1a4dcb22c9d tf_resource_type=elasticstack_kibana_slo timestamp="2025-11-20T15:36:55.336+1100"
2025-11-20T15:36:55.338+1100 [TRACE] provider.terraform-provider-elasticstack_v0.12.1: Received downstream response: diagnostic_error_count=0 tf_proto_version=6.10 tf_req_duration_ms=1952 tf_req_id=4b9430c3-0d53-62d9-876d-c1a4dcb22c9d tf_resource_type=elasticstack_kibana_slo @caller=github.com/hashicorp/terraform-plugin-go@v0.29.0/tfprotov6/internal/tf6serverlogging/downstream_request.go:61 diagnostic_warning_count=0 tf_provider_addr=registry.terraform.io/elastic/elasticstack tf_rpc=ApplyResourceChange @module=sdk.proto timestamp="2025-11-20T15:36:55.338+1100"
2025-11-20T15:36:55.338+1100 [TRACE] provider.terraform-provider-elasticstack_v0.12.1: Served request: tf_provider_addr=registry.terraform.io/elastic/elasticstack tf_resource_type=elasticstack_kibana_slo tf_proto_version=6.10 tf_req_id=4b9430c3-0d53-62d9-876d-c1a4dcb22c9d tf_rpc=ApplyResourceChange @caller=github.com/hashicorp/terraform-plugin-go@v0.29.0/tfprotov6/tf6server/server.go:957 @module=sdk.proto timestamp="2025-11-20T15:36:55.338+1100"
2025-11-20T15:36:55.340+1100 [WARN] Provider "provider[\"registry.terraform.io/elastic/elasticstack\"]" produced an unexpected new value for elasticstack_kibana_slo.timeslice_slo_search_latency_on_boomi_index[0], but we are tolerating it because it is using the legacy plugin SDK.
The following problems may be the cause of any confusing errors from downstream operations:
- .timeslice_metric_indicator[0].metric[0].metrics[1].filter: was cty.StringVal("index_stats.total.search.query_total > 100"), but now cty.StringVal("")
2025-11-20T15:36:55.340+1100 [TRACE] terraform.contextPlugins: Schema for provider "registry.terraform.io/elastic/elasticstack" is in the global cache
2025-11-20T15:36:55.340+1100 [TRACE] NodeAbstractResouceInstance.writeResourceInstanceState to workingState for elasticstack_kibana_slo.timeslice_slo_search_latency_on_boomi_index[0]
2025-11-20T15:36:55.340+1100 [TRACE] NodeAbstractResouceInstance.writeResourceInstanceState: writing state object for elasticstack_kibana_slo.timeslice_slo_search_latency_on_boomi_index[0]
2025-11-20T15:36:55.340+1100 [TRACE] evalApplyProvisioners: elasticstack_kibana_slo.timeslice_slo_search_latency_on_boomi_index[0] is not freshly-created, so no provisioning is required
2025-11-20T15:36:55.340+1100 [TRACE] terraform.contextPlugins: Schema for provider "registry.terraform.io/elastic/elasticstack" is in the global cache
2025-11-20T15:36:55.340+1100 [TRACE] NodeAbstractResouceInstance.writeResourceInstanceState to workingState for elasticstack_kibana_slo.timeslice_slo_search_latency_on_boomi_index[0]
2025-11-20T15:36:55.340+1100 [TRACE] NodeAbstractResouceInstance.writeResourceInstanceState: writing state object for elasticstack_kibana_slo.timeslice_slo_search_latency_on_boomi_index[0]
elasticstack_kibana_slo.timeslice_slo_search_latency_on_boomi_index[0]: Modifications complete after 2s [id=default/cf1b34ff-ec69-4eef-bfe8-ac95caebafe7]
2025-11-20T15:36:55.342+1100 [TRACE] statemgr.Filesystem: creating backup snapshot at terraform.tfstate.backup
2025-11-20T15:36:55.346+1100 [TRACE] statemgr.Filesystem: state has changed since last snapshot, so incrementing serial to 10
2025-11-20T15:36:55.346+1100 [TRACE] statemgr.Filesystem: writing snapshot at terraform.tfstate
2025-11-20T15:36:55.354+1100 [DEBUG] State storage *statemgr.Filesystem declined to persist a state snapshot
2025-11-20T15:36:55.354+1100 [TRACE] vertex "elasticstack_kibana_slo.timeslice_slo_search_latency_on_boomi_index[0]": visit complete
2025-11-20T15:36:55.354+1100 [TRACE] vertex "provider[\"registry.terraform.io/elastic/elasticstack\"] (close)": starting visit (*terraform.graphNodeCloseProvider)
2025-11-20T15:36:55.354+1100 [TRACE] vertex "provider[\"registry.terraform.io/elastic/elasticstack\"] (close)": does not belong to any module instance
2025-11-20T15:36:55.354+1100 [TRACE] GRPCProvider.v6: Close
2025-11-20T15:36:55.356+1100 [DEBUG] provider.stdio: received EOF, stopping recv loop: err="rpc error: code = Unavailable desc = error reading from server: EOF"
2025-11-20T15:36:55.360+1100 [INFO] provider: plugin process exited: plugin=.terraform/providers/registry.terraform.io/elastic/elasticstack/0.12.1/darwin_arm64/terraform-provider-elasticstack_v0.12.1 id=33837
2025-11-20T15:36:55.360+1100 [DEBUG] provider: plugin exited
2025-11-20T15:36:55.360+1100 [TRACE] vertex "provider[\"registry.terraform.io/elastic/elasticstack\"] (close)": visit complete
2025-11-20T15:36:55.360+1100 [TRACE] vertex "root": starting visit (*terraform.nodeCloseModule)
2025-11-20T15:36:55.360+1100 [TRACE] vertex "root": does not belong to any module instance
2025-11-20T15:36:55.360+1100 [TRACE] vertex "root": visit complete
2025-11-20T15:36:55.360+1100 [TRACE] terraform.contextPlugins: Schema for provider "registry.terraform.io/elastic/ec" is in the global cache
2025-11-20T15:36:55.360+1100 [TRACE] terraform.contextPlugins: Schema for provider "registry.terraform.io/elastic/elasticstack" is in the global cache
2025-11-20T15:36:55.360+1100 [TRACE] terraform.contextPlugins: Schema for provider "registry.terraform.io/hashicorp/aws" is in the global cache
2025-11-20T15:36:55.360+1100 [TRACE] statemgr.Filesystem: have already backed up original terraform.tfstate to terraform.tfstate.backup on a previous write
2025-11-20T15:36:55.361+1100 [TRACE] statemgr.Filesystem: state has changed since last snapshot, so incrementing serial to 11
2025-11-20T15:36:55.361+1100 [TRACE] statemgr.Filesystem: writing snapshot at terraform.tfstate
2025-11-20T15:36:55.367+1100 [TRACE] statemgr.Filesystem: removing lock metadata file .terraform.tfstate.lock.info
2025-11-20T15:36:55.369+1100 [TRACE] statemgr.Filesystem: unlocking terraform.tfstate using fcntl flock
Apply complete! Resources: 0 added, 1 changed, 0 destroyed.
Versions (please complete the following information):
- OS: MacOS
- Terraform Version 1.13.5
- Provider version 0.11.17 and 0.12.1
- Elasticsearch Version 9.x.
Additional context
Similar observation is also made when the filter is initially added to the SLO but the following terraform plan and terraform apply continue to find the same change in filter in each execution without changing any configuration.