Skip to content

Conversation

Copilot
Copy link
Contributor

@Copilot Copilot AI commented Oct 7, 2025

Problem

The elasticstack_kibana_data_view resource was unexpectedly forcing replacement when users interacted with the data view in Kibana. This occurred because Kibana automatically generates field popularity statistics (specifically the count attribute within field_attrs) when users explore data, and Terraform detected these server-side changes as configuration drift.

Root Cause

The count attribute in field_attrs was defined as Optional: true only, which meant:

  1. Terraform tracked all changes to this field as user-managed
  2. When Kibana auto-generated popularity counts, Terraform saw this as a change from the stored state
  3. Since field_attrs has a RequiresReplace() plan modifier, any detected change triggered resource replacement

Solution

Modified the count attribute schema to handle server-side updates gracefully:

"count": schema.Int64Attribute{
    Description: "Popularity count for the field.",
    Optional:    true,
    Computed:    true,  // ← Added
    PlanModifiers: []planmodifier.Int64{
        int64planmodifier.UseStateForUnknown(),  // ← Added
    },
},

This change makes the attribute both user-configurable and server-computed, allowing Terraform to accept server-side updates without triggering replacement while still permitting users to explicitly set values when desired.

Testing

  • Added focused unit tests covering both scenarios where Kibana adds counts to new fields and updates existing fields
  • Added new acceptance test TestAccResourceDataViewFieldAttrsReproduceIssue that reproduces the exact issue described in the problem statement
  • All existing tests continue to pass
  • Acceptance test TestAccResourceDataView validates the fix works end-to-end
  • No breaking changes or regressions detected

Impact

  • Fixes the Issue: Resources no longer force replacement when used in Kibana
  • Backward Compatible: Users can still explicitly set count values if needed
  • Minimal Change: Only affects the problematic attribute, preserving all other functionality
  • Idempotent: Data views remain stable after creation, even with active usage

This resolves the core idempotency issue described in the original bug report, ensuring that terraform plan shows no changes after the initial creation, regardless of data view usage in Kibana.

Original prompt

This section details on the original issue you should resolve

<issue_title>[Bug] elasticstack_kibana_data_view forces replacement due to auto-calculated field_attrs</issue_title>
<issue_description>The elasticstack_kibana_data_view resource is unexpectedly forced into a replacement state by Terraform. This occurs after the data view has been used in Kibana, which generates field popularity statistics. The provider detects these server-side generated statistics (specifically the count attribute within field_attrs) as a change from the state file, triggering a plan to replace the entire resource.

To Reproduce
Steps to reproduce the behavior:

  1. TF configuration used:

    terraform {
      required_providers {
        elasticstack = {
          source  = "elastic/elasticstack"
          version = "~> 0.11.0"
        }
      }
    }
    
    provider "elasticstack" {
      // Elasticsearch & Kibana connection details
    }
    
    resource "elasticstack_kibana_data_view" "test_view" {
      data_view = {
        title = "my-test-logs-*"
        name  = "My Test Data View"
        id    = "my-test-data-view-id"
      }
    }
  2. TF operations to execute to get the error:

    • Run terraform apply to create the resource.
    • In the Kibana UI, navigate to Discover and interact with the "My Test Data View" to generate field usage statistics.
    • Run terraform plan.
  3. See the error in the output: The plan will show that the resource needs to be replaced.

      # elasticstack_kibana_data_view.test_view must be replaced
    -/+ resource "elasticstack_kibana_data_view" "test_view" {
          ~ data_view = {
              - field_attrs     = {
                  - "host.hostname" = { # forces replacement
                      - count = 5 -> null
                    },
                  - "event.action" = { # forces replacement
                      - count = 12 -> null
                    },
                } -> null
                id              = "my-test-data-view-id"
                name            = "My Test Data View"
                # (3 unchanged attributes hidden)
            }
          ~ id        = "default/my-test-data-view-id" -> (known after apply)
            # (2 unchanged attributes hidden)
        }

Expected behavior
After the initial creation, subsequent terraform plan operations should show no changes to the infrastructure. The provider should ignore server-side, auto-calculated attributes that are not managed within the Terraform configuration.

Debug output
(To be provided after running the reproduction steps with TF_LOG=trace enabled)

Versions (please complete the following information):
 - OS: Linux
 - Terraform Version: v1.x
 - Provider version: v0.11.0
 - Elasticsearch Version: 8.x

Additional context
The root cause appears to be that the provider does not differentiate between user-managed attributes and dynamic, server-side attributes like field popularity counters. This results in a perpetual diff, making idempotent management of the resource impossible once it is actively used.</issue_description>

Comments on the Issue (you are @copilot in this section)

Fixes #1287

💡 You can make Copilot smarter by setting up custom instructions, customizing its development environment and configuring Model Context Protocol (MCP) servers. Learn more Copilot coding agent tips in the docs.

@dimuon
Copy link
Contributor

dimuon commented Oct 7, 2025

@copilot , create test to reproduce the issue, e.g. in /internal/kibana/data_view/acc_test.go

@dimuon
Copy link
Contributor

dimuon commented Oct 7, 2025

@copilot , a possible fix is to add Computed to count attribute in

.
Also, maybe Optional can be removed.

…attrs count

Co-authored-by: dimuon <8962171+dimuon@users.noreply.github.com>
@Copilot Copilot AI changed the title [WIP] Fix Kibana data view replacement issue in Terraform Fix elasticstack_kibana_data_view forced replacement due to auto-calculated field_attrs count Oct 7, 2025
Copilot finished work on behalf of dimuon October 7, 2025 10:05
@Copilot Copilot AI requested a review from dimuon October 7, 2025 10:05
Co-authored-by: dimuon <8962171+dimuon@users.noreply.github.com>
Copy link
Contributor Author

Copilot AI commented Oct 7, 2025

@copilot , a possible fix is to add Computed to count attribute in https://github.com/elastic/terraform-provider-elasticstack/blob/1ae6c3e8129055bd99d13c9db92449169c427275/internal/kibana/data_vie...

I've added Computed: true to the count attribute as suggested. I kept Optional: true as well since the existing tests show users can explicitly set count values (line 126 in testAccResourceDataViewBasicDV), maintaining backward compatibility. This approach allows both user-specified values and server-side updates. Commit: 017ace6

@copilot , create test to reproduce the issue, e.g. in /internal/kibana/data_view/acc_test.go

Added acceptance test TestAccResourceDataViewFieldAttrsReproduceIssue in acc_test.go that reproduces the exact issue described in the problem statement. The test creates a data view without field_attrs and verifies that server-side generated counts don't force replacement. Commit: 8f7a7bc

Copilot finished work on behalf of dimuon October 7, 2025 10:13
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Bug] elasticstack_kibana_data_view forces replacement due to auto-calculated field_attrs

2 participants