[Bug] Terraform apply fails when creating a Kibana space

[stefanfreitag]

The creation of a Kibana space works partially. During the terraform apply phase

• The space itself is created
• the error message "Error: cannot retrieve informations from Elasticsearch" is displayed

Steps to reproduce the behavior:

1. Create a terraform file with the resource definition as below

resource "elasticstack_kibana_space" "example" {
  space_id    = "test_space"
  name        = "Test Space"
  description = "A fresh space for testing visualisations"
  initials    = "ts"
}

2. Create a terraform file containing details for provider

terraform {
  required_version = "~>1.4"
  required_providers {
    elasticstack = {
      source = "elastic/elasticstack"
      version ="0.6.1"
    }
  }
}

provider "elasticstack" {
  elasticsearch {
    password  = "XXX"
    username  = "XXX"
    endpoints = ["https://XXX:9243"]
  }
  kibana {
    password  = "XXX"
    username  = "XXX"
    endpoints = ["https://XXX:9243"]
  }
}

4. Execute a terraform apply

Expected behavior

A space can be created without errors.

Debug output

elasticstack_kibana_space.example: Creating...
2023-05-30T21:15:23.372+0200 [INFO]  Starting apply for elasticstack_kibana_space.example
2023-05-30T21:15:23.372+0200 [DEBUG] elasticstack_kibana_space.example: applying the planned Create change
2023-05-30T21:15:23.372+0200 [TRACE] GRPCProvider: ApplyResourceChange
2023-05-30T21:15:23.372+0200 [TRACE] provider.terraform-provider-elasticstack_v0.6.1: Received request: @caller=github.com/hashicorp/terraform-plugin-go@v0.15.0/tfprotov5/tf5server/server.go:805 tf_proto_version=5.3 tf_provider_addr=registry.terraform.io/elastic/elasticstack tf_req_id=df9cb2ad-c653-677d-ad5c-cb31156b89a2 tf_resource_type=elasticstack_kibana_space @module=sdk.proto tf_rpc=ApplyResourceChange timestamp=2023-05-30T21:15:23.372+0200
2023-05-30T21:15:23.372+0200 [TRACE] provider.terraform-provider-elasticstack_v0.6.1: Sending request downstream: @module=sdk.proto tf_proto_version=5.3 tf_provider_addr=registry.terraform.io/elastic/elasticstack tf_req_id=df9cb2ad-c653-677d-ad5c-cb31156b89a2 tf_resource_type=elasticstack_kibana_space tf_rpc=ApplyResourceChange @caller=github.com/hashicorp/terraform-plugin-go@v0.15.0/tfprotov5/internal/tf5serverlogging/downstream_request.go:17 timestamp=2023-05-30T21:15:23.372+0200
2023-05-30T21:15:23.372+0200 [TRACE] provider.terraform-provider-elasticstack_v0.6.1: calling downstream server: @module=sdk.mux tf_mux_provider=*schema.GRPCProviderServer tf_rpc=ApplyResourceChange @caller=github.com/hashicorp/terraform-plugin-mux@v0.9.0/internal/logging/mux.go:16 timestamp=2023-05-30T21:15:23.372+0200
2023-05-30T21:15:23.372+0200 [TRACE] provider.terraform-provider-elasticstack_v0.6.1: Calling downstream: @module=sdk.helper_schema tf_mux_provider=*schema.GRPCProviderServer tf_resource_type=elasticstack_kibana_space tf_rpc=ApplyResourceChange @caller=github.com/hashicorp/terraform-plugin-sdk/v2@v2.26.1/helper/schema/resource.go:836 tf_provider_addr=registry.terraform.io/elastic/elasticstack tf_req_id=df9cb2ad-c653-677d-ad5c-cb31156b89a2 timestamp=2023-05-30T21:15:23.372+0200
2023-05-30T21:15:23.832+0200 [TRACE] provider.stdio: received data: channel=STDERR len=1024
2023-05-30T21:15:23.832+0200 [WARN]  unexpected data: registry.terraform.io/elastic/elasticstack:stderr="2023/05/30 21:15:23.832011 DEBUG RESTY 
==============================================================================
~~~ REQUEST ~~~
POST  /api/spaces/space  HTTP/1.1
HOST   : XXX:9243
HEADERS:
        Accept: application/json
        Authorization: Basic XXX
        Content-Type: application/json
        Kbn-Xsrf: true
        User-Agent: go-resty/2.7.0 (https://github.com/go-resty/resty)
BODY   :
"eyJpZCI6InRlc3Rfc3BhY2UiLCJuYW1lIjoiVGVzdCBTcGFjZSIsImRlc2NyaXB0aW9uIjoiQSBmcmVzaCBzcGFjZSBmb3IgdGVzdGluZyB2aXN1YWxpc2F0aW9ucyIsImluaXRpYWxzIjoidHMifQ=="
------------------------------------------------------------------------------
~~~ RESPONSE ~~~
STATUS       : 200 OK
PROTO        : HTTP/2.0
RECEIVED AT  : 2023-05-30T21:15:23.8318009+02:00
TIME DURATION: 458.986461ms
HEADERS      :
        Cache-Control: private, no-cache, no-store, must-revalidate
        Content-Length: 134
        Content-Security-Policy: script-src 'self'; worker-src blob: 'self'; style-src 'unsafe-inline' 'self'
        Conten"
2023-05-30T21:15:23.832+0200 [TRACE] provider.stdio: waiting for stdio data
2023-05-30T21:15:23.832+0200 [TRACE] provider.stdio: received data: channel=STDERR len=717
2023-05-30T21:15:23.832+0200 [WARN]  unexpected data: registry.terraform.io/elastic/elasticstack:stderr="t-Type: application/json; charset=utf-8
        Cross-Origin-Opener-Policy: same-origin
        Date: Tue, 30 May 2023 19:15:23 GMT
        Kbn-License-Sig: db4f474762cc21583349b38115caba58aaaa8ebc6ab8c264ec55b3bcde49a9e5
        Kbn-Name: instance-0000000001
        Referrer-Policy: no-referrer-when-downgrade
        X-Cloud-Request-Id: _mr6WbhLSpa-ZMZ4lxnnmg
        X-Content-Type-Options: nosniff
        X-Found-Handling-Cluster: 1609fd9b8cbc403fb0714417c38ed36f
        X-Found-Handling-Instance: instance-0000000001
BODY         :
{
   "id": "test_space",
   "name": "Test Space",
   "description": "A fresh space for testing visualisations",
   "initials": "ts",
   "disabledFeatures": []
}
=============================================================================="
2023-05-30T21:15:23.832+0200 [TRACE] provider.stdio: waiting for stdio data
2023-05-30T21:15:23.916+0200 [DEBUG] provider.terraform-provider-elasticstack_v0.6.1: elasticsearch request [GET https://XXX:9243/] executed. Took 83.977055ms. <nil>: @module=elasticstack tf_provider_addr=registry.terraform.io/elastic/elasticstack tf_req_id=df9cb2ad-c653-677d-ad5c-cb31156b89a2 tf_resource_type=elasticstack_kibana_space @caller=github.com/elastic/terraform-provider-elasticstack/internal/clients/debug.go:37 tf_mux_provider=*schema.GRPCProviderServer tf_rpc=ApplyResourceChange timestamp=2023-05-30T21:15:23.916+0200
2023-05-30T21:15:23.916+0200 [DEBUG] provider.terraform-provider-elasticstack_v0.6.1: elasticsearch API Response for [GET https://XXX:9243/] Details:
---[ RESPONSE ]--------------------------------------
HTTP/2.0 302 Found
Cache-Control: private, no-cache, no-store, must-revalidate
Content-Security-Policy: script-src 'self'; worker-src blob: 'self'; style-src 'unsafe-inline' 'self'
Cross-Origin-Opener-Policy: same-origin
Date: Tue, 30 May 2023 19:15:23 GMT
Kbn-License-Sig: db4f474762cc21583349b38115caba58aaaa8ebc6ab8c264ec55b3bcde49a9e5
Kbn-Name: instance-0000000001
Location: /spaces/space_selector
Referrer-Policy: no-referrer-when-downgrade
X-Cloud-Request-Id: bSkDd5SvTi-WQdYFgHpCKQ
X-Content-Type-Options: nosniff
X-Found-Handling-Cluster: 1609fd9b8cbc403fb0714417c38ed36f
X-Found-Handling-Instance: instance-0000000001
Content-Length: 0


-----------------------------------------------------: @module=elasticstack tf_mux_provider=*schema.GRPCProviderServer tf_resource_type=elasticstack_kibana_space @caller=github.com/elastic/terraform-provider-elasticstack/internal/clients/debug.go:70 tf_provider_addr=registry.terraform.io/elastic/elasticstack tf_req_id=df9cb2ad-c653-677d-ad5c-cb31156b89a2 tf_rpc=ApplyResourceChange timestamp=2023-05-30T21:15:23.916+0200
2023-05-30T21:15:23.916+0200 [TRACE] provider.terraform-provider-elasticstack_v0.6.1: Called downstream: tf_mux_provider=*schema.GRPCProviderServer tf_resource_type=elasticstack_kibana_space tf_rpc=ApplyResourceChange tf_provider_addr=registry.terraform.io/elastic/elasticstack tf_req_id=df9cb2ad-c653-677d-ad5c-cb31156b89a2 @caller=github.com/hashicorp/terraform-plugin-sdk/v2@v2.26.1/helper/schema/resource.go:838 @module=sdk.helper_schema timestamp=2023-05-30T21:15:23.916+0200
2023-05-30T21:15:23.916+0200 [TRACE] provider.terraform-provider-elasticstack_v0.6.1: Received downstream response: tf_req_duration_ms=543 tf_req_id=df9cb2ad-c653-677d-ad5c-cb31156b89a2 tf_resource_type=elasticstack_kibana_space @caller=github.com/hashicorp/terraform-plugin-go@v0.15.0/tfprotov5/internal/tf5serverlogging/downstream_request.go:37 diagnostic_warning_count=0 tf_proto_version=5.3 tf_provider_addr=registry.terraform.io/elastic/elasticstack tf_rpc=ApplyResourceChange @module=sdk.proto diagnostic_error_count=1 timestamp=2023-05-30T21:15:23.916+0200
2023-05-30T21:15:23.916+0200 [ERROR] provider.terraform-provider-elasticstack_v0.6.1: Response contains error diagnostic: @module=sdk.proto diagnostic_severity=ERROR tf_resource_type=elasticstack_kibana_space tf_proto_version=5.3 tf_provider_addr=registry.terraform.io/elastic/elasticstack tf_req_id=df9cb2ad-c653-677d-ad5c-cb31156b89a2 @caller=github.com/hashicorp/terraform-plugin-go@v0.15.0/tfprotov5/internal/diag/diagnostics.go:55 diagnostic_detail= diagnostic_summary="cannot retrieve informations from Elasticsearch" tf_rpc=ApplyResourceChange timestamp=2023-05-30T21:15:23.916+0200
2023-05-30T21:15:23.916+0200 [TRACE] provider.terraform-provider-elasticstack_v0.6.1: Served request: tf_rpc=ApplyResourceChange tf_req_id=df9cb2ad-c653-677d-ad5c-cb31156b89a2 tf_resource_type=elasticstack_kibana_space tf_provider_addr=registry.terraform.io/elastic/elasticstack @caller=github.com/hashicorp/terraform-plugin-go@v0.15.0/tfprotov5/tf5server/server.go:831 @module=sdk.proto tf_proto_version=5.3 timestamp=2023-05-30T21:15:23.916+0200
2023-05-30T21:15:23.917+0200 [TRACE] maybeTainted: elasticstack_kibana_space.example encountered an error during creation, so it is now marked as tainted
2023-05-30T21:15:23.917+0200 [TRACE] NodeAbstractResouceInstance.writeResourceInstanceState to workingState for elasticstack_kibana_space.example
2023-05-30T21:15:23.917+0200 [TRACE] NodeAbstractResouceInstance.writeResourceInstanceState: removing state object for elasticstack_kibana_space.example
2023-05-30T21:15:23.917+0200 [TRACE] evalApplyProvisioners: elasticstack_kibana_space.example is tainted, so skipping provisioning
2023-05-30T21:15:23.917+0200 [TRACE] maybeTainted: elasticstack_kibana_space.example was already tainted, so nothing to do
2023-05-30T21:15:23.917+0200 [TRACE] NodeAbstractResouceInstance.writeResourceInstanceState to workingState for elasticstack_kibana_space.example
2023-05-30T21:15:23.917+0200 [TRACE] NodeAbstractResouceInstance.writeResourceInstanceState: removing state object for elasticstack_kibana_space.example
2023-05-30T21:15:23.917+0200 [TRACE] statemgr.Filesystem: creating backup snapshot at terraform.tfstate.backup
2023-05-30T21:15:23.918+0200 [TRACE] statemgr.Filesystem: state has changed since last snapshot, so incrementing serial to 28
2023-05-30T21:15:23.918+0200 [TRACE] statemgr.Filesystem: writing snapshot at terraform.tfstate
2023-05-30T21:15:23.932+0200 [ERROR] vertex "elasticstack_kibana_space.example" error: cannot retrieve informations from Elasticsearch
2023-05-30T21:15:23.933+0200 [TRACE] vertex "elasticstack_kibana_space.example": visit complete, with errors
2023-05-30T21:15:23.933+0200 [TRACE] dag/walk: upstream of "provider[\"registry.terraform.io/elastic/elasticstack\"] (close)" errored, so skipping
2023-05-30T21:15:23.933+0200 [TRACE] dag/walk: upstream of "root" errored, so skipping
2023-05-30T21:15:23.933+0200 [TRACE] statemgr.Filesystem: have already backed up original terraform.tfstate to terraform.tfstate.backup on a previous write
2023-05-30T21:15:23.933+0200 [TRACE] statemgr.Filesystem: state has changed since last snapshot, so incrementing serial to 29
2023-05-30T21:15:23.933+0200 [TRACE] statemgr.Filesystem: writing snapshot at terraform.tfstate
╷
│ Error: cannot retrieve informations from Elasticsearch
│ 
│   with elasticstack_kibana_space.example,
│   on main.tf line 8, in resource "elasticstack_kibana_space" "example":
│    8: resource "elasticstack_kibana_space" "example" {
│ 
╵
2023-05-30T21:15:23.940+0200 [TRACE] statemgr.Filesystem: removing lock metadata file .terraform.tfstate.lock.info
2023-05-30T21:15:23.940+0200 [TRACE] statemgr.Filesystem: unlocking terraform.tfstate using fcntl flock
2023-05-30T21:15:23.940+0200 [DEBUG] provider.stdio: received EOF, stopping recv loop: err="rpc error: code = Unavailable desc = error reading from server: EOF"
2023-05-30T21:15:23.942+0200 [DEBUG] provider: plugin process exited: path=.terraform/providers/registry.terraform.io/elastic/elasticstack/0.6.1/linux_amd64/terraform-provider-elasticstack_v0.6.1 pid=12566
2023-05-30T21:15:23.942+0200 [DEBUG] provider: plugin exited

Versions (please complete the following information):

• OS: Ubuntu Linux
• Terraform Version 1.4.6
• Provider version 0.6.1
• Elasticsearch Version 8.7.0

[Kushmaro]

@stefanfreitag , i'm not sure if this is the case, but the kibana connection details specify same port as elasticsearch connection details, and usually kibana and es do not communicate over the same port.

[tobio]

Like @Kushmaro says, it looks like the Elasticsearch endpoint is incorrect here.

From the logs, you have the provider thinking it's making a request to Elasticsearch:

elasticsearch API Response for [GET https://XXX:9243/] Details:

but getting a Kibana response:

Kbn-Name: instance-0000000001
Location: /spaces/space_selector

It feels like perhaps your Elasticsearch endpoint it pointing to your Kibana instance.

FWIW, the space resource is created fine against a new Cloud deployment for me with:

resource "ec_deployment" "cluster" {
  region  = "us-east-1"
  version = "8.7.0"
  name    = "tf-342"

  deployment_template_id = "aws-general-purpose-arm-v6"
  elasticsearch = {
    autoscale = "false"
    hot = {
      size        = "1g"
      zone_count  = 1
      autoscaling = {}
    }
  }
  kibana = {}
}

provider "elasticstack" {
  elasticsearch {
    endpoints = ["${ec_deployment.cluster.elasticsearch.https_endpoint}"]
    password  = ec_deployment.cluster.elasticsearch_password
    username  = ec_deployment.cluster.elasticsearch_username
  }
  kibana {
    endpoints = [ec_deployment.cluster.kibana.https_endpoint]
  }
}

resource "elasticstack_kibana_space" "example" {
  space_id    = "test_space"
  name        = "Test Space"
  description = "A fresh space for testing visualisations"
  initials    = "ts"
}

[stefanfreitag]

@tobio @Kushmaro Thanks for the feedback. I will give it a try this evening and provide feedback!

[stefanfreitag]

Managed it - we can close the ticket as invalid. I updated the configuration to make use of the correct urls .es.eu-central-1.aws.cloud.es.io (elastic) and .kb.eu-central-1.aws.cloud.es.io (kibana)