Make terraform outputs available as variables in tests

[andrewkroh]

This is a feature request to make the data from terraform outputs -json available in system tests. An example use case is a test that creates an S3 bucket and SQS topic and needs information about those resources in the test case.

# _dev/deploy/tf/outputs.tf

output "queue_url" {
  value = aws_sqs_queue.test-queue.url
}

And then be able to reference those values in some way in test parameters.

# _dev/test/system/test-awss3-config.yml

input: aws-s3
data_stream:
  vars:
    access_key_id: "{{{ AWS_ACCESS_KEY_ID }}}"
    secret_access_key: "{{{ AWS_SECRET_ACCESS_KEY }}}"
    queue_url: "{{{ TF_OUTPUT_queue_url }}}"

[cmacknz]

Take a look at Localstack and see if it will cover this without needing real AWS resources https://docs.localstack.cloud/user-guide/aws/s3/

[andrewkroh]

@bhapas has been doing some experimentation. A few key learning points have been:

• Using localstack to emulate AWS S3 and SQS is working as expected for testing purposes.
• Terraform can be used provision the the S3 bucket, SQS queue, and enable ObjectCreated notifications in Localstack.
• However, elastic-package does not allow using both the Terraform service deployer together with the Docker deployer. We need to be able to use them in combination. We need to system test to be able to start Localstack in docker and then run the Terraform deployer.
• The Filebeat aws-s3 input is not working with the format of the queue_url from Localstack. It has some implicit expectations¹ about the format of the hostname in URL. And http://localstack:4566/000000000000/s3-event-notification-queue is not satisfying them.

Footnotes

1. https://github.com/elastic/beats/blob/a927de0fb2463273dbb0126fd9ebdd296a9a26a9/x-pack/filebeat/input/awss3/input.go#L303-L316 ↩

[andrewkroh]

I think there are three independent issues to solve to be able test integrations the use the aws-s3 input in SQS mode.

• Use Terraform outputs as variables in test config. - Add Terraform Outputs to ServiceContext to use them in agent policy test config #1272
  - [ ] Support deploying multiple services in system tests #1302 Tracked separately along with [Filebeat S3 input]Provide support to run integration test against localstack beats#35758
• [Filebeat] aws-s3 input should support any queue_url  beats#35496

The ordering above is semi-relevant. By adding the Terraform outputs feature first we can enable aws-s3 testing against real AWS services. Next only if we can use Terraform and Docker together will we need the input to support the localstack queue_url.

[bhapas]

The primary goal of this issue was to be able to enable terraform outputs for system test config. An additional issue has been created to track the testing against localstack - elastic/beats#35758 and related issue #1302 to support creation of localstack service along with terraform service.

[bhapas]

A bug was identified in the solution and fixed with #1299