Use Terraform for System Testing

[feluelle]

Description

As @ashb @potiuk and me already shortly discussed in a slack thread..

We could use terraform for testing Airflow integrations with real services from providers.
Key benefits:

• Write declarative configuration files
• Plan and predict changes
• Create reproducible infrastructure

I didn't think about the implementation details (pytest + terraform) yet, but I think it would make it a lot easier using terraform instead of having to write a lot of system helper classes in python.

EDIT:

Some interesting resources:

• https://www.fugue.co/blog/blog-post-creating-an-automated-cloud-infrastructure-testing-tool-with-terraform-and-pytest

[boring-cyborg]

Thanks for opening your first issue here! Be sure to follow the issue template!

[potiuk]

Absolutely!

[szha]

HashiCorp recently added a clause to their term of evaluation that forbids the usage of their enterprise software in China. While this may or may not affect this proposal, it does pose potential legal risk to contributors in China.

Update:
HashiCorp revised its terms to only include the enterprise version of vault.

[turbaszek]

I think it would make it a lot easier using terraform instead of having to write a lot of system helper classes in python.

In my opinion it depends... Is creating a single GCS bucket worth adding terraform? However, I fully support the idea for usting terraform to setup complex environment.

[potiuk]

Fully Agree with Tomek. Especially with some setup/teardown that is quick and can be repeated for each test every time, using terraform makes no sense. But when you have longer setup that is more complex and takes quite some time, this might be definitely worthwhile.

The usual scenario for that will be

a) create the environment -> terraform script

b) iterate on the tests
alternating with
c) potentially changing target state of the environment - updating and re-running the terraform script applying the changes.

d) tear-down the environment (using terraform script to delete the environment).

This already pretty much reflects our complex "helpers" for system tests - basically when we really need a helper, seems that terraform scripts might to better job.

[feluelle]

So what I have in mind is that the def setUpClass automatically terraform init and applys the infra located in tests/providers/*/infrastructure for a file named as the example dag file and should end with .tf if exists

What we could also do would be only running the terraform init fortests/providers/*/infrastructure once for all example dags when the providers system flag is used --system amazon for example.

Is creating a single GCS bucket worth adding terraform?

@turbaszek in my opinion we should do it consistenly for all example_dags. Do you think it would be slow or just superfluous? Or what is your concern?
Under the hood terraform is also using the cli's we use to create resources for aws and google, right? So why not do it safely and more manageable using terraform? I think the biggest plus is that the infra automatically gets destroyed so we do not accidently let something exists longer as the test needs it.

[turbaszek]

So what I have in mind is that the def setUpClass automatically terraform init and applys the infra located in tests/providers/*/infrastructure for a file named as the example dag file and should end with .tf if exists

I like this idea!

Is creating a single GCS bucket worth adding terraform?

@turbaszek in my opinion we should do it consistenly for all example_dags. Do you think it would be slow or just superfluous? Or what is your concern?

My main concern is that normally I would never use terraform to create single bucket. But it's not a strong opinion. I'm in favor of unified and opinionated approach 👌

I think the biggest plus is that the infra automatically gets destroyed so we do not accidently let something exists longer as the test needs it.

Theoretically tearDown should always be executed so currently all infrastructure should also be destroyed. However, the advantage of using .tf is that everything we defined will be created and destroyed, so no place for discrepancies between setUp and tearDown.

[feluelle]

I have a question: When I am adding terraform to airflow and I want it through docker like it was done here what volumes should I mount? I need to mount the .aws, .azure, etc. folder but also the folders containing the .tf files. How should I do it? Mount the whole airflow dir?

WDYT? @potiuk @turbaszek :)

[potiuk]

Yeah. Airflow sources are best. But for that, you will need to create a HOST_AIRFLOW_SOURCES variable (similarly as I do with HOST_HOME) as an environment variable and mount it as /opt/airlfow.

[potiuk]

And BTW. IT would be great to also add this mapping -v ${HOST_ARIFLOW_SOURCES}:/opt/airflow also to the other commands. We might wan to upload stuff to gcloud/aws as well.

[feluelle]

Terraform was introduced during #8877

For further integration i.e. automatisation of system tests we can also use terraform but that should be a separate issue.