Resource should support Terraform state backends

[ljfranklin]

Backends: https://www.terraform.io/docs/backends/index.html

Unfortunately this resource can't use the built-in remote state backends at all right now. Concourse resources are required to implement a check function that can look for new versions of a resource. To have tighter control over the state files, the resource currently supports only S3. However, now that backends are listed directly in the Terraform config files it becomes even more desirable to support all the backends. I'll have to do some more thinking on this.

[gberche-orange]

We are currently using terraform within our concourse pipeline with the local backend through custom tasks, including a task that pushes terraform state file into git. See orange-cloudfoundry/cf-ops-automation#1 (although we are in the process of pushing our pipeline automation: the repo is still empty today.)

We hope the support for the local backend in this concourse terraform-resource could help us leverage the out meta-data file to push the provisionned resources (say Iaas network id) down in our pipeline (e.g. into the bosh-director cloud-config).

Eventually, we'd like to push the terraform state into directly into credhub, possibly through a future credhub terraform backend (see orange-cloudfoundry/terraform-provider-cloudfoundry#20 (comment)). We'd love to hear if there is interest in such credhub TF backend and credhub TF provider in the cloudfoundry community.

/CC @o-orand @ArthurHlt

[ljfranklin]

@gberche-orange out of curiosity, why did choose git as a backend? If it was mainly for auditing purposes, you could use this resource's output_statefile feature to pull the statefile down and copy it to your git repo. The canonical state file would still live in S3 however.

Still thinking about how to implement this issue but this use case does help show a use case where the build-in S3-compatible backend isn't sufficient.

[ArthurHlt]

EDIT: It was late and I was wrong about the behaviour I have expected from terraform, this is a new way to do the trick (this was my second options). I've made too much change from my first comment that's why I've created new comment

[ArthurHlt]

I'm working with @gberche-orange and I will be happy to give some useful information to help:

First, about the check command, this is not necessary in concourse to implement a check command as you can see here.

But with this resource have a check will be useful and with new changes inside terraform this is pretty easy to create such function without any third-party storage which was created in this repo.

The terraform plan command provide a -detailed-exitcode parameter and if we have the exit code 2 it means that there is a change. The resource can have a dummy behaviour which will simply check if there is a change and create a version from the checksum of the tfplan file.

Like terraform pull/push from remote state file efficiently in command can simply create the tfplan in the given folder by concourse and out command apply it.

---

I didn't answer about your original question when you have answered to @gberche-orange this was, for them, the easiest solution when you don't have any supported backend by terraform that's why they plan to move to another backend and why not a secured one.

[ljfranklin]

First, about the check command, this is not necessary in concourse to implement a check command as you can see here.

The Concourse team is planning a change that all resources must implement check to get metadata, issue.

The terraform plan command...

The terraform plan command only works for a single environment at a time and requires check to know the name of the environment in advance which might not be true if the env_name is randomly generated.

I think the new Terraform workspace feature might be a good fit. I'll need to play with it a bit more but I'm optimistic that the resource will be able to leverage this feature. At the moment, S3 and consul are the only backends that support workspaces but I'd imagine more could be added.

A credhub remote backend sounds nice, but I'd feel uneasy adding support for the local backend as it is too easy to for a concourse user to make a mistake in a bash script and get your state files out of sync.

[ArthurHlt]

Well, i've just misspoke, check command always need to be there even before but there is no obligation to emit a version (so no obligation to have real implementation) which even with those changes will be still true (check command will emit metadata and not anymore in out or in command).

I'm not sure about what you mean with random env_name, could you explain ?
For me, this resource should take care of one environment at a time to follow what official resources do: git-resource manage one repo, bosh-io-release manage one release, cf resource manage one app, ...

As a user i don't need that one resource can manage multiple terraform environment, if i need so i will simply create another resource entry. I can be wrong but I'm wondering what is the usecase to have multiple environments in one configuration ?

It is already possible to encrypt/decrypt state file with S3 from aws and encryption will arrive on terraform (see proposal: hashicorp/terraform#9556 ), it seems that it's easy if user decide to modify directly the tfstate file manually which is not what user want to do, no ?

[ljfranklin]

@ArthurHlt one of the primary use cases of this resource is managing a large pool of uniform environments. This can be done by setting generate_random_name: true and storing the output metadata file in something like the concourse pool-resource. Having a fluid pool of environments allows flexibility to claim an environment temporarily, run some tests, then release the environment back into the pool. Both CI and devs can work in parallel without stepping on the same env. There are other resources that support managing multiple environments like the bosh-deployment-resource and cf-resource.

For encryption, this resource support server_side_encryption to S3 as does the built-in S3 terraform backend. All uploads and downloads to S3 are over HTTPS, so the data is encrypted both a rest and in transit. A general encryption mechanism across all backends would be nice though.

easy if user decide to modify directly the tfstate file manually which is not what user want to do, no ?

Again, I wouldn't want to encourage this as it's easy to make a mistake in a bash script, CI fails before git pushing your state file, and your state files are now out of sync.

[gberche-orange]

thanks @ljfranklin for your answers

@gberche-orange out of curiosity, why did choose git as a backend? If it was mainly for auditing purposes, you could use this resource's output_statefile feature to pull the statefile down and copy it to your git repo. The canonical state file would still live in S3 however.

Thanks for your suggestion to mix S3 and then still push to our git backend. We'll discuss the following impact in our team and will give it a try:

• Having an S3-compatible service as another source of truth (in addition to git where we currently store our secrets until we move them to credhub). Operators sometimes have to read the full tfstate file in addition to outputs for troubleshooting purposes.

I'd feel uneasy adding support for the local backend as it is too easy to for a concourse user to make a mistake in a bash script and get your state files out of sync.

I understand your concern. Our pipeline currently does not allow user scripts to touch the state file directly. Instead modifications need to be pushed to git, preserving auditing and integrity. See related code at https://github.com/orange-cloudfoundry/cf-ops-automation/blob/a0da05b36bce43764f3f98c5072afd1da492be8f/concourse/pipelines/template/depls-pipeline.yml.erb#L979-L1001

But I can see how a native concourse usage could allow this.

Would you be more comfortable to have the terraform-resource supporting a git storage driver instead of a raw local terraform backend ?

[ljfranklin]

Would you be more comfortable to have the terraform-resource supporting a git storage driver instead of a raw local terraform backend ?

In the short-term I'd like to move away completely from the current custom S3 backend and instead leverage the official terraform remote state backends directly: https://www.terraform.io/docs/backends/types/s3.html. I view this resource as a thin wrapper around the Terraform CLI, so I wouldn't want to add a custom backend that doesn't exist in Terraform.

[ljfranklin]

Started working on adding backends, you can see the proposed property changes here. Unfortunately this work is currently blocked as Terraform backends don't allow you to store the plan file in a way that is compatible with this resource. Longer discussion here hashicorp/terraform#16061 which indicates there are long-term plans to store plans in the backends but there's no timeline at this time.

[gberche-orange]

@ljfranklin

A credhub remote backend sounds nice,

FYI, @ArthurHlt produced the following implementation of the terraform http backend backed by credhub: https://github.com/orange-cloudfoundry/terraform-secure-backend This is still early work, and we're welcoming feedback.

[gberche-orange]

@ljfranklin

Unfortunately this work is currently blocked as Terraform backends don't allow you to store the plan file in a way that is compatible with this resource.

Could terraform-resource work with a TF backend if degraded to not yet support in 1st step the plan_only=true and plan_run=true options ?

[ljfranklin]

Could terraform-resourcework with a TF backend if degraded to not yet support in 1st step the plan_only=true and plan_run=true options ?

Yeah. Now that I'm back from vacation I'll try to put some time into finishing the backend support work over the next couple weeks. Once that's done I can push a separate docker image with backend support, but no plan file support. I'll probably still wait to make backends the default until the plan feature works.

[gberche-orange]

great, thanks Lyle

[ljfranklin]

@gberche-orange apologies for the delay on this, was waiting for some Terraform features to land before finishing the first pass on the backend work. Branch is here: https://github.com/ljfranklin/terraform-resource/tree/WIP-tf-backends#backend-beta.

Please give it a spin and let me know if you run into issues, especially confusion around anything in the README.

[gberche-orange]

thanks @ljfranklin , I hope to be able to test with terraform-secure-backendthe and provide feedback in the coming weeks

[iuriaranda]

Hi, I've been testing the new tf backend functionality the last couple of days and so far it's working good. Although the fact that it doesn't support the plan and apply in two steps is sort of a deal-breaker for us.

Maybe a workaround for now would be to give more flexibility to the user, by providing the plan file as an output of the put, and being able to provide the plan as a file in subsequent puts.
So a put with plan_only: true would run plan and put the plan file as an output, then we could use the s3 resource (or whatever other method) to move the plan as an artifact to the next job in the pipeline, where we could provide it in the put as plan_run_file: <the-plan-file>. Not sure if I have explained it correctly 😅

I understand that you don't want to depend on other resources to support the plan&apply in two steps, but in my opinion having that is better than having nothing, and it would be optional anyway. Also, this resource already depends on the pool-resource if you want to manage a pool of environments, so I think this is a similar situation.

[ljfranklin]

providing the plan file as an output of the put...

@iuriaranda Unfortunately, you can't. Somewhat surprisingly a put can't output anything directly, Concourse follows each put with an implicit get to do the downloading. So the only way to pass the plan file from put to get is to persist it somewhere and Terraform doesn't yet support storing plan files in the backend. You could try to bug the terraform folks to see if there's been any progress on that front, but I can't think of any easy way to work around that limitation.

[iuriaranda]

hmmm ok, I see.

Then the options are limited indeed. Either we wait until Terraform supports storing plans in the backend (which doesn't seem to be happening any time soon), or we implement a custom s3-based solution inside this resource, which is not ideal neither. 🤔

[ljfranklin]

@iuriaranda If you're using s3 as your backend, you can go back to using the ljfranklin/terraform-resource:latest image which does have a custom s3 storage backend and supports the plan workflow: https://github.com/ljfranklin/terraform-resource#source-configuration. Or maybe you meant storing the planfile in S3 and the statefile is some other backend.

[iuriaranda]

@ljfranklin yes we're using S3 as our Terraform backend. So if I understand correctly, the custom S3 storage backend implementation in latest is fully compatible with the official Terraform s3 backend?
So, under the hood, I assume it initializes Terraform with the "local" backend, and it's then the resource that uploads the state file to S3. Is that right? Will it also work with the Terraform workspaces?

The reason I wanted to try out the WIP-tf-backends branch of the resource was to leverage Terraform as much as possible, as I didn't like the idea of having a custom s3 backend implementation that might break if Terraform decides to change the way they store state files in the official s3 backend. Although I know that it's unlikely to happen without prior notice.

For now we're just using custom tasks to run Terraform, and moving the plan file from the plan job to the apply job using the S3 resource. But I might give the custom s3 storage backend a try. 👍

[ljfranklin]

the custom S3 storage backend implementation in latest is fully compatible with the official Terraform s3 backend?

Unfortunately no, I shouldn't have used the word "backend" so many times in my previous comment. The latest image knows how to store your statefile in S3, but this s3 storage feature existed in the terraform-resource since well before Terraform Backends existed and the S3 paths used are slightly different. If you need to interact with Terraform statefile both in Concourse and manually via Terraform workspaces AND you need that plan feature, I'm afraid the resource can't support that at the moment.

[iuriaranda]

ok, thanks for the clarification. We'll keep using our own custom implementation for the moment, but we'll keep an eye on this issue as we would like to move to this resource whenever possible in the future.

[kurtmc]

@ljfranklin Thanks for working on this. We have a use case where we would like to set the s3 object acl (this is an option in the terraform s3 backend), so for that reason I have tested out this beta image. It seems to work for that use case (object acl) but we rely on the plan_only feature for pull requests, we don't actually need the plan file at all but it is nice to run plan only for pull requests.

[michaelmeelis]

Hi I'm curious if this is still open and worked on? I'm using concourse and terraform but I use the azure backend and it's blocking me from using this.

[ljfranklin]

I've been kind of dragging my feet in the hopes that Terraform would support storing the plan files in the backend directly. But this issue hasn't moved at all. At this point I'll probably have to add some workaround hack to store the statefile using the backend configuration. The idea that comes to mind is using like this stateful provider to storage some arbitrary string in a state file. That way the resource could generate a plan file then store the contents of that file using the stateful provider. A bit convoluted but seems like it should work. I can't give an ETA on any of this unfortunately.

[kurtmc]

@ljfranklin Thanks for updating this issue. I have been using the ljfranklin/terraform-resource:beta-backend docker image for a while now and it's working well for us. I have used a workaround to run plans for pull requests just using a concourse task and just throwing the plan file away.

If you plan on making any breaking changes to the beta-backend tagged docker image could you let me know so that I can lock the version I currently use in my own dockerhub account and migrate to the latest version when it is ready.

Thanks!

[ljfranklin]

Finally got around to adding plan support to the backend branch using the approach described here. I've merged that branch into master and the ljfranklin/terraform-resource:latest image now supports both backends and plans. Steps to migrate from storage to backend_config is here. Closing this out!