Story: Using terraform resources in Wing code

[eladb]

Summary

Seamlessly use arbitrary Terraform resources in Wing

Feature Spec

You can use any resource from the vast Terraform Registry in your Wing applications.

Let's walk through an example. We will use the pre-built CDKTF provider for GitHub called @cdktf/provider-github.

First, let's install the provider module:

$ npm i @cdktf/provider-github

Now, bring this into your code and define a Repository resource:

bring cloud;
bring git;
bring fs;
bring "@cdktf/provider-github" as github;

let filename = "message.txt";
let repo = new github.repository.Repository(
  name: "my-repo",
  description: "My new github repository",
  visibility: "public",
);

let update_message = (req: cloud.ApiRequest, res: cloud.ApiResponse) => {
  let repo = new git.Repository(fs.mkdtemp());
  repo.clone(repo.git_clone_url, depth: 0);
  fs.File.write(fs.Path.join(repo.dir, filename), req.body));
  repo.commit("update message", add: true);
  repo.push(remote: "origin", branch: "main");
};

let api = cloud.Api();
api.on_put("/messages", update_message);

This is all dandy. If we compile our code to a Terraform target (such as tfaws), this should Just Work™️:

$ wing compile -t tf-aws my-repo.w

The Terraform output will includes:

resource "github_repository" "Repository" {
  name        = "example"
  description = "An awesome example"
  visibility  = "public"
}

But what happens when we try to compile this to the sim target?

$ wing compile -t sim my-repo.w
ERROR: Unable to resolve "sim:@cdktf/provider-github.repository.Repository".
Add a "--resolve sim:@cdktf/provider-github.repository.Repository=./factory.ts"

Implementation note: we will need the simulator synthesizer to proactively look for TerraformResource constructs in the tree in order to produce this error. Impure albeit pragmatic (good catch @Chriscbr).

This error indicates that I need to inject a sim backend for this resource through a TypeScript factory:

OK, let's create one:

// TO BE CONTINUNED

Use Cases

P0 is to simply support terraform resources when compiling to terraform (perhaps this is already supported? Worth adding a test/example...

Next, we should figure out what happens when we set the target to sim. I suspect that this will simply cause the TerraformResource to emit an error saying it couldn't find the TerraformStack scope, isn't it?

So now, what do we do?

Users may want different solutions to this situation:

1. Be able to supply a simulated version of the resource. That's not trivial at the moment because the terraform construct is not a polycon, which means that we don't get to inject a different object upon construction. I am wondering if the Wing compiler can help here by basically shimming every construct with a polycon (some kind of implicit polyconization).
2. Hybrid output - both sim and terraform. To me this seems like a very nice general purpose default solution. Meaning - if I add a terraform resource that is not supported by sim, my output will include a terraform output which the Wing Console (??) will be able to apply when it's starting and the local resources will be wired into the cloud resources. This is a holy grail worth perusing because its the reality of the cloud...
3. Treat as external - in this case we treat the unsupported sim resource as external to the application (meaning - it is already provisioned) and all we need to do is to give the user a way to supply the relevant attributes that will allow inflight code to reach this external resource.

Implementation Notes

No response

Component

No response

[Chriscbr]

What happens when you target sim?

Maybe we should use Terraform for all targets?

Related: #446, #223

[eladb]

Added "use cases" with some thoughts.

[Chriscbr]

Random terraform-specific technical detail - when you import a GitHub resource, I believe a GithubProvider construct needs to be added to the construct tree (one time - see https://developer.hashicorp.com/terraform/cdktf/concepts/providers). We might need some mechanism so that when you write new github.xxx.yyy(...), "If my app doesn't already have a github provider construct, than add it to my construct tree automatically".

Need to double check how this works in practice - if I can just call new GithubProvider as many times as I want under the hood and it doesn't make a difference, then we could just rely on that behavior instead.

(misc: we might also want some mechanism to "hide" some constructs from the tree view by default? kind of like how dotfiles are hidden in your file explorer initially?)

[eladb]

@Chriscbr good point about Terraform providers.

When the provider doesn't take have any required props, we can implicitly add it to the stack as needed, but the question is what happens when there are some required options, or when users want to configure anything there.

It feels like this falls somewhere under the infrastructure policy area, so its basically some kind of HCL that can be merged into the output:

provider "github" {
  app_auth {} 
}

And then:

$ wing compile hello.w --with my-policy.hcl

(-with is made up of course)

we might also want some mechanism to "hide" some constructs from the tree view by default? kind of like how dotfiles are hidden in your file explorer initially?)

Actually just talked about this with @ainvoner, @skyrpex and @polamoros that we want to start adding some annotations to constructs to improve the way they look and feel in the Wing Console. I'll create an issue to start the conversation.

[github-actions]

Hi,

This issue hasn't seen activity in 60 days. Therefore, we are marking this issue as stale for now. It will be closed after 7 days.
Feel free to re-open this issue when there's an update or relevant information to be added.
Thanks!

[eladb]

@Chriscbr I believe this is done, right?

[Chriscbr]

Yep, I think the rest of the use cases (like how to override a terraform resource with a simulator implementation) is addressed by #1365