Proposal: "external" provider

[apparentlymart]

We repeatedly see people trying to use external code (usually via local-exec provisioners, but sometimes via wrapper scripts) to extend Terraform for their use-cases in a lighter way than writing provider code in Go.

This is a proposal for a more "official" way to do these lightweight integrations, such that they can work within the primary Terraform workflow.

The crux of this proposal is to define a "gateway interface" between Terraform and external programs, so that a user can a write separate program in any language of their choice that implements the gateway protocol, and have Terraform execute it. This approach is inspired by designs such as inetd and CGI which interact with a child process using simple protocol primitives: environment variables, command line arguments, and stdio streams.

external_data_source data source

data "external_data_source" "example" {
  program     = "${path.module}/example.py"
  interpreter = "python"

  query {
    # Mapping of values that are of significance only to the external program
    foo = "baz"
  }
}

When evaluating this data source, Terraform will create a child process and exec python ./example.py, writing a JSON-encoded version of the contents of query to the program's stdin.

The program should read the JSON payload from stdin, do any work it wants to do, and then print a single valid JSON object to stdout before exiting with a status of zero to indicate success.

Terraform then parses that JSON payload and exposes it in a map attribute result, from which the results can be interpolated elsewhere in the configuration:

    bar = "${external_data_source.example.result.bar}"

It's the responsibility of the developer of the external program to make sure it acts in a side-effect-free fashion, as is expected for data source reads.

external_resource resource

resource "external_resource" "example" {
  program     = "${path.module}/example.py"
  interpreter = "python"

  arguments {
    # Mapping of values that are of significance only to the external program
    foo = "baz"
  }
}

When evaluating this resource, there is a separate protocol for each of the resource lifecycle actions. The common aspect of all of these is that Terraform creates a child process and runs python ./example.py with a single additional argument containing the name of the action: "create", "read", "update" or "delete".

Just as with the data source above, there is a map attribute result. For resources, the result attribute corresponds to Computed attributes in first-class resources, while arguments corresponds to Optional (though the external program may have some additional validation rules it enforces once it's run).

The protocol for each of these actions described in the following sections.

read

For read, the given program is run with read as an argument, and Terraform writes to its stdin a JSON payload like the following, using arguments and id from the existing state:

{
    "id": "abc123",
    "arguments": {
        "foo": "baz"
    }
}

The program must print a valid JSON mapping to its stdout, with top-level keys id, arguments and result, as follows:

{
    "id": "abc123",
    "arguments": {
        "foo": "baz"
    },
    "result": {
        "bar": "xyxy"
    }
}

The program must exit with a status of zero to signal success in order for Terraform to accept the result.

Terraform updates the arguments and result mappings in the state to match what's returned by the program here.

It's the responsibility of the developer of the external program to make sure that read acts in a side-effect-free fashion.

create and update

Create and update follow a very similar protocol: the given program is run with either create or update as a command line argument, and Terraform writes to its stdin a JSON payload like the following:

{
    "id": "abc123",
    "arguments": {
        "foo": "baz"
    },
    "old_arguments": {
        "foo": "baz"
    }
}

In both cases the "arguments" come from the configuration. In the "update" case the "id" and "old_arguments" come from the state, while in the "create" case they are omitted altogether.

After doing whatever side-effects are required to effect the requested change, the program must respond in the same way as for the read action, and Terraform will make the same changes to the state.

delete

Delete is the same as update except that the command line argument is delete and the program is not expected to produce any output on stdout.

If the exit status is zero then Terraform will remove the resource from the state.

Child Process Environment

For both the data source and the resource, the child process inherits the environment variables from the Terraform process, possibly giving it access to e.g. AWS security credentials.

The current working directory is undefined, and so gateway programs shouldn't depend on it. As with most cases in Terraform, it'd be better to pass the result of the file(...) function into the program as an argument rather than have it read files from disk itself, though the program may also choose to build paths relative to its own directory in order to load resources, etc.

Error Handling

If the target program exits with a non-zero status, Terraform collects anything written to stderr and uses it as an error message for failing the operation in question.

For operations where a valid JSON object is expected on stdout, any parse error is also surfaced as an error.

Example Python Data Source

import json
import os

arguments = json.load(os.stdin)
foo = arguments["foo"]

# .. do something with "foo" ..

json.dump({
    "bar": "xyxy",
}, os.stdout)

General Thoughts/Observations

Intended Uses

There are two high-level use-cases for this sort of feature, based on examples I've seen people share elsewhere in the community:

• Terraform supports the service I want to use but doesn't support some detail of it that I care about.
• I want to integrate Terraform with an internal, proprietary system.

For the former case, it would be good if people would still share their use-cases as feature requests in this issue tracker so that we can eventually implement all of the capabilities of the services we support. In this case I hope the user would aspire to eliminate the use of this provider eventually.

The latter case seems like the main legitimate reason to use the "external" provider as a "final solution", particularly if an organization already has tooling in place written in another language and doesn't have the desire or resources to port it to Go.

Effect on the Terraform Ecosystem?

Were this provider to be implemented, it could be used as a crutch for integrating with systems where a Terraform provider is not yet available. Pessimistically, this could cause a lower incentive to implement "first-class" Terraform providers, hurting the Terraform ecosystem in the long run.

However, my expectation is that this interface is clumsy and inconvenient enough that it will be tolerated for short-term solutions to small problems but that there will still be a drive to implement first-class provider support for complex and common services.

Higher-level Abstractions

Just as happened with CGI, it's possible that motivated authors may create higher-level abstractions around this low-level gateway protocol in their favorite implementation language. For example, a small Python library could allow a resource to be implemented as a Python class, automatically dealing with the reading/writing and JSON serialization behind the scenes.

With that said, this protocol is intended to be simple enough to implement without much overhead in most languages, so I expect most people wouldn't bother with this sort of thing and would just code directly to the protocol.

One area that could be interesting is something that can take the arguments and old_arguments properties of the update request payload and make an interface like Terraform's schema.ResourceData, though I'd hope that people would consider writing a real Terraform provider if they find themselves doing something that complicated.

---

Relevant other issues:

• No way to use data "template_file" on a file generated on the fly by local-exec as it fails during planning #8406 use an external program to generate a template for template_file

[mrwilby]

+1 to this bad boy. Would really help obviate the need for complex workarounds.

[sl1pm4t]

@apparentlymart - looks good, and reminds me a lot of how drone.io implemented their plugin system.
A couple of comments:

• In your external_resource example config above, the config block is a terraform data block, rather than resource. Is that what you intended?
• With a Read operation on the external resource, there needs to be a way to signal back to terraform that the real resource has gone away. i.e. could the external provider return an empty ID value, as is common with the internal resource providers?

[radeksimko]

@apparentlymart I understand where you're coming from, it sounds like #1156 I raised a while ago which eventually emerged into getting null_resource documented.

I have mixed feelings when thinking about it ~1year after opening the mentioned issue. Part of me agrees with @phinze

I'm still sort of concerned that implementing either one of these features could end up allowing people to shoot themselves in the foot too much.

and the other part of me agrees with you - #1156 (comment)

---

To comment on the actual proposal:

I would be concerned about breaking cross-platform compatibility + dependency expectations. This is one of Terraform's/Go advantages people don't always realise and they may get bitten by when they try to implement their own Terraform 😃 . e.g. In your example above you just expect the system to provide Python of a particular version, which may grow into expecting some PyPI dependencies too. The same story applies to pretty much any interpreted language (shell included).

[apparentlymart]

@radeksimko yeah, the dependency on python and whatever else is something that would make this hard to use in my company's production environment, where we run Terraform on deliberately-very-minimal dedicated machines.

My assumption here was that people were using local-exec to do these things anyway, so whoever is doing that must already be running Terraform in such a way that external scripts will work. My feeling was that this is just part of why using this resource would not be a magic bullet and that users would have to make some careful tradeoffs in using it (which could be discussed in detail in the docs), one of which would be ensuring that Terraform only runs in appropriately-configured contexts.

(I would hope that in most "production-ish" situations Terraform isn't being running on assorted different machines anyway, and so it shouldn't be incredibly hard to ensure that it has an appropriate environment around it for something like this. That's already true if e.g. you have out-of-tree Terraform plugins that your configuration depends on.)

I think the key difference in this proposal over what you proposed in #1156 is that in your proposal you were essentially just providing an alternative syntax for running arbitrary commands, whereas the protocol described here makes the full data source and resource lifecycles available to external glue code, thus hopefully making these external scripts act as more well-behaved elements of Terraform's model... hopefully resulting in less self-foot-shooting. 😀

Continuing the theme of self-foot-shooting: I suppose having seen loads of examples of what people are doing to work around this problem, I'm feeling like Terraform should provide a recommended path that can come with suitable docs that explain the caveats. That way at least people are going into this with eyes wide open, and not instead getting something to work and then discovering only later the implications of what they built.

[mtougeron]

FWIW, this would be something that I could use to solve me Chef de-provisioning issues from #4121 (comment) after a stack has been destroyed.

[StyleT]

@apparentlymart I suggest instead of supporting multiple interpreters do the following:

• drop interpreter parameter
• replace program with command like in the local-exec

This will allow to use any program you want which supports I/O standard you described in 1st post. And formally TF still will be dependencies free :)

[StyleT]

Also maybe it's reasonable to start with data provider only and implement resource in 2nd development iteration. Looks like external_resource requires much more discussion/development because of complexity. On the other hand data source looks very simple and can be easily implemented.

[apparentlymart]

@StyleT the data source is indeed simpler. I'd like to discuss them both together for a little while because the two should ideally end up having a very similar protocol, but you are right that there's no need for them to be implemented at the same time.