Explore using Terraform to implement Crossplane providers

[bassam]

• Feature Request

In a discussion with @armon he suggested that we consider using Terraform modules (or plugins) as a way to implement crossplane resource controllers. I think this is worth exploring even in cases where there might be a reconciler that does more than just CRUD operations on external APIs. If we were to pursue this approach we would need to think about where to store state (tfstate) and if it can be managed at a granular level from inside controllers.

[hasheddan]

@bassam is there still an appetite for this? It would involve some dependency on Terraform, but that might actually be cleaner than being dependent on multiple API's. My sense is that for the major providers we would stick to the API, but potentially use Terraform for others. However, it would reduce some engineering work if we just relied on the work the good folks over at HashiCorp are doing. tfstate is still a question, and we would have to give the ability for users to implement their own state management solutions if it was being managed outside of the controllers (i.e. some users prefer S3 buckets, some prefer HashiCorp's hosted solution)

[bassam]

I just noticed https://github.com/kubeform/kubeform /cc @tamalsaha

[srueg]

Any news on this one? Sounds like an interesting approach to reuse the Terraform ecosystem.

[tamalsaha]

@srueg , we have been working on the terrform based controller. Please feel free to ask in kubeform repo https://github.com/kubeform/project/issues . We have a working demo of individual resources and tf modules at this point.

[bassam]

@srueg would definitely be great to leverage the great work in the terraform community for crossplane. We've explored this lightly at this point, and ran into a few issues. The terraform providers have a number of baked in assumptions that are not compatible with reconcile loop approach we would like to explore. For example, create/deletion patterns, and assumption about what state stored in tfstate. We would love some community help on this, and it looks like @tamalsaha are exploring this with kubeform. We will watch as this develops.

[tamalsaha]

@bassam , what issues you have seen with create/deletion patterns ?

[resouer]

ref: #1488

I personally suggest we can explore how to leverage Terraform modules to make Crossplane be usable in much wider use cases. It's also beneficial to advocate the advantage of "abstraction" idea in Crossplane when users can experience the complexity in raw resources.

@tamalsaha I personally think kubeform is great integration candidate to make this happen.

[tamalsaha]

Thanks @resouer ! I think @kubeform project has shown that Terraform can be used to achieve this. We are looking forward to further extend it. I am happy to discuss further. You can reach me tamal@appscode.com or on Twitter.

[negz]

I'm assigning this to @kasey, who has been doing some R&D in this area.

[resouer]

Thanks Nic. Maybe sth like "external providers" in https://github.com/crossplane-contrib which can easily integrate community operators/orchestrators and use Crossplane as the unified control plane (install, configure, composition and consume).

[kasey]

Synopsis

We investigated using the terraform provider grpc plugin protocol. Once the general approach for talking to the provider over the wire was understood, and nuances of the encoding scheme worked out, I have a prototype that is able to read the current state of a resource, and patterns for translating to/from resource.Managed / yaml / cty.Values. I am confident we can have creation/updating working in short order. While there are still some open questions, detailed below, I think that it makes sense to keep moving in this direction.

Deletion

Crossplane never wants to delete a resource unless the underlying kubernetes object representing the object is deleted. The deletion flag, which we've previously discussed, is used at a lower level representation than what the grpc protocol offers. This means that the strategy of always setting this flag to false is not going to work. We will need to vet the assumptions around deletion to ensure that we never accidentally trigger deletion code paths. We do not know which fields will trigger deletion from schema objects given by the GetSchema RPC (ie the ForceNew attribute is not included). To guard against accidental deletion, we can use the PlanResourceChange RPC to check if a deletion would occur. The response from this RPC includes a structured representation of the plan for deletion, inc. indicating fields that could have triggered deletion if changed (up to the client code to filter that list down to the fields that really did change).

Terraform ID / external-name

Terraform Resource "Id" fields are analogous to our external-name annotation. Terraform resources internally implement logic to generate the Id field in the create and import methods, but we have no way to inspect this logic to generate validation. When specified by a user, we may want to use the ImportResourceState GRPC method to validate the key, and also keep an additional piece of state representing whether the value has been validated, before we attempt to use it to read a resource.

We have a convention where we exclude parameters which replicate information contained in external-name. For instance, in the case of google service accounts, we excluded the account_id field from the resource, because it could be instead derived from an external id. In the case of terraform providers, I suggest we take no special effort to exclude these values from our resources and allow terraform to construct them as usual, using external-name to hold the Id value returned by the Terraform provider, or to let the user specify an Id value, analogous to running terraform import.

Documentation

Terraform has a separate tree of documentation for resources, so this doesn't align easily with our strategy of generating documentation from CRD go types. The path of least resistance is likely to generate skeleton docs that infer the canonical structure of terraform docs to generate links between our CRD docs and their resource docs. However we should look at terraform's doc site generation process to see how difficult it would be to incorporate their resource documentation more natively into ours.

Name Casing

Terraform field resource and field names are snake_case, whereas we follow k8s API conventions in making our field names. We'll want to preserve the original name of a resource in the type registry, store original field names in struct tags.

References

Terraform does not expose references in a way that we can infer from looking at the schema. We will need some human intervention to create structured data that the code generation tool can read to manage references. One possibility I would like to explore is an API for retrieving values from resource.Managed objects which supports callbacks that override the default behavior of retrieving a value from the generated Resource object.
Options:

• Provide injection points to add hand-written or customizable reference resolution callbacks
• Define metadata format for describing references and building a system to generate provider controller code to create resolution methods
• Externalize references into a new design / controller
• Extend Composition to resolve lazy references

Readiness

This concept is not part of the Terraform data model. The controller could use the presence of Terraform's ID field to determine if the resource has been provisioned by the cloud provider.

Note that currently resource references rely on resources being ready before they are considered resolvable. We also have a deeper notion of readiness that relies on eg successfully connecting to a database. In order to reproduce these kinds of checks we will also need to introduce hand-written code.

Version Management

In order to circumvent any compatibility issues with different versions of provider plugins, we'll want to download the terraform plugin and package the combined generated code + provider plugin binary in an image for build/test/distribution. The process of upgrading to newer versions of a terraform provider is still uncertain as we don't know what kinds of changes could break backwards compatibility, but given the workflow we know as users of terraform this shouldn't pose any significant issues.

Nesting

Some resources are deeply nested, for an example look at google_container_cluster. The most straightforward way to translate these resources will be for our CRDs to specify an identically nested representation.

Configuration

Many resources offer a Configuration block, often allowing provider API call timeouts to be customized on a per-resource basis. I suggest we exclude these from the user-facing CRDs, although our generated code will need to understand them in order to generate valid resource objects.

Extensibility Model

A forthcoming design doc will show how we intend to mix generated code with handwritten code, to allow additional functionality to be added to resources.

Licensing

Make sure that the terraform licensing scheme doesn't preclude us from integration with the provider binaries.

Secret Ref

Similar to Readiness, we will need to implement secret management as injectable code, to be written by humans after the fact.

[negz]

We're using Terraform in a number of ways these days!

https://github.com/crossplane-contrib/provider-terraform-vsphere
https://github.com/crossplane-contrib/provider-terraform

[negz]

I'd like a github.com/crossplane issue we can use to track our efforts around Terraform-wrapped providers on our project board. We've had a lot of efforts in this area in the past so there's quite a few different existing (closed) issues. I figured I'd open this original issue to use it to track. :)

#2536 reflects our current area of exploration.

[muvaf]

We're implementing the design referenced earlier and here is the repository that we're using.