Author a managed resource import/adopt tool

[negz]

What problem are you facing?

Many folks use Terraform to managed their infrastructure. Crossplane can 'adopt' existing infrastructure, including that originally provisioned by Terraform, by statically provisioning a Crossplane managed resource and setting its external name annotation (crossplane.io/external-name) to the name of the existing resource. One challenge with this approach is that the statically provisioned managed resource's configuration must match that of the existing resource; if it does not Crossplane will immediately update the existing resource to match the configuration specified by Crossplane.

How could Crossplane help solve your problem?

Crossplane could provide a tool that translated from a Terraform HCL configuration (or state file?) to the corresponding Crossplane managed resource. For example:

kubectl crossplane import-from terraform --resource google_sql_database_instance --name bigdb

[negz]

I suspect this would fit well in https://github.com/crossplaneio/crossplane-cli, which powers the kubectl crossplane functionality, but I'm going to leave the issue here for now.

[muvaf]

Related to https://github.com/crossplaneio/crossplane/issues/1140

[muvaf]

One challenge with this approach is that the statically provisioned managed resource's configuration must match that of the existing resource; if it does not Crossplane will immediately update the existing resource to match the configuration specified by Crossplane.

A note for the future implementer: you only need to fill the required fields of the CR correctly and leave the rest empty. Crossplane will fetch the empty fields from the provider because of the late-initialization.

[negz]

I'm increasingly thinking it might be more useful to build a tool that can create a Crossplane CR given a running external resource. This would help folks migrate from Terraform, but also help folks migrate any external resource, even those that were not managed via Terraform.

[negz]

Relates to crossplane/docs#287.

[rozcietrzewiacz]

Creating CRs based on given external resources is only half of the problem imho. An even harder thing would be to assure 1:1 configuration parity, including resource linking (such as assigning managed services to the right vpc/security group) that's represented in terraform code. This also ties in with the concept of generic resource references (#1770).

[rozcietrzewiacz]

@negz I see you've changed the title to omit explicit reference to Terraform - I assume this was in relation to your last comment

I'm increasingly thinking it might be more useful to build a tool that can create a Crossplane CR given a running external resource. This would help folks migrate from Terraform, but also help folks migrate any external resource, even those that were not managed via Terraform.

However I think that a generic import tool would better be tackled separately. Consider that terraform code:

• often adds extra information (eg. linking/relations between resources) that might otherwise not be expressed through vendor APIs,
• provides a standardized intermediate layer (either HCL or json) on top of those APIs

Thus the task of importing resources specifically from terraform looks to me better defined and distinct from the task of writing a generic tool.

[rozcietrzewiacz]

@negz Here's another argument why using terraform as a source serves different scenarios: Consider a case where parts of infrastructure belong to different projects even though they are provisioned under the same provider account. On the terraform side, they live in separate git repos, but on the provider side, you don't see that grouping.

For such case it's not useful to rely on a tool that identifies and imports all cloud resources as crossplane CRs. In order to complete terraform->crossplane migration, you'd still need to match individual CRs with respective repositories just as they were defined in terraform.

[muvaf]

In order to import a resource, the following information should be given at the bare minimum:

• Service type
• Identifier in the cloud provider, i.e. the value for crossplane.io/external-name annotation
• Additional values that are required for GET call (for example, region is required in some cases)

The provider-specific tool can query the API and get the required fields and then create a managed resource with retrieved information (that includes values of required fields). However, if the user already has the values for the required fields, then it won't need to make a GET call and it can create the managed resource with that information, let the controller of that resource fill the rest of the fields with late-initialization. So, the key here is how to construct the required fields before the creation of the managed resource object in the cluster.

One option is using terraform state as the information store; tool will read it to get the required fields and print bunch of managed resource YAMLs. However, it needs to be able to match terraform resource kinds & fields to Crossplane kinds & fields, which is not straightforward and easy to automate. IMO, it is not worth the effort.

The other option is to write a tool that will call Observe method of the controller of the resource with a mock managed resource object that's constructed using kind and crossplane.io/external-name annotation. Since there is no validation of the required fields, not having them won't cause a problem unless some of them are needed for the GET call. The advantage of this option is that we'll reuse existing implementation of GET in Observe and a terraform addon on top of this can be built (it'd generate input for that tool by parsing the tf state). But the disadvantage is that it's not always guaranteed that you can make the GET call without the required fields and figuring out what's expected in crossplane.io/external-name can be tricky. The UX would probably look like the following:

# kind corresponds to literal kind of the managed resource, which may not be same as what you see in console.
$ crossplane import gcp --kind CloudSQLInstance --external-name my_database
Fetching infromation from GCP...
my_database/cloudsqlinstance.gcp.crossplane.io created

Then additional tools can be built on top of that because it doesn't depend on another intermediary like TF state. For example, a tool that will list every instance of every service can use this tool to import those resources without having to go through terraform.

[RealHarshThakur]

@muvaf I think it would be nice if we can let user point to the GCP service and we import all the instances in crosspalne. For instance, if user wants to import all CloudSqlInstances, crossplane should be able to do that, it seems like most APIs have the list option to help with that. . An example would look like:
crossplane import-gcp --kind CloudSQLInstance --all

If I interpreted the rest correctly, idea is to parse terraform config which was used to provision the resource to figure out the GET parameters?

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[nalshamaajc]

@negz @jbw976 @muvaf
Any expectations on this one and the migration guide?
Is there anything that can provide speculative insight to help assess whether the created manifest matches the existing resource which could be kind of similar to

kubectl apply -f resource.yaml --dry-run=server

[muvaf]

@nalshamaajc I think using observe-only management policy can get you what you want.

[jeanduplessis]

Related: #3999