Support for data sources

[malthe]

In Terraform, you can declare a data source which is essentially a resource that is externally managed, but must exist.

The Databricks Terraform provider already supports this for a number of resources including schemas, e.g.

data "databricks_schemas" "sandbox" {
  catalog_name = "sandbox"
}

DABs should use this to support defining resources such as schemas that are required to exist in the catalog, but which aren't managed by the bundle.

The motivation is that sometimes we want to reference resources without having the bundle own them (for example schemas).

Syntax

We could introduce a property bind which would default to true.

This is how it would look for a schema resource:

resources:
  schemas:
    bronze:
      bind: false
      name: bronze
      catalog_name: dev

Setting bind: false would mean that the resource should already exist and we'll compare the resource data to the definition.

An alternative would be to introduce a concept of sources which would be in addition to resources. This would allow these sources to be specified by name without a complete definition, similar to Terraform's data sources:

sources:
  schemas:
    bronze:
      name: bronze
      catalog_name: dev

[github-actions]

This issue has not received a response in a while. If you want to keep this issue open, please leave a comment below and auto-close will be canceled.

[aborsato-metco]

This is key for us to be able to create development, temporary schemas.
In mode: development, we want developers to create their own temp schema (i.e. dev_me_bronze) which can be created or dropped at will; while after CICD, we want to deploy and bind to an existing schema, but without managing it (i.e. bronze).

This solution would solve both problems for us is we parametrized:

resources:
  schemas:
    bronze:
      bind: ${var.bind_my_shema}
      name: bronze
      catalog_name: dev

Please, make it happen.