What are best practices for making Snowflake Terraform modules?

[jmkacz]

Cross-posting from https://community.snowflake.com/s/question/0D5VI000005xtFR0AY/what-are-best-practices-for-making-snowflake-terraform-modules as the communities might not overlap significantly.

Background, we went through the Snowflake Quickstart, Terraforming Snowflake (https://quickstarts.snowflake.com/guide/terraforming_snowflake/index.html#0).

We want to make a helper module for creating schemas, as there's a pile of common configurations we want create for every new schema (roles, privileges, etc). We have separate providers for each of SECURITYADMIN, SYSADMIN, and USERADMIN, as we are following Snowflake best practices and not creating everything under ACCOUNTADMIN.

A couple of questions,

1. Do we define the providers in both our main configuration (because we'll be creating snowflake resources in our main code as well as in the module), and in this module?
2. Do we define the providers in our main configuration and the module implicitly uses them?
3. Do we define the providers in our main configuration and explicitly pass them to the module on every invocation?

Example for the third option:

module "foo_bar_schema" {
  source = "../../../modules/snowflake/snowflake_schema/v1"
  schema_name = "bar_schema"
  schema_database = "foo_db"
  providers = {
    snowflake.user_admin = snowflake.user_admin
    snowflake.sys_admin = snowflake.sys_admin
    snowflake.security_admin = snowflake.security_admin
  }
}

The right-hand side of these providers passed in will not change and will be the same for all of these invocations.

This came from a discussion on this documentation, https://developer.hashicorp.com/terraform/language/modules/develop/providers#providers-within-modules

There is nuanced difference though. In the Hashicorp documentation, they are switching between providers, say for different aws regions; whereas we're not switching providers.

So, net, what is the recommended way to build a helper module for the snowflake terraform plugin?

[sfc-gh-swinkler]

@jmkacz hello! I hope you are doing well. In regards to your questions, I would highly discourage method 1. Although this is still supported for legacy reasons, it is not considered current best practice. Method 2 is only an option if there is a default provider (no alias). This works if you make multiple modules, each using only one implicit provider. You could still have "foo_bar_schema" module as described above, but internally it would be composed of three modules, each using a different implicit provider. This is convenient because you do not need to add any additional blocks to resources within these modules, but it does require you to split up your Terraform code, which make not make sense with your current architecture.

snowflake_schema module

module "user_admin" {
  source    = "./user_admin"
  providers = {
    snowflake = snowflake.user_admin
  }
}
module "sys_admin" {
  source    = "./sys_admin"
  providers = {
    snowflake = snowflake.sys_admin
  }
}
module "security_admin" {
  source    = "./security_admin"
  providers = {
    snowflake = snowflake.security_admin
  }
}

Method 3 is also a valid option, but it would require you to explicitly select the provider for each resource, which is a bit more code. This is what is done in the Terraforming Snowflake tutorial, but personally I do not like it as much.

Note that you could mix method 2 and method 3, just pick a provider to be the default. Finally, I would not thinking of it as "pass them to the module on every invocation" as much as "pass reference to each module for use during apply" since the plugin still only gets initialized once, not multiple times.

[jmkacz]

Sorry, I'm not sure I'm following. Are you suggesting that code snippet live in my main.tf or in ../../../modules/snowflake/snowflake_schema/v1/main.tf?

[sfc-gh-swinkler]

Both options are valid, but in the comment I was referring to the latter option. So in your main.tf (root module) you would still have.

module "foo_bar_schema" {
  source = "../../../modules/snowflake/snowflake_schema/v1"
  schema_name = "bar_schema"
  schema_database = "foo_db"
  providers = {
    snowflake.user_admin = snowflake.user_admin
    snowflake.sys_admin = snowflake.sys_admin
    snowflake.security_admin = snowflake.security_admin
  }
}

and then in the ../../../modules/snowflake/snowflake_schema/v1/main.tf file you would have

module "user_admin" {
  source    = "./user_admin"
  providers = {
    snowflake = snowflake.user_admin
  }
}
module "sys_admin" {
  source    = "./sys_admin"
  providers = {
    snowflake = snowflake.sys_admin
  }
}
module "security_admin" {
  source    = "./security_admin"
  providers = {
    snowflake = snowflake.security_admin
  }
}

then you would have three more children modules each defined in a folder local to that module, for example: ../../../modules/snowflake/snowflake_schema/v1/main.tf/modules/security_admin
../../../modules/snowflake/snowflake_schema/v1/main.tf/modules/sys_admin
../../../modules/snowflake/snowflake_schema/v1/main.tf/modules/account_admin

In each of those modules you would just declare the resources you need, implicitly using the provider that was passed to that module.
Hope this helps.

[jmkacz]

Ah, thank your for your efforts here but that doesn't address my core concern. Please let me try to clarify.

It seems like the proposed solution is focused on not setting the provider line, in the module code itself. This is not something that concerns me; I'm fine with setting the specific provider on these individual resources.

What I'm concerned about is using the module in my main.tf file. The providers section below feels overly verbose as it is static and copied between every usage of the module. It feels like there should be a simpler, more concise way to call the module in Terraform, but I could be mistaken.

module "foo_bar_schema" {
  source = "../../../modules/snowflake/snowflake_schema/v1"
  schema_name = "bar_schema"
  schema_database = "foo_db"
  providers = {
    snowflake.user_admin = snowflake.user_admin
    snowflake.sys_admin = snowflake.sys_admin
    snowflake.security_admin = snowflake.security_admin
  }
}

[sfc-gh-swinkler]

@jmkacz i understand now, thank you for clarifying. Unfortunately it's not possible to be more concise. Would be nice if this could be set to a variable block like;

locals {
    sf_providers = {
      snowflake.user_admin = snowflake.user_admin
      snowflake.sys_admin = snowflake.sys_admin
      snowflake.security_admin = snowflake.security_admin
   }
}

module "a" {
    source = "./modules/a"
    providers = locals.sf_providers
}

but the above does not actually work, if you try this you will get the error message: Error: Invalid expression: A static map expression is required.. Provider initialization is a bit more strict compared to other things.

The only alternative I can suggest is maybe you could have multiple terraform workspaces? Each workspace would utilize only a single Terraform role. I understand this is kind of like asking you to go one step forward and two steps back, but this is a limitation of Terraform.

[harukitake]

@sfc-gh-swinkler Hello!
What type of user do you expect to apply that?
For example, you prepare a user for running terraform and grant sysadmin, security admin, and useradmin to the user.

I think account admin is also necessary like when creating a storage integration.
https://docs.snowflake.com/ja/user-guide/security-access-control-privileges#global-privileges-account-level-privileges

[sfc-gh-asawicki]

Hey @harukitake.

There are also situations where even the ORGADMIN is needed (e.g. https://docs.snowflake.com/en/sql-reference/sql/alter-account).

Our recommendation is to always consult the Snowflake docs and check there what the minimal required role for the given object is.

We are also considering for the future the possibility of setting the role used inside each resource (essentially by invoking USE ROLE before any other statements run inside the resource), but this is a matter of months, not weeks.

[harukitake]

@sfc-gh-asawicki Thanks!

We are also considering for the future the possibility of setting the role used inside each resource (essentially by invoking USE ROLE before any other statements run inside the resource), but this is a matter of months, not weeks.

That's a great idea.