-
Notifications
You must be signed in to change notification settings - Fork 770
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
* plugin folder, gitignore, serverless connector example * add support to fast plugin variables and outputs to tfdoc * rename folder, READMEs * add variable description * show diffs * check documentation, use multiple files * debug check doc * try a different glob * debug tfdoc names * more debug * and even more debug * fix gitignore * fix links * support extra files in tests * fix fixture, switch stage 2 peering to new tests * tfdoc * Allow globs in extra files --------- Co-authored-by: Julio Castillo <jccb@google.com>
- Loading branch information
Showing
40 changed files
with
401 additions
and
145 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,2 @@ | ||
| stages/[0-9]*/local-*.tf | ||
| stages-multitenant/[0-9]*/local-*.tf |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,40 @@ | ||
| # VPC Serverless Connector | ||
|
|
||
| This FAST plugin adds centralized [Serverless VPC Access Connectors](https://cloud.google.com/vpc/docs/serverless-vpc-access) to network stages. | ||
|
|
||
| This plugin does not manage | ||
|
|
||
| - IAM bindings for the connectors, which should be added via the stage project-level variables | ||
| - firewall rules for the connectors, which should be added via the stage factory | ||
|
|
||
| The plugin only requires a specific configuration if the defaults it uses need to be changed: | ||
|
|
||
| - the connector-specific subnets default to the `10.255.255.0` range | ||
| - the machine type, number of instances and thoughput use the API defaults | ||
|
|
||
| To enable the plugin, simply copy or link its files in the networking stage. | ||
|
|
||
| <!-- TFDOC OPTS files:1 show_extra:1 --> | ||
| <!-- BEGIN TFDOC --> | ||
|
|
||
| ## Files | ||
|
|
||
| | name | description | modules | resources | | ||
| |---|---|---|---| | ||
| | [local-serverless-connector-outputs.tf](./local-serverless-connector-outputs.tf) | Serverless Connector outputs. | | <code>google_storage_bucket_object</code> · <code>local_file</code> | | ||
| | [local-serverless-connector-variables.tf](./local-serverless-connector-variables.tf) | Serverless Connector variables. | | | | ||
| | [local-serverless-connector.tf](./local-serverless-connector.tf) | Serverless Connector resources. | <code>net-vpc</code> | <code>google_vpc_access_connector</code> | | ||
|
|
||
| ## Variables | ||
|
|
||
| | name | description | type | required | default | producer | | ||
| |---|---|:---:|:---:|:---:|:---:| | ||
| | [serverless_connector_config](local-serverless-connector-variables.tf#L19) | VPC Access Serverless Connectors configuration. | <code title="object({ dev-primary = object({ ip_cidr_range = optional(string, "10.255.255.128/28") machine_type = optional(string) instances = optional(object({ max = optional(number) min = optional(number) }), {}) throughput = optional(object({ max = optional(number) min = optional(number) }), {}) }) prod-primary = object({ ip_cidr_range = optional(string, "10.255.255.0/28") machine_type = optional(string) instances = optional(object({ max = optional(number) min = optional(number) }), {}) throughput = optional(object({ max = optional(number) min = optional(number) }), {}) }) })">object({…})</code> | | <code title="{ dev-primary = {} prod-primary = {} }">{…}</code> | | | ||
|
|
||
| ## Outputs | ||
|
|
||
| | name | description | sensitive | consumers | | ||
| |---|---|:---:|---| | ||
| | [plugin_sc_connectors](local-serverless-connector-outputs.tf#L43) | VPC Access Connectors. | | | | ||
|
|
||
| <!-- END TFDOC --> |
46 changes: 46 additions & 0 deletions
46
fast/plugins/2-networking-serverless-connector/local-serverless-connector-outputs.tf
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,46 @@ | ||
| /** | ||
| * Copyright 2023 Google LLC | ||
| * | ||
| * Licensed under the Apache License, Version 2.0 (the "License"); | ||
| * you may not use this file except in compliance with the License. | ||
| * You may obtain a copy of the License at | ||
| * | ||
| * http://www.apache.org/licenses/LICENSE-2.0 | ||
| * | ||
| * Unless required by applicable law or agreed to in writing, software | ||
| * distributed under the License is distributed on an "AS IS" BASIS, | ||
| * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. | ||
| * See the License for the specific language governing permissions and | ||
| * limitations under the License. | ||
| */ | ||
|
|
||
| # tfdoc:file:description Serverless Connector outputs. | ||
|
|
||
| locals { | ||
| plugin_sc_tfvars = { | ||
| dev = google_vpc_access_connector.dev-primary.0.id | ||
| prod = google_vpc_access_connector.prod-primary.0.id | ||
| } | ||
| } | ||
|
|
||
| # generate tfvars file for subsequent stages | ||
|
|
||
| resource "local_file" "plugin_sc_tfvars" { | ||
| for_each = var.outputs_location == null ? {} : { 1 = 1 } | ||
| file_permission = "0644" | ||
| filename = "${try(pathexpand(var.outputs_location), "")}/tfvars/2-networking-serverless-connnector.auto.tfvars.json" | ||
| content = jsonencode(local.plugin_sc_tfvars) | ||
| } | ||
|
|
||
| resource "google_storage_bucket_object" "plugin_sc_tfvars" { | ||
| bucket = var.automation.outputs_bucket | ||
| name = "tfvars/2-networking-serverless-connnector.auto.tfvars.json" | ||
| content = jsonencode(local.plugin_sc_tfvars) | ||
| } | ||
|
|
||
| # outputs | ||
|
|
||
| output "plugin_sc_connectors" { | ||
| description = "VPC Access Connectors." | ||
| value = local.plugin_sc_tfvars | ||
| } |
52 changes: 52 additions & 0 deletions
52
fast/plugins/2-networking-serverless-connector/local-serverless-connector-variables.tf
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,52 @@ | ||
| /** | ||
| * Copyright 2023 Google LLC | ||
| * | ||
| * Licensed under the Apache License, Version 2.0 (the "License"); | ||
| * you may not use this file except in compliance with the License. | ||
| * You may obtain a copy of the License at | ||
| * | ||
| * http://www.apache.org/licenses/LICENSE-2.0 | ||
| * | ||
| * Unless required by applicable law or agreed to in writing, software | ||
| * distributed under the License is distributed on an "AS IS" BASIS, | ||
| * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. | ||
| * See the License for the specific language governing permissions and | ||
| * limitations under the License. | ||
| */ | ||
|
|
||
| # tfdoc:file:description Serverless Connector variables. | ||
|
|
||
| variable "serverless_connector_config" { | ||
| description = "VPC Access Serverless Connectors configuration." | ||
| type = object({ | ||
| dev-primary = object({ | ||
| ip_cidr_range = optional(string, "10.255.255.128/28") | ||
| machine_type = optional(string) | ||
| instances = optional(object({ | ||
| max = optional(number) | ||
| min = optional(number) | ||
| }), {}) | ||
| throughput = optional(object({ | ||
| max = optional(number) | ||
| min = optional(number) | ||
| }), {}) | ||
| }) | ||
| prod-primary = object({ | ||
| ip_cidr_range = optional(string, "10.255.255.0/28") | ||
| machine_type = optional(string) | ||
| instances = optional(object({ | ||
| max = optional(number) | ||
| min = optional(number) | ||
| }), {}) | ||
| throughput = optional(object({ | ||
| max = optional(number) | ||
| min = optional(number) | ||
| }), {}) | ||
| }) | ||
| }) | ||
| default = { | ||
| dev-primary = {} | ||
| prod-primary = {} | ||
| } | ||
| nullable = false | ||
| } |
84 changes: 84 additions & 0 deletions
84
fast/plugins/2-networking-serverless-connector/local-serverless-connector.tf
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,84 @@ | ||
| /** | ||
| * Copyright 2023 Google LLC | ||
| * | ||
| * Licensed under the Apache License, Version 2.0 (the "License"); | ||
| * you may not use this file except in compliance with the License. | ||
| * You may obtain a copy of the License at | ||
| * | ||
| * http://www.apache.org/licenses/LICENSE-2.0 | ||
| * | ||
| * Unless required by applicable law or agreed to in writing, software | ||
| * distributed under the License is distributed on an "AS IS" BASIS, | ||
| * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. | ||
| * See the License for the specific language governing permissions and | ||
| * limitations under the License. | ||
| */ | ||
|
|
||
| # tfdoc:file:description Serverless Connector resources. | ||
|
|
||
| locals { | ||
| plugin_sc_subnets = { | ||
| dev = { | ||
| for k, v in module.dev-spoke-vpc-serverless.subnets : k => v.name | ||
| } | ||
| prod = { | ||
| for k, v in module.prod-spoke-vpc-serverless.subnets : k => v.name | ||
| } | ||
| } | ||
| } | ||
|
|
||
| module "dev-spoke-vpc-serverless" { | ||
| source = "../../../modules/net-vpc" | ||
| project_id = module.dev-spoke-project.project_id | ||
| name = module.dev-spoke-vpc.name | ||
| vpc_create = false | ||
| subnets = [{ | ||
| name = "access-connector" | ||
| description = "VPC Serverless Connector for the primary region." | ||
| ip_cidr_range = var.serverless_connector_config.dev-primary.ip_cidr_range | ||
| region = var.regions.primary | ||
| }] | ||
| } | ||
|
|
||
| module "prod-spoke-vpc-serverless" { | ||
| source = "../../../modules/net-vpc" | ||
| project_id = module.prod-spoke-project.project_id | ||
| name = module.prod-spoke-vpc.name | ||
| vpc_create = false | ||
| subnets = [{ | ||
| name = "access-connector" | ||
| description = "VPC Serverless Connector for the primary region." | ||
| ip_cidr_range = var.serverless_connector_config.prod-primary.ip_cidr_range | ||
| region = var.regions.primary | ||
| }] | ||
| } | ||
|
|
||
| resource "google_vpc_access_connector" "dev-primary" { | ||
| count = var.serverless_connector_config.dev-primary == null ? 0 : 1 | ||
| project = module.dev-spoke-project.project_id | ||
| region = var.regions.primary | ||
| name = "access-connector-${local.region_shortnames[var.regions.primary]}" | ||
| subnet { | ||
| name = local.plugin_sc_subnets.dev["${var.regions.primary}/access-connector"] | ||
| } | ||
| machine_type = var.serverless_connector_config.dev-primary.machine_type | ||
| max_instances = var.serverless_connector_config.dev-primary.instances.max | ||
| max_throughput = var.serverless_connector_config.dev-primary.throughput.max | ||
| min_instances = var.serverless_connector_config.dev-primary.instances.min | ||
| min_throughput = var.serverless_connector_config.dev-primary.throughput.min | ||
| } | ||
|
|
||
| resource "google_vpc_access_connector" "prod-primary" { | ||
| count = var.serverless_connector_config.prod-primary == null ? 0 : 1 | ||
| project = module.prod-spoke-project.project_id | ||
| region = var.regions.primary | ||
| name = "access-connector-${local.region_shortnames[var.regions.primary]}" | ||
| subnet { | ||
| name = local.plugin_sc_subnets.prod["${var.regions.primary}/access-connector"] | ||
| } | ||
| machine_type = var.serverless_connector_config.prod-primary.machine_type | ||
| max_instances = var.serverless_connector_config.prod-primary.instances.max | ||
| max_throughput = var.serverless_connector_config.prod-primary.throughput.max | ||
| min_instances = var.serverless_connector_config.prod-primary.instances.min | ||
| min_throughput = var.serverless_connector_config.prod-primary.throughput.min | ||
| } |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,25 @@ | ||
| # FAST plugin system | ||
|
|
||
| This folders details a simple mechanism that can be used to add extra functionality to FAST stages, and a few examples that implement simple plugins that can be used as-is. | ||
|
|
||
| ## Available plugins | ||
|
|
||
| ### Networking | ||
|
|
||
| - [Serverless VPC Access Connector](./2-networking-serverless-connector/) | ||
|
|
||
| ## Anatomy of a plugin | ||
|
|
||
| FAST plugins are much simpler and easier to code than full-blown stages: each plugin is meant to add a single feature using a small set of resources, and interacting directly with stage modules and variables. | ||
|
|
||
| A simple plugin might be composed of a single file with one resource, and grow up to the canonical set of one "main" (resources), one variables, and outputs file. | ||
|
|
||
| Plugin file names start with the `local-` prefix which is purposefully excluded in FAST stages via Git ignore, so that plugins are not accidentally committed to stages during development and staying aligned with our master branch is possible. | ||
|
|
||
| Plugins are structured here as individual folders, organized in top-level folders according to the FAST stage they are designed to work with. | ||
|
|
||
| As an example, the [`2-networking/serverless-connector` plugin](./2-networking-serverless-connector/) implements centralized [Serverless VPC Access Connectors](https://cloud.google.com/vpc/docs/serverless-vpc-access) for our networking stages, and is composed of three files: | ||
|
|
||
| - [`local-serverless-connector.tf`](./2-networking-serverless-connector/local-serverless-connector.tf) managing resources including the subnets needed in each VPC and the connectors themselves | ||
| - [`local-serverless-connector-outputs.tf`](./2-networking-serverless-connector/local-serverless-connector-outputs.tf) defining a single `serverless_connectors` output for the plugin, and optional output files | ||
| - [`local-serverless-connector-variables.tf`](./2-networking-serverless-connector/local-serverless-connector-variables.tf) defining a single `serverless_connector_config` variable used to configure the plugin |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -1,4 +1,3 @@ | ||
| data_dir = "../../../fast/stages/2-networking-a-peering/data/" | ||
| automation = { | ||
| outputs_bucket = "test" | ||
| } | ||
|
|
||
Empty file.
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.