Skip to content

Commit

Permalink
feat!: split network step (#735)
Browse files Browse the repository at this point in the history
* adding copy of 3-network folder for hub-and-spoke definition

* Removing unnecessary hub-and-spoke setup on 3-networks step

* Adding script to undo the disable_tf_files.sh removal

* adding 3-networks-hub-and-spoke directory to scripts disable/restore

* Adding 3-networks-hub-and-spoke/shared to disable/restore  scripts

* Adding hub-and-spoke IT for networks and shared test folders

* Adding conditional for specific netwotk folder based on environment mode

* adjusting 3-networks-hub-and-spoke to only contain this topology

* Removing unnecessary additional test files

* Adjusting projects test to hit the specific networks folder

* Renaming 3-networks to 3-networks-dual-svpc

* Making the scripts sensitive to the TF_VAR_example_foundations_mode env

* Removing hub and spoke references

* Adjusting hub-and-spoke .tf setup

* generate docs

* Updating readme with new folders

* Internal review issues changes

* Cherry pick from "Feature/private service connect module"

* Lint fixes

* Internal PR small issues

* Adding missing tfvars file for environments on new networks folders

* Updating network version of the modules in advance to merge conflict

* Removing unnecessary transitivity module and adjusting some readme details

* "network architecture" description updated on readme

* Removing some unnecessary conditional flow from hub and spoke

* Resolving the conflicts from master

* Corrections after generate docs and lint

* FIxing some more lint issues

* Fixing wrong conditional removes

* Generate docs fix

* putting enable HS transitivity back
  • Loading branch information
mauro-cit authored Jul 6, 2022
1 parent b11b987 commit 512430b
Show file tree
Hide file tree
Showing 216 changed files with 6,694 additions and 148 deletions.
11 changes: 9 additions & 2 deletions 0-bootstrap/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -26,13 +26,20 @@ organizational policy.</td>
Google Cloud organization that you've created.</td>
</tr>
<tr>
<td><a href="../3-networks">3-networks</a></td>
<td><a href="../3-networks-dual-svpc">3-networks-dual-svpc</a></td>
<td>Sets up base and restricted shared VPCs with default DNS, NAT (optional),
Private Service networking, VPC service controls, on-premises Dedicated
Interconnect, and baseline firewall rules for each environment. Also sets
Interconnect, and baseline firewall rules for each environment. It also sets
up the global DNS hub.</td>
</tr>
<tr>
<td><a href="../3-networks-hub-and-spoke">3-networks-hub-and-spoke</a></td>
<td>Sets up base and restricted shared VPCs with all the default configuration
found on step 3-networks-dual-svpc, but here the architecture will be based on the
Hub and Spoke network model. It also sets up the global DNS hub</td>
</tr>
</tr>
<tr>
<td><a href="../4-projects">4-projects</a></td>
<td>Set up a folder structure, projects, and application infrastructure pipeline for applications,
which are connected as service projects to the shared VPC created in the previous stage.</td>
Expand Down
9 changes: 8 additions & 1 deletion 1-org/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -26,13 +26,20 @@ organizational policy.</td>
Google Cloud organization that you've created.</td>
</tr>
<tr>
<td><a href="../3-networks">3-networks</a></td>
<td><a href="../3-networks-dual-svpc">3-networks-dual-svpc</a></td>
<td>Sets up base and restricted shared VPCs with default DNS, NAT (optional),
Private Service networking, VPC service controls, on-premises Dedicated
Interconnect, and baseline firewall rules for each environment. It also sets
up the global DNS hub.</td>
</tr>
<tr>
<td><a href="../3-networks-hub-and-spoke">3-networks-hub-and-spoke</a></td>
<td>Sets up base and restricted shared VPCs with all the default configuration
found on step 3-networks-dual-svpc, but here the architecture will be based on the
Hub and Spoke network model. It also sets up the global DNS hub</td>
</tr>
</tr>
<tr>
<td><a href="../4-projects">4-projects</a></td>
<td>Sets up a folder structure, projects, and application infrastructure pipeline for applications,
which are connected as service projects to the shared VPC created in the previous stage.</td>
Expand Down
11 changes: 9 additions & 2 deletions 2-environments/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -26,13 +26,20 @@ organizational policy.</td>
Google Cloud organization that you've created.</td>
</tr>
<tr>
<td><a href="../3-networks">3-networks</a></td>
<td><a href="../3-networks-dual-svpc">3-networks-dual-svpc</a></td>
<td>Sets up base and restricted shared VPCs with default DNS, NAT (optional),
Private Service networking, VPC service controls, on-premises Dedicated
Interconnect, and baseline firewall rules for each environment. It also sets
up the global DNS hub.</td>
</tr>
<tr>
<td><a href="../3-networks-hub-and-spoke">3-networks-hub-and-spoke</a></td>
<td>Sets up base and restricted shared VPCs with all the default configuration
found on step 3-networks-dual-svpc, but here the architecture will be based on the
Hub and Spoke network model. It also sets up the global DNS hub</td>
</tr>
</tr>
<tr>
<td><a href="../4-projects">4-projects</a></td>
<td>Sets up a folder structure, projects, and application infrastructure pipeline for applications,
which are connected as service projects to the shared VPC created in the previous stage.</td>
Expand Down Expand Up @@ -131,7 +138,7 @@ commands. The `-T` flag is needed for Linux, but causes problems for MacOS.
git push origin production
```
1. Review the apply output in your cloud build project https://console.cloud.google.com/cloud-build/builds?project=YOUR_CLOUD_BUILD_PROJECT_ID
1. You can now move to the instructions in the step [3-networks](../3-networks/README.md).
1. You can now move to the instructions in the step go to for the Dual Shared VPC mode [3-networks-dual-svpc](../3-networks-dual-svpc/README.md), or go to [3-networks-hub-and-spoke](../3-networks-hub-and-spoke/README.md) to use the Hub and Spoke network mode.

### Deploying with Jenkins

Expand Down
File renamed without changes.
30 changes: 19 additions & 11 deletions 3-networks/README.md → 3-networks-dual-svpc/README.md
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
# 3-networks
# 3-networks-dual-svpc

This repo is part of a multi-part guide that shows how to configure and deploy
the example.com reference architecture described in
Expand Down Expand Up @@ -26,13 +26,20 @@ organizational policy.</td>
Google Cloud organization that you've created.</td>
</tr>
<tr>
<td>3-networks (this file)</td>
<td><a>3-networks-dual-svpc (this file)</a></td>
<td>Sets up base and restricted shared VPCs with default DNS, NAT (optional),
Private Service networking, VPC service controls, Dedicated or Partner
Private Service networking, VPC service controls, on-premises Dedicated
Interconnect, and baseline firewall rules for each environment. It also sets
up the global DNS hub.</td>
</tr>
<tr>
<td><a href="../3-networks-hub-and-spoke">3-networks-hub-and-spoke</a></td>
<td>Sets up base and restricted shared VPCs with all the default configuration
found on step 3-networks-dual-svpc, but here the architecture will be based on the
Hub and Spoke network model. It also sets up the global DNS hub</td>
</tr>
</tr>
<tr>
<td><a href="../4-projects">4-projects</a></td>
<td>Sets up a folder structure, projects, and application infrastructure pipeline for applications,
which are connected as service projects to the shared VPC created in the previous stage.</td>
Expand Down Expand Up @@ -80,29 +87,30 @@ commands. The `-T` flag is needed for Linux, but causes problems for MacOS.

### Networking Architecture

You need to set variables `enable_hub_and_spoke` and `enable_hub_and_spoke_transitivity` to `true` to be able to use the **Hub-and-Spoke** architecture detailed in the **Networking** section of the [Google cloud security foundations guide](https://services.google.com/fh/files/misc/google-cloud-security-foundations-guide.pdf).
This step makes use of the **Dual Shared VPC** architecture, and more details can be found described at the **Networking** section of the [Google cloud security foundations guide](https://cloud.google.com/architecture/security-foundations/networking). To see the version that makes use the Hub and Spoce mode, check the step [3-networks-hub-and-spoke](../3-networks-hub-and-spoke).


### Using Dedicated Interconnect

If you provisioned the prerequisites listed in the [Dedicated Interconnect README](./modules/dedicated_interconnect/README.md), follow these steps to enable Dedicated Interconnect to access on-premises resources.

1. Rename `interconnect.tf.example` to `interconnect.tf` in base_env folder in `3-networks/modules/base_env`.
1. Rename `interconnect.tf.example` to `interconnect.tf` in base_env folder in `3-networks-dual-svpc/modules/base_env`.
1. Update the file `interconnect.tf` with values that are valid for your environment for the interconnects, locations, candidate subnetworks, vlan_tag8021q and peer info.
1. The candidate subnetworks and vlan_tag8021q variables can be set to `null` to allow the interconnect module to auto generate these values.

### Using Partner Interconnect

If you provisioned the prerequisites listed in the [Partner Interconnect README](./modules/partner_interconnect/README.md) follow this steps to enable Partner Interconnect to access on-premises resources.

1. Rename `partner_interconnect.tf.example` to `partner_interconnect.tf` in the base-env folder in `3-networks/modules/base_env` .
1. Rename `partner_interconnect.tf.example` to `partner_interconnect.tf` in the base-env folder in `3-networks-dual-svpc/modules/base_env` .
1. Update the file `partner_interconnect.tf` with values that are valid for your environment for the VLAN attachments, locations, and candidate subnetworks.
1. The candidate subnetworks variable can be set to `null` to allow the interconnect module to auto generate this value.

### OPTIONAL - Using High Availability VPN

If you are not able to use Dedicated or Partner Interconnect, you can also use an HA Cloud VPN to access on-premises resources.

1. Rename `vpn.tf.example` to `vpn.tf` in base-env folder in `3-networks/modules/base_env`.
1. Rename `vpn.tf.example` to `vpn.tf` in base-env folder in `3-networks-dual-svpc/modules/base_env`.
1. Create secret for VPN private preshared key.
```
echo '<YOUR-PRESHARED-KEY-SECRET>' | gcloud secrets create <VPN_PRIVATE_PSK_SECRET_NAME> --project <ENV_SECRETS_PROJECT> --replication-policy=automatic --data-file=-
Expand All @@ -127,7 +135,7 @@ If you are not able to use Dedicated or Partner Interconnect, you can also use a
```
1. Copy contents of foundation to new repo.
```
cp -RT ../terraform-example-foundation/3-networks/ .
cp -RT ../terraform-example-foundation/3-networks-dual-svpc/ .
```
1. Copy Cloud Build configuration files for Terraform.
```
Expand Down Expand Up @@ -204,7 +212,7 @@ If you are not able to use Dedicated or Partner Interconnect, you can also use a
```
1. Copy contents of foundation to new repo.
```
cp -RT ../terraform-example-foundation/3-networks/ .
cp -RT ../terraform-example-foundation/3-networks-dual-svpc/ .
```
1. Copy the Jenkinsfile script to the root of your new repository.
```
Expand Down Expand Up @@ -267,7 +275,7 @@ If you are not able to use Dedicated or Partner Interconnect, you can also use a

### Run Terraform locally

1. Change into the 3-networks folder.
1. Change into the 3-networks-dual-svpc folder.
1. Run `cp ../build/tf-wrapper.sh .`
1. Run `chmod 755 ./tf-wrapper.sh`.
1. Rename `common.auto.example.tfvars` to `common.auto.tfvars` and update the file with values from your environment and bootstrap. See any of the envs folder [README.md](./envs/production/README.md) files for additional information on the values in the `common.auto.tfvars` file.
Expand All @@ -280,7 +288,7 @@ If you are not able to use Dedicated or Partner Interconnect, you can also use a
You can run `terraform output gcs_bucket_tfstate` in the 0-bootstrap folder to obtain the bucket name.

We will now deploy each of our environments(development/production/non-production) using this script.
When using Cloud Build or Jenkins as your CI/CD tool each environment corresponds to a branch in the repository for 3-networks step
When using Cloud Build or Jenkins as your CI/CD tool each environment corresponds to a branch in the repository for 3-networks-dual-svpc step
and only the corresponding environment is applied.

To use the `validate` option of the `tf-wrapper.sh` script, please follow the [instructions](https://github.com/GoogleCloudPlatform/terraform-validator/blob/main/docs/install.md) in the **Install Terraform Validator** section and install version `v0.4.0` in your system. You will also need to rename the binary from `terraform-validator-<your-platform>` to `terraform-validator` and the `terraform-validator` binary must be in your `PATH`.
Expand Down
27 changes: 27 additions & 0 deletions 3-networks-dual-svpc/common.auto.example.tfvars
Original file line number Diff line number Diff line change
@@ -0,0 +1,27 @@
/**
* Copyright 2021 Google LLC
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/

org_id = "000000000000"

terraform_service_account = "org-terraform@prj-b-seed-2334.iam.gserviceaccount.com"

// The DNS name of peering managed zone. Must end with a period.
domain = "example.com."

// Optional - for an organization with existing projects or for development/validation.
// Must be the same value used in previous steps.
//parent_folder = "000000000000"

46 changes: 46 additions & 0 deletions 3-networks-dual-svpc/envs/development/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,46 @@
# 3-networks-dual-svpc/development

The purpose of this step is to set up base and restricted shared VPCs with default DNS, NAT (optional), Private Service networking, VPC service controls, onprem Dedicated Interconnect, onprem VPN and baseline firewall rules for environment development.

## Prerequisites

1. 0-bootstrap executed successfully.
1. 1-org executed successfully.
1. 2-environments/envs/development executed successfully.
1. 3-networks-dual-svpc/envs/shared executed successfully.
1. Obtain the value for the access_context_manager_policy_id variable. Can be obtained by running `gcloud access-context-manager policies list --organization YOUR_ORGANIZATION_ID --format="value(name)"`.

<!-- BEGINNING OF PRE-COMMIT-TERRAFORM DOCS HOOK -->
## Inputs

| Name | Description | Type | Default | Required |
|------|-------------|------|---------|:--------:|
| access\_context\_manager\_policy\_id | The id of the default Access Context Manager policy created in step `1-org`. Can be obtained by running `gcloud access-context-manager policies list --organization YOUR_ORGANIZATION_ID --format="value(name)"`. | `number` | n/a | yes |
| domain | The DNS name of peering managed zone, for instance 'example.com.'. Must end with a period. | `string` | n/a | yes |
| folder\_prefix | Name prefix to use for folders created. Should be the same in all steps. | `string` | `"fldr"` | no |
| org\_id | Organization ID | `string` | n/a | yes |
| parent\_folder | Optional - for an organization with existing projects or for development/validation. It will place all the example foundation resources under the provided folder instead of the root organization. The value is the numeric folder ID. The folder must already exist. Must be the same value used in previous step. | `string` | `""` | no |
| terraform\_service\_account | Service account email of the account to impersonate to run Terraform. | `string` | n/a | yes |

## Outputs

| Name | Description |
|------|-------------|
| base\_host\_project\_id | The base host project ID |
| base\_network\_name | The name of the VPC being created |
| base\_network\_self\_link | The URI of the VPC being created |
| base\_subnets\_ips | The IPs and CIDRs of the subnets being created |
| base\_subnets\_names | The names of the subnets being created |
| base\_subnets\_secondary\_ranges | The secondary ranges associated with these subnets |
| base\_subnets\_self\_links | The self-links of subnets being created |
| restricted\_access\_level\_name | Access context manager access level name |
| restricted\_host\_project\_id | The restricted host project ID |
| restricted\_network\_name | The name of the VPC being created |
| restricted\_network\_self\_link | The URI of the VPC being created |
| restricted\_service\_perimeter\_name | Access context manager service perimeter name |
| restricted\_subnets\_ips | The IPs and CIDRs of the subnets being created |
| restricted\_subnets\_names | The names of the subnets being created |
| restricted\_subnets\_secondary\_ranges | The secondary ranges associated with these subnets |
| restricted\_subnets\_self\_links | The self-links of subnets being created |

<!-- END OF PRE-COMMIT-TERRAFORM DOCS HOOK -->
File renamed without changes.
84 changes: 84 additions & 0 deletions 3-networks-dual-svpc/envs/development/main.tf
Original file line number Diff line number Diff line change
@@ -0,0 +1,84 @@
/**
* Copyright 2021 Google LLC
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/

locals {
env = "development"
environment_code = substr(local.env, 0, 1)
default_region1 = "us-west1"
default_region2 = "us-central1"
/*
* Base network ranges
*/
base_private_service_cidr = "10.16.64.0/21"
base_subnet_primary_ranges = {
(local.default_region1) = "10.0.64.0/21"
(local.default_region2) = "10.1.64.0/21"
}
base_subnet_secondary_ranges = {
(local.default_region1) = [
{
range_name = "rn-${local.environment_code}-shared-base-${local.default_region1}-gke-pod"
ip_cidr_range = "100.64.64.0/21"
},
{
range_name = "rn-${local.environment_code}-shared-base-${local.default_region1}-gke-svc"
ip_cidr_range = "100.64.72.0/21"
}
]
}
/*
* Restricted network ranges
*/
restricted_private_service_cidr = "10.24.64.0/21"
restricted_subnet_primary_ranges = {
(local.default_region1) = "10.8.64.0/21"
(local.default_region2) = "10.9.64.0/21"
}
restricted_subnet_secondary_ranges = {
(local.default_region1) = [
{
range_name = "rn-${local.environment_code}-shared-restricted-${local.default_region1}-gke-pod"
ip_cidr_range = "100.72.64.0/21"
},
{
range_name = "rn-${local.environment_code}-shared-restricted-${local.default_region1}-gke-svc"
ip_cidr_range = "100.72.72.0/21"
}
]
}
}

module "base_env" {
source = "../../modules/base_env"

env = local.env
environment_code = local.environment_code
org_id = var.org_id
access_context_manager_policy_id = var.access_context_manager_policy_id
terraform_service_account = var.terraform_service_account
default_region1 = local.default_region1
default_region2 = local.default_region2
domain = var.domain
parent_folder = var.parent_folder
enable_partner_interconnect = false
base_private_service_cidr = local.base_private_service_cidr
base_subnet_primary_ranges = local.base_subnet_primary_ranges
base_subnet_secondary_ranges = local.base_subnet_secondary_ranges
restricted_private_service_cidr = local.restricted_private_service_cidr
restricted_subnet_primary_ranges = local.restricted_subnet_primary_ranges
restricted_subnet_secondary_ranges = local.restricted_subnet_secondary_ranges

}
File renamed without changes.
File renamed without changes.
47 changes: 47 additions & 0 deletions 3-networks-dual-svpc/envs/development/variables.tf
Original file line number Diff line number Diff line change
@@ -0,0 +1,47 @@
/**
* Copyright 2021 Google LLC
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/

variable "org_id" {
type = string
description = "Organization ID"
}

variable "access_context_manager_policy_id" {
type = number
description = "The id of the default Access Context Manager policy created in step `1-org`. Can be obtained by running `gcloud access-context-manager policies list --organization YOUR_ORGANIZATION_ID --format=\"value(name)\"`."
}

variable "terraform_service_account" {
type = string
description = "Service account email of the account to impersonate to run Terraform."
}

variable "domain" {
type = string
description = "The DNS name of peering managed zone, for instance 'example.com.'. Must end with a period."
}

variable "parent_folder" {
description = "Optional - for an organization with existing projects or for development/validation. It will place all the example foundation resources under the provided folder instead of the root organization. The value is the numeric folder ID. The folder must already exist. Must be the same value used in previous step."
type = string
default = ""
}

variable "folder_prefix" {
description = "Name prefix to use for folders created. Should be the same in all steps."
type = string
default = "fldr"
}
File renamed without changes.
Loading

0 comments on commit 512430b

Please sign in to comment.