diff --git a/.github/workflows/terraform-ci-cd.yml b/.github/workflows/terraform-ci-cd.yml index 91bdf089..33de6ab7 100644 --- a/.github/workflows/terraform-ci-cd.yml +++ b/.github/workflows/terraform-ci-cd.yml @@ -34,14 +34,14 @@ jobs: GO111MODULE="on" go get github.com/terraform-docs/terraform-docs@v0.10.0-rc.1 export PATH=$HOME/go/bin:$PATH pip3 install --pre slalom.dataops - s-infra check_tf_metadata ./catalog \ + s-infra check_tf_metadata ./catalog/aws \ --check_module_headers \ --required_input_vars=[name_prefix,resource_tags,environment] \ --required_output_vars=[summary] \ --check_input_descriptions \ --check_output_descriptions \ --raise_error - s-infra check_tf_metadata ./components \ + s-infra check_tf_metadata ./components/aws \ --check_module_headers \ --required_input_vars=[name_prefix,resource_tags,environment] \ --required_output_vars=[] \ diff --git a/.gitignore b/.gitignore index 48e659c5..950ec075 100644 --- a/.gitignore +++ b/.gitignore @@ -26,9 +26,11 @@ plan.out.json !**/.secrets/*sample* !**/.secrets/*template* -# .tfstate files +# Terraform local state and lock files *.tfstate *.tfstate.* +*.tfstate-* +.terraform.lock.hcl # tfplan tfplan @@ -36,7 +38,7 @@ tfplan # .tfvars files *.tfvars -# .vs files +# .vs files **/.vs/** # Other (Python) @@ -44,3 +46,12 @@ __pycache__ .mypy_cache **.pyc .gitignore + +# Terraform executable + +terraform +terraform.exe + +# Other + +.DS_Store diff --git a/autodocs/README.md b/autodocs/README.md index 7d8eb021..a1ca6e1e 100644 --- a/autodocs/README.md +++ b/autodocs/README.md @@ -2,8 +2,16 @@ _These documents are published automatically to [infra.dataops.tk](https://infra.dataops.tk) using [Github Pages](https://help.github.com/en/github/working-with-github-pages) and [Jekyll](https://jekyllrb.com/docs/)._ +## Dependencies + +```bash +pip3 install uio logless runnow +brew install terraform-docs +``` + ## Rebuilding Docs + To rebuild or update all module README docs, run the following command on your dev machine: ```bash diff --git a/autodocs/requirements.txt b/autodocs/requirements.txt new file mode 100644 index 00000000..0a3b8960 --- /dev/null +++ b/autodocs/requirements.txt @@ -0,0 +1,3 @@ +logless +runnow +uio diff --git a/catalog/README.md b/catalog/README.md index a657f4a3..96dab669 100644 --- a/catalog/README.md +++ b/catalog/README.md @@ -50,7 +50,7 @@ Airflow is an open source platform to programmatically author, schedule and moni #### Overview -The `bastion-host` catalog module deploys an ECS-backed container which can be used to remotely test +The `bastion-host` module deploys an ECS-backed container which can be used to remotely test or develop using the native cloud environment. Applicable use cases include: @@ -58,7 +58,7 @@ Applicable use cases include: - Debugging network firewall and routing rules - Debugging components which can only be run from whitelisted IP ranges - Offloading heavy processing from the developer's local laptop -- Mitigating network relability issues when working from WiFi or home networks +- Mitigating network reliability issues when working from WiFi or home networks #### Documentation diff --git a/catalog/aws/bastion-host/README.md b/catalog/aws/bastion-host/README.md new file mode 100644 index 00000000..4026d79a --- /dev/null +++ b/catalog/aws/bastion-host/README.md @@ -0,0 +1,191 @@ +--- +parent: Infrastructure Catalog +title: AWS Bastion-Host +nav_exclude: false +--- +# AWS Bastion-Host + +[`source = "git::https://github.com/slalom-ggp/dataops-infra/tree/main/catalog/aws/bastion-host?ref=main"`](https://github.com/slalom-ggp/dataops-infra/tree/main/catalog/aws/bastion-host) + +## Overview + + +The `bastion-host` module deploys an ECS-backed container which can be used to remotely test +or develop using the native cloud environment. + +Applicable use cases include: + +- Debugging network firewall and routing rules +- Debugging components which can only be run from whitelisted IP ranges +- Offloading heavy processing from the developer's local laptop +- Mitigating network reliability issues when working from WiFi or home networks + +## Requirements + +No requirements. + +## Providers + +No provider. + +## Required Inputs + +The following input variables are required: + +### name\_prefix + +Description: Standard `name_prefix` module input. + +Type: `string` + +### environment + +Description: Standard `environment` module input. + +Type: + +```hcl +object({ + vpc_id = string + aws_region = string + public_subnets = list(string) + private_subnets = list(string) + }) +``` + +### resource\_tags + +Description: Standard `resource_tags` module input. + +Type: `map(string)` + +### aws\_credentials\_file + +Description: Path to the AWS credentials file, used to ensure that the correct credentials are used during upload of the ECR image. + +Type: `string` + +## Optional Inputs + +The following input variables are optional (have default values): + +### standard\_image + +Description: Optional. The name of an available prebuilt bastion image to use for the container. + +- Currently only `dataopstk/bastion:python-3.8` is supported (default). +- Ignored if `custom_base_image` is provided. + +Type: `string` + +Default: `"dataopstk/bastion:python-3.8"` + +### custom\_base\_image + +Description: Optional. The name of a custom base image, on top of which to build a custom bastion image. + +- Overrides any setting provided for `standard_image`. +- This option has additional workstation requirements, including Golang, Docker, and special docker + config as defined here: https://infra.dataops.tk/components/aws/ecr-image/#prereqs + +Type: `string` + +Default: `null` + +### settings + +Description: Map of environment variables. + +Type: `map(string)` + +Default: `{}` + +### secrets + +Description: Map of environment secrets. + +Type: `map(string)` + +Default: `{}` + +### container\_entrypoint + +Description: Optional. Override the docker image's entrypoint. + +Type: `any` + +Default: `null` + +### container\_num\_cores + +Description: Optional. Specify the number of cores to use in the container. + +Type: `number` + +Default: `0.5` + +### container\_ram\_gb + +Description: Optional. Specify the amount of RAM to be available to the container. + +Type: `number` + +Default: `1` + +### use\_private\_subnet + +Description: If True, tasks will use a private subnet and will require a NAT gateway to pull the docker +image, and for any outbound traffic. If False, tasks will use a public subnet and will +not require a NAT gateway. + +Type: `bool` + +Default: `false` + +### ssh\_public\_key\_filepath + +Description: Optional. Path to a valid public key for SSH connectivity. + +Type: `string` + +Default: `null` + +### ssh\_private\_key\_filepath + +Description: Optional. Path to a valid public key for SSH connectivity. + +Type: `string` + +Default: `null` + +### admin\_cidr + +Description: List of source IP CIDR blocks which should be allowed to connect to the bastion host. + +Type: `list(string)` + +Default: `null` + +## Outputs + +The following outputs are exported: + +### summary + +Description: Summary of resources created by this module. + +--------------------- + +## Source Files + +_Source code for this module is available using the links below._ + +* [main.tf](https://github.com/slalom-ggp/dataops-infra/tree/main//catalog/aws/bastion-host/main.tf) +* [outputs.tf](https://github.com/slalom-ggp/dataops-infra/tree/main//catalog/aws/bastion-host/outputs.tf) +* [variables.tf](https://github.com/slalom-ggp/dataops-infra/tree/main//catalog/aws/bastion-host/variables.tf) + +--------------------- + +_**NOTE:** This documentation was auto-generated using +`terraform-docs` and `s-infra` from `slalom.dataops`. +Please do not attempt to manually update this file._ diff --git a/catalog/aws/bastion-host/main.tf b/catalog/aws/bastion-host/main.tf index 5808a1f6..f76fd8d4 100644 --- a/catalog/aws/bastion-host/main.tf +++ b/catalog/aws/bastion-host/main.tf @@ -7,7 +7,7 @@ * - Debugging network firewall and routing rules * - Debugging components which can only be run from whitelisted IP ranges * - Offloading heavy processing from the developer's local laptop -* - Mitigating network relability issues when working from WiFi or home networks +* - Mitigating network reliability issues when working from WiFi or home networks * */ diff --git a/catalog/aws/data-lake-users/README.md b/catalog/aws/data-lake-users/README.md index 35061f59..877a3347 100644 --- a/catalog/aws/data-lake-users/README.md +++ b/catalog/aws/data-lake-users/README.md @@ -22,10 +22,10 @@ No requirements. The following providers are used by this module: -- aws - - local +- aws + ## Required Inputs The following input variables are required: diff --git a/catalog/aws/ml-ops/README.md b/catalog/aws/ml-ops/README.md index 02cd626d..c05faa89 100644 --- a/catalog/aws/ml-ops/README.md +++ b/catalog/aws/ml-ops/README.md @@ -24,10 +24,10 @@ No requirements. The following providers are used by this module: -- null - - local +- null + - random - aws @@ -659,6 +659,7 @@ _Source code for this module is available using the links below._ * [lambda.tf](https://github.com/slalom-ggp/dataops-infra/tree/main//catalog/aws/ml-ops/lambda.tf) * [main.tf](https://github.com/slalom-ggp/dataops-infra/tree/main//catalog/aws/ml-ops/main.tf) * [outputs.tf](https://github.com/slalom-ggp/dataops-infra/tree/main//catalog/aws/ml-ops/outputs.tf) +* [predictive-db.tf](https://github.com/slalom-ggp/dataops-infra/tree/main//catalog/aws/ml-ops/predictive-db.tf) * [s3.tf](https://github.com/slalom-ggp/dataops-infra/tree/main//catalog/aws/ml-ops/s3.tf) * [variables.tf](https://github.com/slalom-ggp/dataops-infra/tree/main//catalog/aws/ml-ops/variables.tf) diff --git a/catalog/aws/singer-taps/README.md b/catalog/aws/singer-taps/README.md index 392d7cef..aeb19d7a 100644 --- a/catalog/aws/singer-taps/README.md +++ b/catalog/aws/singer-taps/README.md @@ -132,6 +132,15 @@ Type: `any` Default: `null` +### data\_lake\_logging\_path + +Description: The remote folder for storing tap execution logs and log artifacts. +Currently only S3 paths (s3://...) are supported. + +Type: `string` + +Default: `null` + ### data\_lake\_storage\_path Description: The root path where files should be stored in the data lake. diff --git a/catalog/azure/data-lake/README.md b/catalog/azure/data-lake/README.md new file mode 100644 index 00000000..1279d8aa --- /dev/null +++ b/catalog/azure/data-lake/README.md @@ -0,0 +1,129 @@ +--- +parent: Infrastructure Catalog +title: Azure Data-Lake +nav_exclude: false +--- +# Azure Data-Lake + +[`source = "git::https://github.com/slalom-ggp/dataops-infra/tree/main/catalog/azure/data-lake?ref=main"`](https://github.com/slalom-ggp/dataops-infra/tree/main/catalog/azure/data-lake) + +## Overview + + +Deploys either a Gen1 or Gen2 Data Lake Store. + +## Requirements + +No requirements. + +## Providers + +No provider. + +## Required Inputs + +The following input variables are required: + +### name\_prefix + +Description: Standard `name_prefix` module input. + +Type: `string` + +### resource\_tags + +Description: Standard `resource_tags` module input. + +Type: `map(string)` + +### data\_lake\_name + +Description: The name of the Data Lake that will be created. + +Type: `string` + +### data\_lake\_type + +Description: The type of Data Lake to be created. Possible values are Gen1 or Gen2. + +Type: `string` + +## Optional Inputs + +The following input variables are optional (have default values): + +### azure\_location + +Description: The location the Data Lake will exist. + +Type: `string` + +Default: `""` + +### resource\_group\_name + +Description: The name of the Resource Group which the Data Lake will be created within. + +Type: `string` + +Default: `""` + +### storage\_account\_id + +Description: The ID for the Storage Account in which this Data Lake will be created. + +Type: `string` + +Default: `""` + +### encryption\_state + +Description: Defines whether exncryption is enabled on this Data Lake account. Possible values are Enabled or Disabled. + +Type: `string` + +Default: `"Enabled"` + +### firewall\_allow\_azure\_ips + +Description: Defines whether Azure Service IPs are allowed through the firewall. Possible values are Enabled and Disabled. + +Type: `string` + +Default: `"Enabled"` + +### firewall\_state + +Description: The state of the firewall for the Data Lake. Possible values are Enabled and Disabled. + +Type: `string` + +Default: `"Enabled"` + +## Outputs + +The following outputs are exported: + +### summary + +Description: Summary of resources created by this module. + +### data\_lake\_name + +Description: The Data Lake name value(s) of the newly created Data Lake(s). + +--------------------- + +## Source Files + +_Source code for this module is available using the links below._ + +* [main.tf](https://github.com/slalom-ggp/dataops-infra/tree/main//catalog/azure/data-lake/main.tf) +* [outputs.tf](https://github.com/slalom-ggp/dataops-infra/tree/main//catalog/azure/data-lake/outputs.tf) +* [variables.tf](https://github.com/slalom-ggp/dataops-infra/tree/main//catalog/azure/data-lake/variables.tf) + +--------------------- + +_**NOTE:** This documentation was auto-generated using +`terraform-docs` and `s-infra` from `slalom.dataops`. +Please do not attempt to manually update this file._ diff --git a/catalog/azure/resource-group/README.md b/catalog/azure/resource-group/README.md new file mode 100644 index 00000000..146e6652 --- /dev/null +++ b/catalog/azure/resource-group/README.md @@ -0,0 +1,81 @@ +--- +parent: Infrastructure Catalog +title: Azure Resource-Group +nav_exclude: false +--- +# Azure Resource-Group + +[`source = "git::https://github.com/slalom-ggp/dataops-infra/tree/main/catalog/azure/resource-group?ref=main"`](https://github.com/slalom-ggp/dataops-infra/tree/main/catalog/azure/resource-group) + +## Overview + + +Deploys a Resource Group within a subscription. + +## Requirements + +No requirements. + +## Providers + +No provider. + +## Required Inputs + +The following input variables are required: + +### name\_prefix + +Description: Standard `name_prefix` module input. + +Type: `string` + +### resource\_tags + +Description: Standard `resource_tags` module input. + +Type: `map(string)` + +### azure\_location + +Description: The location the Storage Account will exist. + +Type: `string` + +### resource\_group\_name + +Description: The name of the Resource Group to be created. + +Type: `string` + +## Optional Inputs + +No optional input. + +## Outputs + +The following outputs are exported: + +### summary + +Description: Summary of resources created by this module. + +### resource\_group\_name + +Description: The `resource_group_name` value to be passed to other Azure Infrastructure Catalog modules. + +--------------------- + +## Source Files + +_Source code for this module is available using the links below._ + +* [main.tf](https://github.com/slalom-ggp/dataops-infra/tree/main//catalog/azure/resource-group/main.tf) +* [outputs.tf](https://github.com/slalom-ggp/dataops-infra/tree/main//catalog/azure/resource-group/outputs.tf) +* [variables.tf](https://github.com/slalom-ggp/dataops-infra/tree/main//catalog/azure/resource-group/variables.tf) + +--------------------- + +_**NOTE:** This documentation was auto-generated using +`terraform-docs` and `s-infra` from `slalom.dataops`. +Please do not attempt to manually update this file._ diff --git a/catalog/azure/storage-account/README.md b/catalog/azure/storage-account/README.md new file mode 100644 index 00000000..78996c87 --- /dev/null +++ b/catalog/azure/storage-account/README.md @@ -0,0 +1,139 @@ +--- +parent: Infrastructure Catalog +title: Azure Storage-Account +nav_exclude: false +--- +# Azure Storage-Account + +[`source = "git::https://github.com/slalom-ggp/dataops-infra/tree/main/catalog/azure/storage-account?ref=main"`](https://github.com/slalom-ggp/dataops-infra/tree/main/catalog/azure/storage-account) + +## Overview + + +Deploys a Storage Account within a subscription. + +## Requirements + +No requirements. + +## Providers + +No provider. + +## Required Inputs + +The following input variables are required: + +### name\_prefix + +Description: Standard `name_prefix` module input. + +Type: `string` + +### resource\_tags + +Description: Standard `resource_tags` module input. + +Type: `map(string)` + +### azure\_location + +Description: The location the Storage Account will exist. + +Type: `string` + +### resource\_group\_name + +Description: The name of the Resource Group which the Storage Account will be created within. + +Type: `string` + +### storage\_account\_name + +Description: The name of the Storage Account to be created. + +Type: `string` + +## Optional Inputs + +The following input variables are optional (have default values): + +### account\_tier + +Description: Defines the tier to use for this Storage Account. + +Type: `string` + +Default: `"Standard"` + +### account\_replication\_type + +Description: The type of replication to use for this Storage Account. Valid options are LRS, GRS, RAGRS, ZRS, GZRS and RAGZRS. + +Type: `string` + +Default: `"GZRS"` + +### account\_kind + +Description: The kind of Storage Account. Valid options are BlobStorage, BlockBlobStorage, FileStorage, Storage and StorageV2. + +Type: `string` + +Default: `"StorageV2"` + +### access\_tier + +Description: The access tier for BlobStorage, FileStorage and StorageV2 accounts. Valid options are Hot and Cool. + +Type: `string` + +Default: `"Hot"` + +### enable\_https\_traffic\_only + +Description: Boolean flag which forces HTTPS if enabled. + +Type: `bool` + +Default: `true` + +### allow\_blob\_public\_access + +Description: Allow or disallow public access to all Blobs or Containers in the Storage Account. + +Type: `bool` + +Default: `false` + +## Outputs + +The following outputs are exported: + +### summary + +Description: Summary of resources created by this module. + +### storage\_account\_name + +Description: The `storage_account_name` value of the newly created storage account. + +### storage\_account\_id + +Description: The ID for the new Storage Account created. + +--------------------- + +## Source Files + +_Source code for this module is available using the links below._ + +* [main.tf](https://github.com/slalom-ggp/dataops-infra/tree/main//catalog/azure/storage-account/main.tf) +* [outputs.tf](https://github.com/slalom-ggp/dataops-infra/tree/main//catalog/azure/storage-account/outputs.tf) +* [variables.tf](https://github.com/slalom-ggp/dataops-infra/tree/main//catalog/azure/storage-account/variables.tf) + +--------------------- + +_**NOTE:** This documentation was auto-generated using +`terraform-docs` and `s-infra` from `slalom.dataops`. +Please do not attempt to manually update this file._ diff --git a/catalog/azure/storage/README.md b/catalog/azure/storage/README.md new file mode 100644 index 00000000..11896df6 --- /dev/null +++ b/catalog/azure/storage/README.md @@ -0,0 +1,116 @@ +--- +parent: Infrastructure Catalog +title: Azure Storage +nav_exclude: false +--- +# Azure Storage + +[`source = "git::https://github.com/slalom-ggp/dataops-infra/tree/main/catalog/azure/storage?ref=main"`](https://github.com/slalom-ggp/dataops-infra/tree/main/catalog/azure/storage) + +## Overview + + +Deploys Storage Containers, Queue Storage, and Table Storage within a storage +account. + +## Requirements + +No requirements. + +## Providers + +No provider. + +## Required Inputs + +The following input variables are required: + +### name\_prefix + +Description: Standard `name_prefix` module input. + +Type: `string` + +### resource\_tags + +Description: Standard `resource_tags` module input. + +Type: `map(string)` + +### storage\_account\_name + +Description: The name of the Storage Account to be created. + +Type: `string` + +## Optional Inputs + +The following input variables are optional (have default values): + +### container\_names + +Description: Names of Storage Containers to be created. + +Type: `list(string)` + +Default: `[]` + +### table\_storage\_names + +Description: Names of Tables to be created. + +Type: `list(string)` + +Default: `[]` + +### queue\_storage\_names + +Description: Names of Queues to be created. + +Type: `list(string)` + +Default: `[]` + +### container\_access\_type + +Description: The access level configured for the Storage Container(s). Possible values are blob, container or private. + +Type: `string` + +Default: `"private"` + +## Outputs + +The following outputs are exported: + +### summary + +Description: Summary of resources created by this module. + +### storage\_container\_names + +Description: The Storage Container name value(s) of the newly created container(s). + +### table\_storage\_names + +Description: The Table Storage name value(s) of the newly created table(s). + +### queue\_storage\_names + +Description: The Queue Storage name value(s) of the newly created table(s). + +--------------------- + +## Source Files + +_Source code for this module is available using the links below._ + +* [main.tf](https://github.com/slalom-ggp/dataops-infra/tree/main//catalog/azure/storage/main.tf) +* [outputs.tf](https://github.com/slalom-ggp/dataops-infra/tree/main//catalog/azure/storage/outputs.tf) +* [variables.tf](https://github.com/slalom-ggp/dataops-infra/tree/main//catalog/azure/storage/variables.tf) + +--------------------- + +_**NOTE:** This documentation was auto-generated using +`terraform-docs` and `s-infra` from `slalom.dataops`. +Please do not attempt to manually update this file._ diff --git a/components/aws/ecr-image/README.md b/components/aws/ecr-image/README.md index 7549e8e4..6902f746 100644 --- a/components/aws/ecr-image/README.md +++ b/components/aws/ecr-image/README.md @@ -134,7 +134,11 @@ go get -u github.com/awslabs/amazon-ecr-credential-helper/ecr-login/cli/docker-c **To install on Mac/Linux:** -* [https://github.com/awslabs/amazon-ecr-credential-helper#user-content-installing](https://github.com/awslabs/amazon-ecr-credential-helper#user-content-installing) +```bash +brew install docker-credential-helper-ecr +``` + +* More info here: [https://github.com/awslabs/amazon-ecr-credential-helper#user-content-installing](https://github.com/awslabs/amazon-ecr-credential-helper#user-content-installing) **Configure Docker to use the ECR Credential Helper:** @@ -164,12 +168,14 @@ Install with: ```ps Install-Module -name AWSPowerShell.NetCore +Install-Module -name AWS.Tools.ECR ``` In order to then use this module for manual executions, you will need to load it into your powershell session: ```ps Import-Module AWSPowerShell.NetCore +Import-Module AWS.Tools.ECR ``` For more info: [https://docs.aws.amazon.com/powershell/latest/userguide/pstools-getting-set-up-windows.html#ps-installing-awspowershellnetcore](https://docs.aws.amazon.com/powershell/latest/userguide/pstools-getting-set-up-windows.html#ps-installing-awspowershellnetcore) diff --git a/components/aws/ecs-cluster/README.md b/components/aws/ecs-cluster/README.md index 59714deb..c8b616dd 100644 --- a/components/aws/ecs-cluster/README.md +++ b/components/aws/ecs-cluster/README.md @@ -24,10 +24,10 @@ No requirements. The following providers are used by this module: -- random - - aws +- random + ## Required Inputs The following input variables are required: diff --git a/components/aws/ecs-task/README.md b/components/aws/ecs-task/README.md index fc231df4..81846d3b 100644 --- a/components/aws/ecs-task/README.md +++ b/components/aws/ecs-task/README.md @@ -24,10 +24,10 @@ No requirements. The following providers are used by this module: -- aws - - http +- aws + - random - null @@ -162,7 +162,7 @@ Default: `"8"` ### environment\_secrets Description: Mapping of environment variable names to secret manager ARNs or local file secrets. Examples: - - arn:aws:secretsmanager:[aws\_region]:[aws\_account]:secret:prod/ECSRunner/AWS\_SECRET\_ACCESS\_KEY + - arn:aws:secretsmanager:[aws\_region]:[aws\_account]:secret:prod/ECSRunner/thisisnottherealsecret - path/to/file.json:MY\_KEY\_NAME\_1 - path/to/file.yml:MY\_KEY\_NAME\_2 diff --git a/components/aws/ecs-task/variables.tf b/components/aws/ecs-task/variables.tf index d60f2d25..2913b0e7 100644 --- a/components/aws/ecs-task/variables.tf +++ b/components/aws/ecs-task/variables.tf @@ -82,7 +82,7 @@ variable "environment_secrets" { default = {} description = < - path/to/file.json:MY_KEY_NAME_1 - path/to/file.yml:MY_KEY_NAME_2 EOF diff --git a/components/aws/glue-job/README.md b/components/aws/glue-job/README.md index c8b9102b..dfe7bfec 100644 --- a/components/aws/glue-job/README.md +++ b/components/aws/glue-job/README.md @@ -134,7 +134,7 @@ For additional information, see: Type: `map(string)` -Default: `null` +Default: `{}` ## Outputs diff --git a/components/aws/lambda-python/README.md b/components/aws/lambda-python/README.md index 7674688f..d269b019 100644 --- a/components/aws/lambda-python/README.md +++ b/components/aws/lambda-python/README.md @@ -23,10 +23,10 @@ No requirements. The following providers are used by this module: -- random - - aws +- random + - null - archive diff --git a/components/azure/data-lake/gen-1/README.md b/components/azure/data-lake/gen-1/README.md new file mode 100644 index 00000000..58ed2865 --- /dev/null +++ b/components/azure/data-lake/gen-1/README.md @@ -0,0 +1,119 @@ +--- +parent: Infrastructure Components +title: Data-Lake Gen-1 +nav_exclude: false +--- +# Data-Lake Gen-1 + +[`source = "git::https://github.com/slalom-ggp/dataops-infra/tree/main/components/azure/data-lake/gen-1?ref=main"`](https://github.com/slalom-ggp/dataops-infra/tree/main/components/azure/data-lake/gen-1) + +## Overview + + +This data lake implementation creates a Gen1 Data Lake Store if configured. + +## Requirements + +No requirements. + +## Providers + +The following providers are used by this module: + +- azurerm + +## Required Inputs + +The following input variables are required: + +### name\_prefix + +Description: Standard `name_prefix` module input. + +Type: `string` + +### resource\_tags + +Description: Standard `resource_tags` module input. + +Type: `map(string)` + +### data\_lake\_name + +Description: The name of the Data Lake that will be created. + +Type: `string` + +### data\_lake\_type + +Description: The type of Data Lake to be created. Possible values are Gen1 or Gen2. + +Type: `string` + +### azure\_location + +Description: The location the Data Lake will exist. + +Type: `string` + +### resource\_group\_name + +Description: The name of the Resource Group which the Data Lake will be created within. + +Type: `string` + +## Optional Inputs + +The following input variables are optional (have default values): + +### encryption\_state + +Description: Defines whether exncryption is enabled on this Data Lake account. Possible values are Enabled or Disabled. + +Type: `string` + +Default: `"Enabled"` + +### firewall\_allow\_azure\_ips + +Description: Defines whether Azure Service IPs are allowed through the firewall. Possible values are Enabled and Disabled. + +Type: `string` + +Default: `"Enabled"` + +### firewall\_state + +Description: The state of the firewall for the Data Lake. Possible values are Enabled and Disabled. + +Type: `string` + +Default: `"Enabled"` + +## Outputs + +The following outputs are exported: + +### data\_lake\_name + +Description: The name of the Data Lake created. + +### data\_lake\_type + +Description: The generation type of the Data Lake created (i.e. Gen1 or Gen2). + +--------------------- + +## Source Files + +_Source code for this module is available using the links below._ + +* [main.tf](https://github.com/slalom-ggp/dataops-infra/tree/main//components/azure/data-lake/gen-1/main.tf) +* [outputs.tf](https://github.com/slalom-ggp/dataops-infra/tree/main//components/azure/data-lake/gen-1/outputs.tf) +* [variables.tf](https://github.com/slalom-ggp/dataops-infra/tree/main//components/azure/data-lake/gen-1/variables.tf) + +--------------------- + +_**NOTE:** This documentation was auto-generated using +`terraform-docs` and `s-infra` from `slalom.dataops`. +Please do not attempt to manually update this file._ diff --git a/components/azure/data-lake/gen-2/README.md b/components/azure/data-lake/gen-2/README.md new file mode 100644 index 00000000..79b37600 --- /dev/null +++ b/components/azure/data-lake/gen-2/README.md @@ -0,0 +1,89 @@ +--- +parent: Infrastructure Components +title: Data-Lake Gen-2 +nav_exclude: false +--- +# Data-Lake Gen-2 + +[`source = "git::https://github.com/slalom-ggp/dataops-infra/tree/main/components/azure/data-lake/gen-2?ref=main"`](https://github.com/slalom-ggp/dataops-infra/tree/main/components/azure/data-lake/gen-2) + +## Overview + + +This data lake implementation creates a Gen2 Data Lake Store if configured. + +## Requirements + +No requirements. + +## Providers + +The following providers are used by this module: + +- azurerm + +## Required Inputs + +The following input variables are required: + +### name\_prefix + +Description: Standard `name_prefix` module input. + +Type: `string` + +### resource\_tags + +Description: Standard `resource_tags` module input. + +Type: `map(string)` + +### data\_lake\_name + +Description: The name of the Data Lake that will be created. + +Type: `string` + +### data\_lake\_type + +Description: The type of Data Lake to be created. Possible values are Gen1 or Gen2. + +Type: `string` + +### storage\_account\_id + +Description: The ID for the Storage Account in which this Data Lake will be created. + +Type: `string` + +## Optional Inputs + +No optional input. + +## Outputs + +The following outputs are exported: + +### data\_lake\_name + +Description: The name of the Data Lake created. + +### data\_lake\_type + +Description: The generation type of the Data Lake created (i.e. Gen1 or Gen2). + +--------------------- + +## Source Files + +_Source code for this module is available using the links below._ + +* [main.tf](https://github.com/slalom-ggp/dataops-infra/tree/main//components/azure/data-lake/gen-2/main.tf) +* [outputs.tf](https://github.com/slalom-ggp/dataops-infra/tree/main//components/azure/data-lake/gen-2/outputs.tf) +* [variables.tf](https://github.com/slalom-ggp/dataops-infra/tree/main//components/azure/data-lake/gen-2/variables.tf) + +--------------------- + +_**NOTE:** This documentation was auto-generated using +`terraform-docs` and `s-infra` from `slalom.dataops`. +Please do not attempt to manually update this file._ diff --git a/components/azure/resource-group/README.md b/components/azure/resource-group/README.md new file mode 100644 index 00000000..d7c5d1a4 --- /dev/null +++ b/components/azure/resource-group/README.md @@ -0,0 +1,79 @@ +--- +parent: Infrastructure Components +title: Azure Resource-Group +nav_exclude: false +--- +# Azure Resource-Group + +[`source = "git::https://github.com/slalom-ggp/dataops-infra/tree/main/components/azure/resource-group?ref=main"`](https://github.com/slalom-ggp/dataops-infra/tree/main/components/azure/resource-group) + +## Overview + + +This is the underlying technical component which supports the Resource Group catalog module. + +## Requirements + +No requirements. + +## Providers + +The following providers are used by this module: + +- azurerm + +## Required Inputs + +The following input variables are required: + +### name\_prefix + +Description: Standard `name_prefix` module input. + +Type: `string` + +### resource\_tags + +Description: Standard `resource_tags` module input. + +Type: `map(string)` + +### azure\_location + +Description: The location the Storage Account will exist. + +Type: `string` + +### resource\_group\_name + +Description: The name of the Resource Group to be created. + +Type: `string` + +## Optional Inputs + +No optional input. + +## Outputs + +The following outputs are exported: + +### resource\_group\_name + +Description: The name of the new Resource Group created. + +--------------------- + +## Source Files + +_Source code for this module is available using the links below._ + +* [main.tf](https://github.com/slalom-ggp/dataops-infra/tree/main//components/azure/resource-group/main.tf) +* [outputs.tf](https://github.com/slalom-ggp/dataops-infra/tree/main//components/azure/resource-group/outputs.tf) +* [variables.tf](https://github.com/slalom-ggp/dataops-infra/tree/main//components/azure/resource-group/variables.tf) + +--------------------- + +_**NOTE:** This documentation was auto-generated using +`terraform-docs` and `s-infra` from `slalom.dataops`. +Please do not attempt to manually update this file._ diff --git a/components/azure/storage-account/README.md b/components/azure/storage-account/README.md new file mode 100644 index 00000000..37e66331 --- /dev/null +++ b/components/azure/storage-account/README.md @@ -0,0 +1,145 @@ +--- +parent: Infrastructure Components +title: Azure Storage-Account +nav_exclude: false +--- +# Azure Storage-Account + +[`source = "git::https://github.com/slalom-ggp/dataops-infra/tree/main/components/azure/storage-account?ref=main"`](https://github.com/slalom-ggp/dataops-infra/tree/main/components/azure/storage-account) + +## Overview + + +This is the underlying technical component which supports the Storage Account catalog module. + +## Requirements + +No requirements. + +## Providers + +The following providers are used by this module: + +- azurerm + +## Required Inputs + +The following input variables are required: + +### name\_prefix + +Description: Standard `name_prefix` module input. + +Type: `string` + +### resource\_tags + +Description: Standard `resource_tags` module input. + +Type: `map(string)` + +### azure\_location + +Description: The location the Storage Account will exist. + +Type: `string` + +### resource\_group\_name + +Description: The name of the Resource Group which the Storage Account will be created within. + +Type: `string` + +### storage\_account\_name + +Description: The name of the Storage Account to be created. + +Type: `string` + +## Optional Inputs + +The following input variables are optional (have default values): + +### account\_tier + +Description: Defines the tier to use for this Storage Account. + +Type: `string` + +Default: `"Standard"` + +### account\_replication\_type + +Description: The type of replication to use for this Storage Account. Valid options are LRS, GRS, RAGRS, ZRS, GZRS and RAGZRS. + +Type: `string` + +Default: `"GZRS"` + +### account\_kind + +Description: The kind of Storage Account. Valid options are BlobStorage, BlockBlobStorage, FileStorage, Storage and StorageV2. + +Type: `string` + +Default: `"StorageV2"` + +### access\_tier + +Description: The access tier for BlobStorage, FileStorage and StorageV2 accounts. Valid options are Hot and Cool. + +Type: `string` + +Default: `"Hot"` + +### enable\_https\_traffic\_only + +Description: Boolean flag which forces HTTPS if enabled. + +Type: `bool` + +Default: `true` + +### allow\_blob\_public\_access + +Description: Allow or disallow public access to all Blobs or Containers in the Storage Account. + +Type: `bool` + +Default: `false` + +### is\_hns\_enabled + +Description: Defines whether Hierarchical Namespace is enabled. Typically used with Azure Data Lake Storage Gen 2. + +Type: `bool` + +Default: `true` + +## Outputs + +The following outputs are exported: + +### storage\_account\_name + +Description: The name of the new Storage Account created. + +### storage\_account\_id + +Description: The ID for the new Storage Account created. + +--------------------- + +## Source Files + +_Source code for this module is available using the links below._ + +* [main.tf](https://github.com/slalom-ggp/dataops-infra/tree/main//components/azure/storage-account/main.tf) +* [outputs.tf](https://github.com/slalom-ggp/dataops-infra/tree/main//components/azure/storage-account/outputs.tf) +* [variables.tf](https://github.com/slalom-ggp/dataops-infra/tree/main//components/azure/storage-account/variables.tf) + +--------------------- + +_**NOTE:** This documentation was auto-generated using +`terraform-docs` and `s-infra` from `slalom.dataops`. +Please do not attempt to manually update this file._ diff --git a/components/azure/storage/container/README.md b/components/azure/storage/container/README.md new file mode 100644 index 00000000..7e63c9b0 --- /dev/null +++ b/components/azure/storage/container/README.md @@ -0,0 +1,90 @@ +--- +parent: Infrastructure Components +title: Storage Container +nav_exclude: false +--- +# Storage Container + +[`source = "git::https://github.com/slalom-ggp/dataops-infra/tree/main/components/azure/storage/container?ref=main"`](https://github.com/slalom-ggp/dataops-infra/tree/main/components/azure/storage/container) + +## Overview + + +This is the underlying technical component which supports the Storage catalog module for +building Containers. + +## Requirements + +No requirements. + +## Providers + +The following providers are used by this module: + +- azurerm + +## Required Inputs + +The following input variables are required: + +### name\_prefix + +Description: Standard `name_prefix` module input. + +Type: `string` + +### resource\_tags + +Description: Standard `resource_tags` module input. + +Type: `map(string)` + +### storage\_account\_name + +Description: The name of the Storage Account the Storage Container(s) will be created under. + +Type: `string` + +## Optional Inputs + +The following input variables are optional (have default values): + +### container\_names + +Description: Names of Storage Containers to be created. + +Type: `list(string)` + +Default: `[]` + +### container\_access\_type + +Description: The access level configured for the Storage Container(s). Possible values are blob, container or private. + +Type: `string` + +Default: `"private"` + +## Outputs + +The following outputs are exported: + +### container\_names + +Description: The name of the Storage Container(s) created. + +--------------------- + +## Source Files + +_Source code for this module is available using the links below._ + +* [main.tf](https://github.com/slalom-ggp/dataops-infra/tree/main//components/azure/storage/container/main.tf) +* [outputs.tf](https://github.com/slalom-ggp/dataops-infra/tree/main//components/azure/storage/container/outputs.tf) +* [variables.tf](https://github.com/slalom-ggp/dataops-infra/tree/main//components/azure/storage/container/variables.tf) + +--------------------- + +_**NOTE:** This documentation was auto-generated using +`terraform-docs` and `s-infra` from `slalom.dataops`. +Please do not attempt to manually update this file._ diff --git a/components/azure/storage/queue/README.md b/components/azure/storage/queue/README.md new file mode 100644 index 00000000..bb5ead36 --- /dev/null +++ b/components/azure/storage/queue/README.md @@ -0,0 +1,80 @@ +--- +parent: Infrastructure Components +title: Storage Queue +nav_exclude: false +--- +# Storage Queue + +[`source = "git::https://github.com/slalom-ggp/dataops-infra/tree/main/components/azure/storage/queue?ref=main"`](https://github.com/slalom-ggp/dataops-infra/tree/main/components/azure/storage/queue) + +## Overview + + +This is the underlying technical component which supports the Storage catalog module for +building Queue Storage. + +## Requirements + +No requirements. + +## Providers + +The following providers are used by this module: + +- azurerm + +## Required Inputs + +The following input variables are required: + +### name\_prefix + +Description: Standard `name_prefix` module input. + +Type: `string` + +### resource\_tags + +Description: Standard `resource_tags` module input. + +Type: `map(string)` + +### storage\_account\_name + +Description: The name of the Storage Account the Queue(s) will be created under. + +Type: `string` + +### queue\_storage\_names + +Description: Names of Queues to be created. + +Type: `list(string)` + +## Optional Inputs + +No optional input. + +## Outputs + +The following outputs are exported: + +### queue\_storage\_names + +Description: The name of the Queue(s) created. + +--------------------- + +## Source Files + +_Source code for this module is available using the links below._ + +* [main.tf](https://github.com/slalom-ggp/dataops-infra/tree/main//components/azure/storage/queue/main.tf) +* [outputs.tf](https://github.com/slalom-ggp/dataops-infra/tree/main//components/azure/storage/queue/outputs.tf) +* [variables.tf](https://github.com/slalom-ggp/dataops-infra/tree/main//components/azure/storage/queue/variables.tf) + +--------------------- + +_**NOTE:** This documentation was auto-generated using +`terraform-docs` and `s-infra` from `slalom.dataops`. +Please do not attempt to manually update this file._ diff --git a/components/azure/storage/table/README.md b/components/azure/storage/table/README.md new file mode 100644 index 00000000..62e94e58 --- /dev/null +++ b/components/azure/storage/table/README.md @@ -0,0 +1,80 @@ +--- +parent: Infrastructure Components +title: Storage Table +nav_exclude: false +--- +# Storage Table + +[`source = "git::https://github.com/slalom-ggp/dataops-infra/tree/main/components/azure/storage/table?ref=main"`](https://github.com/slalom-ggp/dataops-infra/tree/main/components/azure/storage/table) + +## Overview + + +This is the underlying technical component which supports the Storage catalog module for +building Table Storage. + +## Requirements + +No requirements. + +## Providers + +The following providers are used by this module: + +- azurerm + +## Required Inputs + +The following input variables are required: + +### name\_prefix + +Description: Standard `name_prefix` module input. + +Type: `string` + +### resource\_tags + +Description: Standard `resource_tags` module input. + +Type: `map(string)` + +### storage\_account\_name + +Description: The name of the Storage Account the Table(s) will be created under. + +Type: `string` + +### table\_storage\_names + +Description: Names of Tables to be created. + +Type: `list(string)` + +## Optional Inputs + +No optional input. + +## Outputs + +The following outputs are exported: + +### table\_storage\_names + +Description: The name of the Table(s) created. + +--------------------- + +## Source Files + +_Source code for this module is available using the links below._ + +* [main.tf](https://github.com/slalom-ggp/dataops-infra/tree/main//components/azure/storage/table/main.tf) +* [outputs.tf](https://github.com/slalom-ggp/dataops-infra/tree/main//components/azure/storage/table/outputs.tf) +* [variables.tf](https://github.com/slalom-ggp/dataops-infra/tree/main//components/azure/storage/table/variables.tf) + +--------------------- + +_**NOTE:** This documentation was auto-generated using +`terraform-docs` and `s-infra` from `slalom.dataops`. +Please do not attempt to manually update this file._ diff --git a/samples/ml-ops-on-aws/.secrets/README.md b/samples/ml-ops-on-aws/.secrets/README.md deleted file mode 100644 index f867a353..00000000 --- a/samples/ml-ops-on-aws/.secrets/README.md +++ /dev/null @@ -1,21 +0,0 @@ - -# Ml-Ops-On-AWS .Secrets - -`/samples/ml-ops-on-aws/secrets` - -## Overview - - -## Inputs - -No input. - -## Outputs - -No output. - ---------------------- - -_**NOTE:** This documentation was auto-generated using -`terraform-docs` and `s-infra` from `slalom.dataops`. -Please do not attempt to manually update this file._