Creds management (#53)

* bump minio ~> 0.3.0 * bump postgres~> 0.3.0 * mv user provided creds example * refactor ansible playbooks * add vault-provided-creds example Co-authored-by: Claes Gill <claes@claesgill.com> * Update README.md * update variable's names * use canaries & use host volumes in all examples * add resources proxy to example * update contributors list
Skatteetaten · Nov 11, 2020 · 989d774 · 989d774
1 parent 139ae79
commit 989d774
Show file tree

Hide file tree

Showing 24 changed files with 554 additions and 153 deletions.
diff --git a/CHANGELOG.md b/CHANGELOG.md
@@ -7,6 +7,10 @@
 - Add intentions section in readme #42
 - Update box ~> 0.7.x #45
 - Resource allocation variables for proxy #54
+- Improve credentials management (vault provided credentials) #44
+- Bump example: minio ~> 0.3.0 #51
+- Bump example: postgres ~> 0.3.0 #51
+- Output variable for port #49
 
 ### Changed
 

diff --git a/Makefile b/Makefile
@@ -52,10 +52,10 @@ destroy-box:
 remove-tmp:
 	rm -rf ./tmp
 	rm -rf ./.vagrant
+	rm -rf ./dev/tmp
 	rm -rf ./.minio.sys
 	rm -rf ./example/**/.terraform
-	rm -rf ./example/**/terraform.tfstate
-	rm -rf ./example/**/terraform.tfstate.backup
+	rm -rf ./example/**/.terraform*
 
 clean: destroy-box remove-tmp
 

diff --git a/README.md b/README.md
@@ -7,25 +7,39 @@ This module is IaC - infrastructure as code which contains a nomad job of [hive]
 
 ## Content
 0. [Prerequisites](#prerequisites)
-1. [Compatibility](#compatibility)
-2. [Requirements](#requirements)
-    1. [Required software](#required-software)
-3. [Usage](#usage)
-   1. [Providers](#providers)
-   2. [Intentions](#intentions)
-4. [Inputs](#inputs)
-5. [Outputs](#outputs)
-6. [Modes](#modes)
-7. [Example](#example)
+1. [Requirements](#requirements)
+    1. [Required modules](#required-modules)
+    2. [Required software](#required-software)
+3. [Compatibility](#compatibility)
+4. [Providers](#providers)
+5. [Usage](#usage)
     1. [Verifying setup](#verifying-setup)
         1. [Data example upload](#data-example-upload)
-8. [Authors](#authors)
-9. [License](#license)
-10. [References](#references)
+6. [Intentions](#intentions)
+7. [Inputs](#inputs)
+8. [Outputs](#outputs)
+8. [Modes](#modes)
+9. [Examples](#examples)
+10. [Contributors](#contributors)
+11. [License](#license)
+12. [References](#references)
 
 ## Prerequisites
 Please follow [this section in original template](https://github.com/fredrikhgrelland/vagrant-hashistack-template#install-prerequisites)
 
+## Requirements
+
+### Required modules
+|Module|Version|
+|:---|:---|
+|[terraform-nomad-minio](https://github.com/fredrikhgrelland/terraform-nomad-minio)| 0.3.0 or newer|
+|[terraform-nomad-postgres](https://github.com/fredrikhgrelland/terraform-nomad-postgres)| 0.3.0 or newer|
+
+### Required software
+- [GNU make](https://man7.org/linux/man-pages/man1/make.1.html)
+- [Docker](https://www.docker.com/)
+- [Consul](https://releases.hashicorp.com/consul/)
+
 ## Compatibility
 |Software|OSS Version|Enterprise Version|
 |:---|:---|:---|
@@ -34,23 +48,36 @@ Please follow [this section in original template](https://github.com/fredrikhgre
 |Vault|1.5.2.1 or newer|1.5.2.1 or newer|
 |Nomad|0.12.3 or newer|0.12.3 or newer|
 
-## Requirements
-
-### Required software
-All software is provided and run with docker.
-See the [Makefile](Makefile) for inspiration.
+## Providers
+- [Nomad](https://registry.terraform.io/providers/hashicorp/nomad/latest/docs)
+- [Vault](https://registry.terraform.io/providers/hashicorp/vault/latest/docs)
 
 ## Usage
-The following command will run hive in the [example/standalone](example/standalone) folder.
+The following command will run hive in the [example/standalone-vault-provided-creds](example/standalone-vault-provided-creds) folder.
+
 ```sh
 make up
 ```
 
-### Providers
-- [Nomad](https://registry.terraform.io/providers/hashicorp/nomad/latest/docs)
-- [Vault](https://registry.terraform.io/providers/hashicorp/vault/latest/docs)
+### Verifying setup
+You can verify the setup by connection to Hive using the Nomad UI at [localhost:4646](http://localhost:4646/). Follow the steps below.
+1. Locate and click the *hive-metastore* service.
+2. Click the *exec* button and connect to the *metastoreserver* task.
+3. Run `beeline -u jdbc:hive2://` to connect to hive.
+4. Run `SHOW databases;`. Your output should look like this:
+```sh
+OK
++----------------+
+| database_name  |
++----------------+
+| default        |
++----------------+
+```
+
+#### Data example upload
+Check [example/README.md#data-example-upload](example/README.md#data-example-upload)
 
-### Intentions
+## Intentions
 Module is deployed with [service mesh approach using consul-connect integration](https://www.consul.io/docs/connect), where [communication `service-to-service` controlled by intentions](https://learn.hashicorp.com/tutorials/consul/get-started-service-networking#control-communication-with-intentions).
 Intentions are required **`only`** when [consul acl is enabled and default_policy is deny](https://learn.hashicorp.com/tutorials/consul/access-control-setup-production#enable-acls-on-the-agents).
 
@@ -76,99 +103,130 @@ In the examples, intentions are created in the Ansible playboook [00_create_inte
 | hive\_container\_port | Hive container port | number | 9083 | no |
 | hive\_docker\_image | Hive container image | string | "fredrikhgrelland/hive:3.1.0" | no |
 | hive\_container\_environment\_variables | Hive environment variables | list(string) | [""] | no |
-| resource | Resource allocations | object | - | no |
-| resource.cpu | Resource allocation - cpu | number | 500 | no |
-| resource.memory | Resource allocation - memory | number | 1024 | no |
-| resource_proxy | Resource allocations for proxy | object | - | no |
-| resource_proxy.cpu | Resource allocation for proxy - cpu | number | 200 | no |
-| resource_proxy.memory | Resource allocation for proxy - memory | number | 128 | no |
-| hive\_bucket | Hive requires minio buckets | obj(string) |  { default = string, hive = string } | no |
-| minio\_service | Minio data-object contains service_name, port, access_key and secret_key | obj(string) | { service_name = string, port = number, access_key = string, secret_key = string } | no |
-| postgres\_service | Postgres data-object contains service_name, port, database_name, username and password | obj(string) | { service_name  = string, port = number, database_name = string, username = string, password = string } | no |
+| resource | Resource allocations for cpu and memory | obj(number, number) | { <br> cpu = 500, <br> memory = 1024 <br> } | no |
+| resource\_proxy | Resource allocations for proxy | obj(number, number) | { <br> cpu = 200, <br> memory = 128} <br> | no |
+| hive\_bucket | Hive requires minio buckets | obj(string, string) |  { <br> default = string, <br> hive = string <br> } | no |
+| minio\_service | Minio data-object contains service_name, port, access_key and secret_key | obj(string, number, string, string) | - | no |
+| minio\_vault\_secret | Minio data-object contains vault related information to fetch credentials | obj(bool, string, string, string, string) | { <br> use_vault_provider = false, <br> vault_kv_policy_name = "kv-secret", <br> vault_kv_path = "secret/path/to/minio/creds", <br> vault_kv_access_key_name = "access_key", <br> vault_kv_secret_key_name = "secret_key" <br> } | no |
+| postgres\_service | Postgres data-object contains service_name, port, database_name, username and password | obj(string, number, string, string, string) |  | no |
+| postgres\_vault\_secret | Postgres data-object contains vault related information to fetch credentials | obj(bool, string, string, string, string) | { <br> use_vault_provider = false, <br> vault_kv_policy_name = "kv-secret", <br> vault_kv_path = "secret/path/to/postgres/creds", <br> vault_kv_username_name = "username", <br> vault_kv_password_name = "password" <br> } | no |
 
 ## Outputs
 | Name | Description | Type |
 |------|-------------|------|
 | service\_name | Hive service name | string |
 | buckets | Minio buckets for hive | string |
+| port | Hive service port | number |
 
 ## Modes
 Hive can be run in two modes:
 - [hivemetastore](./docker/bin/hivemetastore)
 - [hiveserver](./docker/bin/hiveserver)
 
-`NB!` current implementation supports only [`hivemetastore`](conf/nomad/hive.hcl#L104)
+**NB!** current implementation supports only [`hivemetastore`](conf/nomad/hive.hcl#110)
+
+## Examples
+Folder [example](example) contains examples of module usage, please refer for more details.
 
-## Example
 The example-code shows the minimum of what you need do to set up this module.
-```hcl-terraform
+```hcl
 module "minio" {
-  source = "github.com/fredrikhgrelland/terraform-nomad-minio.git?ref=0.1.0"
+  source = "github.com/fredrikhgrelland/terraform-nomad-minio.git?ref=0.3.0"
 
   # nomad
   nomad_datacenters = ["dc1"]
   nomad_namespace   = "default"
+  nomad_host_volume = "persistence-minio"
 
   # minio
-  service_name                    = "minio"
-  host                            = "127.0.0.1"
-  port                            = 9000
-  container_image                 = "minio/minio:latest"
-  access_key                      = "minio"
-  secret_key                      = "minio123"
+  service_name    = "minio"
+  host            = "127.0.0.1"
+  port            = 9000
+  container_image = "minio/minio:latest" # todo: avoid using tag latest in future releases
+
+  # user provided  credentials
+  access_key = "minio"
+  secret_key = "minio123"
+  vault_secret = {
+    use_vault_provider   = false,
+    vault_kv_policy_name = "",
+    vault_kv_path        = "",
+    vault_kv_access_key  = "",
+    vault_kv_secret_key  = ""
+  }
+
+  data_dir                        = "/minio/data"
   buckets                         = ["default", "hive"]
   container_environment_variables = ["JUST_EXAMPLE_VAR1=some-value", "ANOTHER_EXAMPLE2=some-other-value"]
-  resource = {
-    cpu     = 500,
-    memory  = 1024
-  }
+  use_host_volume                 = true
+  use_canary                      = true
 
   # mc
   mc_service_name                    = "mc"
-  mc_container_image                 = "minio/mc:latest"
+  mc_container_image                 = "minio/mc:latest" # todo: avoid using tag latest in future releases
   mc_container_environment_variables = ["JUST_EXAMPLE_VAR3=some-value", "ANOTHER_EXAMPLE4=some-other-value"]
 }
 
 module "postgres" {
-  source = "github.com/fredrikhgrelland/terraform-nomad-postgres.git?ref=0.1.0"
+  source = "github.com/fredrikhgrelland/terraform-nomad-postgres.git?ref=0.3.0"
 
   # nomad
   nomad_datacenters = ["dc1"]
   nomad_namespace   = "default"
+  nomad_host_volume = "persistence-postgres"
 
   # postgres
-  service_name                    = "postgres"
-  container_image                 = "postgres:12-alpine"
-  container_port                  = 5432
+  service_name    = "postgres"
+  container_image = "postgres:12-alpine"
+  container_port  = 5432
+  vault_secret = {
+    use_vault_provider     = false,
+    vault_kv_policy_name   = "",
+    vault_kv_path          = "",
+    vault_kv_username_name = "",
+    vault_kv_password_name = ""
+  }
   admin_user                      = "hive"
   admin_password                  = "hive"
   database                        = "metastore"
-  container_environment_variables = ["PGDATA=/var/lib/postgresql/data"]
+  volume_destination              = "/var/lib/postgresql/data"
+  use_host_volume                 = true
+  use_canary                      = true
+  container_environment_variables = ["PGDATA=/var/lib/postgresql/data/"]
 }
 
 module "hive" {
-  source = "./.."
+  source = "../.."
 
   # nomad
-  nomad_datacenters      = ["dc1"]
-  nomad_namespace        = "default"
-  local_docker_image     = false
+  nomad_datacenters  = ["dc1"]
+  nomad_namespace    = "default"
+  local_docker_image = false
 
   # hive
-  use_canary                           = false
+  use_canary                           = true
   hive_service_name                    = "hive-metastore"
   hive_container_port                  = 9083
   hive_docker_image                    = "fredrikhgrelland/hive:3.1.0"
   hive_container_environment_variables = ["SOME_EXAMPLE=example-value"]
+  
+  resource = {
+    cpu    = 500,
+    memory = 1024
+  }
+  resource_proxy =  {
+    cpu     = 200,
+    memory  = 128
+  }
 
   # hive - minio
   hive_bucket = {
-    default     = "default",
-    hive        = "hive"
+    default = "default",
+    hive    = "hive"
   }
   minio_service = {
     service_name = module.minio.minio_service_name,
-    port         = 9000,
+    port         = module.minio.minio_port,
     access_key   = module.minio.minio_access_key,
     secret_key   = module.minio.minio_secret_key,
   }
@@ -189,26 +247,14 @@ module "hive" {
 }
 ```
 
-### Verifying setup
-
-You can verify the setup by connection to Hive using the Nomad UI at [localhost:4646](http://localhost:4646/). Follow the steps below.
-1. Locate and click the *hive-metastore* service.
-2. Click the *exec* button and connect to the *metastoreserver* task.
-3. Run `beeline -u jdbc:hive2://` to connect to hive.
-4. Run `SHOW databases;`. Your output should look like this:
-```sh
-OK
-+----------------+
-| database_name  |
-+----------------+
-| default        |
-+----------------+
-```
-
-#### Data example upload
-Check [example/README.md#data-example-upload](example/README.md#data-example-upload)
-
-## Authors
+## Contributors
+[<img src="https://avatars0.githubusercontent.com/u/40291976?s=64&v=4">](https://github.com/fredrikhgrelland)
+[<img src="https://avatars2.githubusercontent.com/u/29984156?s=64&v=4">](https://github.com/claesgill)
+[<img src="https://avatars3.githubusercontent.com/u/15572799?s=64&v=4">](https://github.com/zhenik)
+[<img src="https://avatars3.githubusercontent.com/u/67954397?s=64&v=4">](https://github.com/Neha-Sinha2305)
+[<img src="https://avatars3.githubusercontent.com/u/71001093?s=64&v=4">](https://github.com/dangernil)
+[<img src="https://avatars1.githubusercontent.com/u/51820995?s=64&v=4">](https://github.com/pdmthorsrud)
+[<img src="https://avatars1.githubusercontent.com/u/43400021?s=64&v=4">](https://github.com/hannemariavister)
 
 ## License
 This work is licensed under Apache 2 License. See [LICENSE](./LICENSE) for full details.

diff --git a/conf/nomad/hive.hcl b/conf/nomad/hive.hcl
@@ -93,6 +93,12 @@ job "${service_name}" {
     task "metastoreserver" {
       driver = "docker"
 
+%{ if use_vault_provider }
+      vault {
+        policies = ${vault_kv_policy_name}
+      }
+%{ endif }
+
 %{ if local_docker_image }
       artifact {
         source = "s3::http://127.0.0.1:9000/dev/tmp/hive_local.tar"
@@ -139,24 +145,41 @@ CORE_CONF_fs_defaultFS = "s3a://${default_bucket}"
 CORE_CONF_fs_s3a_connection_ssl_enabled = false
 CORE_CONF_fs_s3a_endpoint = "http://{{ env "NOMAD_UPSTREAM_ADDR_${minio_service_name}" }}"
 CORE_CONF_fs_s3a_path_style_access = true
-        EOH
+EOH
       }
       template {
         destination = "local/additional.env"
         env = true
         data = <<EOH
 ${envs}
-        EOH
+EOH
       }
       template {
         destination = "secrets/.env"
         env = true
         data = <<EOH
+# MINIO credentials
+%{ if minio_use_vault_provider }
+{{ with secret "${minio_vault_kv_path}" }}
+CORE_CONF_fs_s3a_access_key = "{{ .Data.data.${minio_vault_kv_access_key_name} }}"
+CORE_CONF_fs_s3a_secret_key = "{{ .Data.data.${minio_vault_kv_secret_key_name} }}"
+{{ end }}
+%{ else }
 CORE_CONF_fs_s3a_access_key = "${minio_access_key}"
 CORE_CONF_fs_s3a_secret_key = "${minio_secret_key}"
+%{ endif }
+
+# POSTGRES credentials
+%{ if postgres_use_vault_provider }
+{{ with secret "${postgres_vault_kv_path}" }}
+HIVE_SITE_CONF_javax_jdo_option_ConnectionUserName="{{ .Data.data.${postgres_vault_kv_username_name} }}"
+HIVE_SITE_CONF_javax_jdo_option_ConnectionPassword="{{ .Data.data.${postgres_vault_kv_password_name} }}"
+{{ end }}
+%{ else }
 HIVE_SITE_CONF_javax_jdo_option_ConnectionUserName="${postgres_username}"
 HIVE_SITE_CONF_javax_jdo_option_ConnectionPassword="${postgres_password}"
-        EOH
+%{ endif }
+EOH
       }
     }
   }