ops(iac): improve Openstack Docker terraform example

deployment/terraform/examples/openstack-docker/README.md

@@ -23,9 +23,27 @@ Create a `terraform.tfvars` file, based on `terraform.tfvars.example`, containin
 
 ### 2. Run Terraform
 
+Terraform is run on two modules, so we will run one terraform apply in one folder, then another terraform apply in a second folder. This split is needed to solve dependency ordering with terraform providers. 
+
 ```bash
+# Create VMs in openstack
+cd openstack-vms
 terraform init
-terraform apply
+terraform apply --auto-approve
+
+# Export the created values as environment variables, for usage as terraform variables
+OPENSTACK_HOSTS=$(terraform output -json created_hosts)
+PORTAINER_INSTANCE=$(terraform output -json portainer_instance)
+SSH_PRIVATE_KEY=$(terraform output -json ssh_keys | jq -r .private_key_file)
+
+export TF_VAR_portainer_instance=$PORTAINER_INSTANCE
+export TF_VAR_hosts=$OPENSTACK_HOSTS
+export TF_VAR_ssh_private_key_file=$SSH_PRIVATE_KEY
+
+# Deploy services using docker and portainer 
+cd ../docker-deployment
+terraform init
+terraform apply --auto-approve
 ```
 
 Initial provisioning takes up to 10 minutes, where time is mostly downloading large docker images
@@ -35,28 +53,16 @@ Initial provisioning takes up to 10 minutes, where time is mostly downloading la
 Once the deployment is complete and all services are running, you can access the CogStack platform and its components using the following URLs:
 
 ```bash
-terraform output service_urls
+terraform output 
 ```
 
-## Troubleshooting
-
-
-### unsupported protocol scheme
-If you make changes to the created VM infrastructure, and want to reapply, you can run into this error
-
-```
-│ Error: Get "/api/endpoints/4": unsupported protocol scheme ""
-│ 
-│   with module.cogstack_docker_services.portainer_environment.portainer_envs["cogstack-devops"],
-│   on ../../modules/cogstack-docker-services/environments.tf line 3, in resource "portainer_environment" "portainer_envs":
-│    3: resource "portainer_environment" "portainer_envs" {
-```
-
-Fix by targetting just the infra module first:
-
-```bash
-terraform apply -target=module.openstack_cogstack_infra
-terraform apply
+```hcl
+created_services = {
+  "service_urls" = {
+    "grafana" = "http://10.0.0.1/grafana"
+    "medcat_service" = "http://10.0.0.1:5000"
+    "prometheus" = "http://10.0.0.1/prometheus"
+  }
+}
 ```
 
-For details: the error specifically occurs after making a change to the controller host, forcing it to be deleted and recreated, however terraform still uses the IP address in the portainer provider. Targetting just the infra module first, means terraform wont call any APIs during the plan stage using the old IP address.

deployment/terraform/examples/openstack-docker/docker-deployment/main.tf

@@ -0,0 +1,10 @@
+
+module "cogstack_docker_services" {
+  source = "../../../modules/cogstack-docker-services"
+  hosts  = var.hosts
+  service_targets = {
+    observability  = { hostname = "cogstack-devops" }
+    medcat_service = { hostname = "medcat-nlp" }
+  }
+  ssh_private_key_file = var.ssh_private_key_file
+}

deployment/terraform/examples/openstack-docker/docker-deployment/outputs.tf

@@ -0,0 +1,3 @@
+output "created_services" {
+  value = module.cogstack_docker_services
+}

deployment/terraform/examples/openstack-docker/docker-deployment/providers.tf

@@ -0,0 +1,20 @@
+terraform {
+  required_providers {
+    portainer = {
+      source  = "portainer/portainer"
+      version = "~> 1.10.0"
+    }
+    ansible = {
+      version = "~> 1.3.0"
+      source  = "ansible/ansible"
+    }
+  }
+}
+
+
+provider "portainer" {
+  endpoint        = var.portainer_instance.endpoint
+  api_user        = var.portainer_instance.username
+  api_password    = var.portainer_instance.password
+  skip_ssl_verify = true # optional (default value is `false`)
+}

deployment/terraform/examples/openstack-docker/docker-deployment/terraform.tfvars.example

@@ -0,0 +1,15 @@
+portainer_instance = {
+    endpoint = "https://10.0.0.1:9443"
+    username = ""
+    password = ""
+}
+
+hosts = {
+  "cogstack-devops" = {
+    "ip_address" = "10.0.0.1"
+    "name" = "cogstack-devops"
+    "unique_name" = "w6R2tw-cogstack-devops"
+  }
+}
+
+ssh_private_key_file = "~/my-key.pem"

deployment/terraform/examples/openstack-docker/docker-deployment/variables.tf

@@ -0,0 +1,30 @@
+# Variables for Docker Deployment
+# It's recommended to follow the README.md and use the output of the openstack-vms module
+
+variable "portainer_instance" {
+  type = object({
+    endpoint = string
+    username = string
+    password = string
+  })
+
+  description = <<EOT
+endpoint = API to call portainer on
+username        = Portainer username
+password        = Portainer password to use
+EOT
+}
+
+variable "ssh_private_key_file" {
+  type        = string
+  description = "A filepath to a SSH Private key that is used to SSH login to created hosts"
+}
+
+variable "hosts" {
+  type = map(object({
+    ip_address  = string,
+    unique_name = string,
+    name        = string
+  }))
+  description = "Created Hosts: A map of { hostname: { data } }"
+}

deployment/terraform/examples/openstack-docker/openstack-vms/main.tf

@@ -0,0 +1,9 @@
+module "openstack_cogstack_infra" {
+  source = "../../../modules/openstack-cogstack-infra"
+  host_instances = [
+    { name = "cogstack-devops", is_controller = true },
+    { name = "medcat-nlp" }
+  ]
+  allowed_ingress_ips_cidr = var.allowed_ingress_ips_cidr
+  ubuntu_immage_name       = var.ubuntu_immage_name
+}

deployment/terraform/examples/openstack-docker/openstack-vms/outputs.tf

@@ -0,0 +1,17 @@
+output "created_controller_host" {
+  value = module.openstack_cogstack_infra.created_controller_host
+}
+
+output "created_hosts" {
+  value = module.openstack_cogstack_infra.created_hosts
+}
+
+output "ssh_keys" {
+  sensitive = true
+  value     = module.openstack_cogstack_infra.compute_keypair
+}
+
+output "portainer_instance" {
+  sensitive = true
+  value     = module.openstack_cogstack_infra.portainer_instance
+}