K3s traefik

README.md

@@ -1,7 +1,16 @@
 # SMARTER Demo Deployment Instructions
 
 [![Artifact Hub](https://img.shields.io/endpoint?url=https://artifacthub.io/badge/repository/smarter)](https://artifacthub.io/packages/search?repo=smarter)
-## This demo makes the following assumptions about your environment
+
+The demo can be deployed by using the terraform script on this repository [Terraform](terraform) and following the [readme](terraform/README.md). It is also described on the section "Deploy using terraform" below.
+
+## Deploy using terraform
+
+If you have an AWS account, a terraform script is available on this repository at [Terraform](terraform) and a [readme](terraform/README.md) describes how to use it. This script will allocate an AWS EC2 Graviton instance, install k3s and helm and install all the charts needed to run this demo. The only missing part is one or more edge nodes that the user needs to provide.
+
+## Step by step deployment
+
+### This demo makes the following assumptions about your environment if deployed using helm charts
 
 In this guide we assume you have done the following:
 - You should have an installed InfluxDB and Grafana instance in a separate kubernetes cluster (cloud or local).
@@ -28,7 +37,6 @@ In this guide we assume you have done the following:
 - You must be able to reach your edge node via IP on ports `22`(ssh) and `2520`(Webserver) from your dev machine for parts of this demo to work 
 - The node must be able to reach your k3s-edge-server and cloud-data-node via IP
 
-## Deploy demo
 - To deploy the base system components common to all edge nodes, as well as the demo applications, we opt to use **Helm v3**. To install helm on the device which you are managing your k3s edge cluster with, you can follow the guide [here](https://helm.sh/docs/intro/install/#from-script).
 - Ensure in your environment that your kubeconfig is set properly. As a quick sanity check you can run:
   ```bash

terraform/.gitignore

@@ -0,0 +1,4 @@
+.terraform.lock.hcl
+.terraform
+ssh
+terraform.tfstate

terraform/README.md

@@ -0,0 +1,100 @@
+# Terraform script to install smarter on AWS EC2
+
+This script installs SMARTER example using helm charts into one AWS EC2 instance. 
+
+This figure shows the components of the application and where they reside. 
+![SMARTER](SMARTER_example.png)
+
+It assumes that the environment variables AWS\_ACCESS\_KEY\_ID, AWS\_SECRET\_ACCESS\_KEY and AWS\_SESSION\_TOKEN are set correctly so Terraform can access AWS.
+Set the following variables to correct values:
+region (provider "aws"): AWS region to allocate an EC2 instance on.
+
+Required variables:
+
+* letsencrypt\_email
+
+Optional variables:
+
+* deployment\_name: Prefix to apply to object names.
+* AWS\_EC2\_instance\_type: instance type to be used
+* AWS\_VPC\_subnet\_id: subnet_id use the default of the VPC if this is not defined
+
+## Running
+
+An template.tfvars is provided that can be copied so all the variables are set in this file and referenced by the option -var-file="smarter-variables.tfvars" where smarter-variables.tfvars is the name of the file used to set the variables. Commented variables are ignored.
+
+### Run the following commands if using the smarter-variables.tfvars optionn
+
+```
+terraform init
+# optional: terraform plan -var-file="smarter-variables.tfvars"
+terraform apply -var-file="smarter-variables.tfvars"
+```
+
+### Run the following commands if setting the variables on the command line
+
+```
+terraform init
+# optional: terraform plan -var "letsencrypt_email=<valid email>"
+terraform apply -var "letsencrypt_email=<valid email>"
+```
+
+## Checking status of installation
+
+Please observe that the full installation of k3s, helm charts in the EC2 instance can take up to 15min (expected around 10min) with various parts of the system being available at different times. If it is desired to follow the installation the command below will print the current log and follow it 
+
+```bash
+ssh -i ssh/<deployment-name>-prod-k3s.pem ubuntu@<EC2 instance allocated> "tail -f /var/log/cloud-init-output.log"
+```
+
+## Outputs
+
+Terraform will output the name of EC2 instance allocated and password/ID generated by Terraform.
+
+Grafana web interface can be accessed by https://grafana.\<External IP of EC2 separated with dash\>.sslip.io with user admin and password \<password/ID\>.
+
+A ssh directory will be created locally containing a private/public SSH key that can be used to access the instance using the following command:
+
+```bash
+ssh -i ssh/<deployment-name>-prod-k3s.pem ubuntu@<EC2 instance allocated>
+```
+
+K3s cloud access on the instance (running the cloud containers) can be achieved by setting KUBECONFIG to /etc/rancher/k3s/k3s.yaml. It should be already be set for the ubuntu user at the end of the installation.
+K3s edge, that manages the edge devices and applications running on them, can be accessed by setting KUBECONFIG as $(pwd)/k3s.yaml.\<password/ID\>, that also will be available at the end of the installation.
+
+Helm was used to install charts and can be used to manage them by setting the correct KUBECONFIG.
+
+The edge devices can be installed (Raspberry pi4 for example) by running the following script. The script will install a k3s agent and configure that agent to be a node for k3s edge running on the EC2 instance.
+
+```
+wget https://grafana.<External IP of EC2 separated with dash>.sslip.io//k3s/k3s-start.sh.<password/ID> | bash -s -
+```
+
+Token and k3s.yaml file can be accessed by:
+
+```
+wget https://grafana.<External IP of EC2 separated with dash>.sslip.io//k3s/token.<password/ID> | bash -s -
+wget https://grafana.<External IP of EC2 separated with dash>.sslip.io//k3s/k3s.yaml.<password/ID> | bash -s -
+```
+
+# Troubleshooting
+
+## AWS authentication
+
+Use the AWS credentials provided in the "Get credentials for ProjAdmins" page.
+Terraform expects the following environment variables: AWS\_ACCESS\_KEY\_ID, AWS\_SECRET\_ACCESS\_KEY and AWS\_SESSION\_TOKEN.
+
+## Networking
+
+If an error is reported about "default subnet not found", a subnet was not defined as default for the VPC. A subnet can be set using the AWS\_VPC\_subnet\_id variable.
+
+## Debugging information
+
+Log in to the EC2 machine  using the ssh command
+
+```bash
+ssh -i ssh/<deployment-name>-prod-k3s.pem ubuntu@<EC2 instance allocated>
+```
+
+Please take a look at the log at /var/log/cloud-init-output.log and /var/log/cloud-init.log at the EC2 machine to determine where the program failed
+The script that is executed is called part-002

terraform/k3s/main.tf

@@ -0,0 +1,143 @@
+terraform {
+  required_providers {
+    aws = {
+      source  = "hashicorp/aws"
+      version = "~> 4.16"
+    }
+  }
+
+  required_version = ">= 1.2.0"
+}
+
+data "aws_ami" "ubuntu" {
+  most_recent = true
+
+  filter {
+    name   = "name"
+    #arm64
+    values = ["ubuntu/images/hvm-ssd/ubuntu-jammy-22.04-arm64-server-*"]
+    #x86_64
+    #values = ["ubuntu/images/hvm-ssd/ubuntu-jammy-22.04-amd64-server-*"]
+  }
+  owners = ["099720109477"]
+}
+
+resource "random_string" "agent_token" {
+  length  = 24
+  special = false
+}
+
+resource "random_string" "k3s_edge_id" {
+  length  = 24
+  special = false
+}
+
+resource "aws_iam_instance_profile" "instance_profile" {
+  name  = "${var.deployment_name}-InstanceProfile"
+  role  = var.iam_role_name
+  count = var.iam_role_name == null ? 0 : 1
+}
+
+data "cloudinit_config" "userData" {
+  part {
+    content      = <<EOF
+#cloud-config
+---
+hostname: "${var.deployment_name}"
+EOF
+    content_type = "text/cloud-config"
+  }
+
+  part {
+    content      = <<EOF
+#!/bin/bash
+echo "----- Installing k3s"
+echo K3S_KUBECONFIG_MODE=${var.kubeconfig_mode} K3S_TOKEN=${random_string.agent_token.result} email=${var.letsencrypt_email} > /tmp/variables
+curl -sfL https://get.k3s.io | K3S_KUBECONFIG_MODE=${var.kubeconfig_mode} K3S_TOKEN=${random_string.agent_token.result} sh - 
+echo "----- updating ubuntu"
+apt-get update -y && apt-get upgrade -y && apt-get install awscli git -y
+echo "----- Adding helm"
+curl https://raw.githubusercontent.com/helm/helm/main/scripts/get-helm-3 | bash
+echo "export KUBECONFIG=/etc/rancher/k3s/k3s.yaml" >> /home/ubuntu/.profile
+echo "export KUBECONFIG=/etc/rancher/k3s/k3s.yaml" >> /home/ubuntu/.bashrc
+export ADVERTISE_IP=$(curl http://169.254.169.254/latest/meta-data/public-ipv4)
+export PUBLIC_HOSTNAME=$(curl http://169.254.169.254/latest/meta-data/public-hostname | cut -d '.' -f 1 | sed 's/^ec2-//')
+export LOCAL_IP=$(curl http://169.254.169.254/latest/meta-data/local-ipv4)
+echo "----- Wating for k3s to start"
+until [ -f /etc/rancher/k3s/k3s.yaml ]
+do
+     sleep 5
+done
+echo "----- Adding smarter-cloud to k3s"
+sudo su - ubuntu bash -c "helm repo add smarter https://smarter-project.github.io/documentation;helm install my-smartercloud smarter/smarter-cloud --set email=${var.letsencrypt_email} --set host=grafana --set domain=$PUBLIC_HOSTNAME.sslip.io --set prometheus.grafana.adminPassword=${random_string.k3s_edge_id.result} --wait"
+echo "----- Checking if TLS certificate was generated"
+until [ ! -z "$(kubectl get secret/my-smartercloud-grafana-tls 2>/dev/null)" ]
+do
+     echo "Certificate not generated yet, wait 5 seconds and test again"
+     sleep 5
+done
+echo "----- Adding smarter-edge to k3s"
+sudo su - ubuntu bash -c "helm install my-smartercloud-edge smarter/smarter-k3s-edge --set configuration.externalHostIP=$ADVERTISE_IP --set configuration.hostIP=$LOCAL_IP --set configuration.port=6444 --set configuration.portHTTP=80 --set configuration.id='${random_string.k3s_edge_id.result}' --set configuration.smarter_demo_labels=true --set configuration.host=grafana --set configuration.domain=$PUBLIC_HOSTNAME.sslip.io --set configuration.traefik=true --set configuration.certificateID=my-smartercloud-grafana-tls --set configuration.wwwpath=/k3s/ --wait"
+echo "----- Waiting for k3s.yaml from k3s-edge"
+until [ -f /home/ubuntu/k3s.yaml.${random_string.k3s_edge_id.result} ]
+do
+     sudo su - ubuntu bash -c "wget --no-check-certificate https://grafana.$PUBLIC_HOSTNAME.sslip.io/k3s/k3s.yaml.${random_string.k3s_edge_id.result}"
+     if [ -z "$(grep 'kind: Config' /home/ubuntu/k3s.yaml.${random_string.k3s_edge_id.result})" ]
+     then
+         echo "Received a file but it is not a k3s.yaml file, removing"
+         rm /home/ubuntu/k3s.yaml.${random_string.k3s_edge_id.result}
+     fi
+     sleep 5
+done
+echo "----- Adding smarter-edge to k3s-edge"
+sudo su - ubuntu bash -c "export KUBECONFIG=/home/ubuntu/k3s.yaml.${random_string.k3s_edge_id.result};helm install --create-namespace --namespace smarter my-smartercloud-edge smarter/smarter-edge --wait;helm install --create-namespace --namespace smarter --set global.domain=$(curl http://169.254.169.254/latest/meta-data/public-hostname | cut -d '.' -f 2-) --set smarter-fluent-bit.fluentd.host=$(curl http://169.254.169.254/latest/meta-data/public-hostname | cut -d '.' -f 1) my-smartercloud-demo smarter/smarter-demo --wait"
+echo "----- Finished installing"
+EOF
+    content_type = "text/x-shellscript"
+  }
+
+  part {
+    content      = var.manifest_bucket_path == "" ? "" : <<EOF
+#!/bin/bash
+aws s3 sync s3://${var.manifest_bucket_path} /var/lib/rancher/k3s/server/manifests/
+EOF
+    content_type = "text/x-shellscript"
+  }
+
+#  part {
+#    content      = <<EOF
+##!/bin/bash
+#apt-get update && \
+#apt-get install ec2-instance-connect -y
+#EOF
+#    content_type = "text/x-shellscript"
+#  }
+}
+
+resource "aws_key_pair" "k3s_keypair" {
+  key_name   = var.deployment_name
+  public_key = var.keypair_content
+  count      = 1
+}
+
+resource "aws_instance" "k3s_instance" {
+  ami                         = var.ami_id == null ? data.aws_ami.ubuntu.id : var.ami_id
+  associate_public_ip_address = var.assign_public_ip
+  instance_type               = var.instance_type
+  key_name                    = aws_key_pair.k3s_keypair[0].key_name
+  iam_instance_profile        = var.iam_role_name == null ? null : aws_iam_instance_profile.instance_profile[0].name
+  subnet_id                   = var.subnet_id == "" ? "" : var.subnet_id
+  vpc_security_group_ids      = var.security_group_ids
+  user_data                   = data.cloudinit_config.userData.rendered
+  tags = {
+      Name = "${var.deployment_name}-k3s"
+  }
+}
+
+output "instance" {
+  value = aws_instance.k3s_instance
+}
+
+output "k3s_edge" {
+  value = random_string.k3s_edge_id
+}

terraform/k3s/variables.tf

@@ -0,0 +1,94 @@
+variable "keypair_path" {
+  type        = string
+  default     = ""
+  description = "The path to the public key to use for the instance."
+}
+
+variable "keypair_content" {
+  type        = string
+  default     = ""
+  description = "The raw data to be used for the public key for the instance. If this is used, no value must be specified for 'keypair_path'."
+}
+
+variable "deployment_name" {
+  type        = string
+  default     = "k3s"
+  description = "A unique name used to generate other names for resources, such as instance names."
+}
+
+variable "iam_role_name" {
+  type        = string
+  default     = null
+  description = "The name of an IAM Role to assign to the instance. If left blank, no role will be assigned."
+}
+
+variable "subnet_id" {
+  type        = string
+  default     = ""
+  description = "The ID of a VPC subnet to assign the instance to. If left blank, the instance will be provisioned in the default subnet of the default VPC."
+}
+
+variable "security_group_ids" {
+  type        = list(string)
+  description = "A list of Security Group IDs to assign to the instance."
+}
+
+variable "assign_public_ip" {
+  type        = bool
+  default     = true
+  description = "If set to 'true', a public IP address will be assigned to the instance."
+}
+
+variable "instance_type" {
+  type        = string
+  default     = "t3.small"
+  description = "The AWS EC2 Instance Type to provision the instance as."
+}
+
+variable "manifest_bucket_path" {
+  type        = string
+  default     = ""
+  description = "The AWS S3 bucket name and path that will be used to download manifest files for auto-installation as per [this documentation](https://rancher.com/docs/k3s/latest/en/advanced/). Should be specified as 'bucket name/folder name/'. The IAM Role assigned to the instance must have GetObject access to this bucket."
+}
+
+variable "enable_worker_nodes" {
+  type        = bool
+  description = "If set to 'true', a separate autoscaling group will be created for worker nodes."
+  default     = false
+}
+
+variable "worker_node_min_count" {
+  type        = number
+  description = "The minimum number of worker node instances to provision."
+  default     = 0
+}
+
+variable "worker_node_max_count" {
+  type        = number
+  description = "The maximum number of worker node instances to provsion."
+  default     = 0
+}
+
+variable "worker_node_desired_count" {
+  type        = number
+  description = "The desired number of worker nodes to provision."
+  default     = 0
+}
+
+variable "kubeconfig_mode" {
+  type        = string
+  description = "Sets the file mode of the generated KUBECONFIG file on the master k3s instance. Defaults to '600'."
+  default     = "600"
+}
+
+variable "letsencrypt_email" {
+  type        = string
+  description = "Email to be used in letsencrypt to generate certificates. No default"
+}
+
+variable "ami_id" {
+  type        = string
+  description = "The AMI ID to use when provisioning the instance. If left at the default null value, the latest Ubuntu server image is used."
+  default     = null
+}
+