This repository contains Terraform code and Github Actions workflows to deploy and manage a Tutor Kubernetes-managed production installation of Open edX that will automatically scale up, reliably supporting several hundred thousand learners.
- LMS: https://courses.openedx-devops.abzt.de
- Course Management Studio: https://studio.courses.openedx-devops.abzt.de
- Remote Data Backup: prod-abzt-openedx-eu-backup.s3.amazonaws.com
- Open edX Application User Storage: prod-abzt-openedx-eu-storage.s3.amazonaws.com
- Content Delivery Network (CDN): https://cdn.courses.openedx-devops.abzt.de linked to a public read-only S3 bucket named courses-abzt-openedx-eu-storage
- Bastion: bastion.service.openedx-devops.abzt.de:22. Public ssh access to a t3.micro Ubuntu 20.04 LTS bastion EC2 instance that's preconfigure with all of the software that you'll need to adminster this stack.
- MySQL: mysql.service.openedx-devops.abzt.de:3306. Private VPC access to your AWS RDS MySQL db.t2.small instance with allocated storage of 10.
- MongoDB: mongodb.service.openedx-devops.abzt.de:27017. Private VPC access to your EC2-based installation of MongoDB on a t3.medium instance with allocated storage of 100.
- Kubernetes Dashboard: Dashboard is a web-based Kubernetes user interface. You can use Dashboard to deploy containerized applications to a Kubernetes cluster, troubleshoot your containerized application, and manage the cluster resources. You can use Dashboard to get an overview of applications running on your cluster, as well as for creating or modifying individual Kubernetes resources (such as Deployments, Jobs, DaemonSets, etc). For example, you can scale a Deployment, initiate a rolling update, restart a pod or deploy new applications using a deploy wizard. See: Kubernetes Dashboard Quickstart
- Kubeapps: https://kubeapps.service.openedx-devops.abzt.de. Kubeapps is an in-cluster web-based application that enables users with a one-time installation to deploy, manage, and upgrade applications on a Kubernetes cluster
- Kubecost: https://kubecost.service.openedx-devops.abzt.de. Kubecost provides real-time cost visibility and insights for teams using Kubernetes, helping you continuously reduce your cloud costs.
- Grafana: https://grafana.service.openedx-devops.abzt.de. Grafana is a multi-platform open source analytics and interactive visualization web application. It provides charts, graphs, and alerts for the web when connected to supported data sources.
You can also optionally automatically create additional environments for say, dev and test and QA and so forth. These would result in environments like the following:
- LMS at https://dev.courses.openedx-devops.abzt.de
- CMS at https://studio.dev.courses-openedx-devops.abzt.de
- CDN at https://cdn.dev.courses.openedx-devops.abzt.de linked to an S3 bucket named dev-abzt-openedx-eu-storage
- daily data backups archived into an S3 bucket named dev-abzt-openedx-eu-mongodb-backup
- System Administration Overview
- Passwords, Credentials and Sensitive Data Management
- Remote Data Backup & Restore
- Updating This Repository
- this code only works for AWS.
- the root domain openedx-devops.abzt.de must be hosted in AWS Route53. Terraform will create several DNS entries inside of this hosted zone, and it will optionally create additional hosted zones (one for each additional optional environment) that will be linked to the hosted zone of your root domain.
- resources are deployed to this AWS region:
eu-north-1 - the Github Actions workflows depend on secrets `located here <settings> (see 'secrets/actions' from the left menu bar) `_
- the Github Actions use an AWS IAM key pair from this manually-created user named *ci*
- the collection of resources created by these scripts will generate AWS costs of around $0.41 USD per hour ($10.00 USD per day) while the platform is in a mostly-idle pre-production state. This cost will grow proportionally to your production work loads. You can view your AWS Billing dashboard here
- BE ADVISED that MySQL RDS, MongoDB and Redis ElastiCache are vertically scaled manually and therefore require some insight and potential adjustments on your part. All of these services are defaulted to their minimum instance sizes which you can modify in the environment configuration file
The scripts in the terraform folder provide 1-click functionality to create and manage all resources in your AWS account. These scripts generally follow current best practices for implementing a large Python Django web platform like Open edX in a secure, cloud-hosted environment. Besides reducing human error, there are other tangible improvements to managing your cloud infrastructure with Terraform as opposed to creating and managing your cloud infrastructure resources manually from the AWS console. For example, all AWS resources are systematically tagged which in turn facilitates use of CloudWatch and improved consolidated logging and AWS billing expense reporting.
These scripts will create the following resources in your AWS account:
- Compute Cluster. uses AWS EC2 behind a Classic Load Balancer.
- Kubernetes. Uses `AWS Elastic Kubernetes Service `_ to implement a Kubernetes cluster onto which all applications and scheduled jobs are deployed as pods.
- MySQL. uses AWS RDS for all MySQL data, accessible inside the vpc as mysql.courses.openedx-devops.abzt.de:3306. Instance size settings are located in the environment configuration file, and other common configuration settings are located here. Passwords are stored in Kubernetes Secrets accessible from the EKS cluster.
- MongoDB. uses AWS DocumentDB for all MongoDB data, accessible insid the vpc as mongodb.master.courses.openedx-devops.abzt.de:27017 and mongodb.reader.courses.openedx-devops.abzt.de. Instance size settings are located in the environment configuration file, and other common configuration settings are located here. Passwords are stored in Kubernetes Secrets accessible from the EKS cluster.
- Redis. uses AWS ElastiCache for all Django application caches, accessible inside the vpc as cache.courses.openedx-devops.abzt.de. Instance size settings are located in the environment configuration file. This is necessary in order to make the Open edX application layer completely ephemeral. Most importantly, user's login session tokens are persisted in Redis and so these need to be accessible to all app containers from a single Redis cache. Common configuration settings are located here. Passwords are stored in Kubernetes Secrets accessible from the EKS cluster.
- Container Registry. uses this automated Github Actions workflow to build your tutor Open edX container and then register it in Amazon Elastic Container Registry (Amazon ECR). Uses this automated Github Actions workflow to deploy your container to AWS Amazon Elastic Kubernetes Service (EKS). EKS worker instance size settings are located in the environment configuration file. Note that tutor provides out-of-the-box support for Kubernetes. Terraform leverages Elastic Kubernetes Service to create a Kubernetes cluster onto which all services are deployed. Common configuration settings are located here
- User Data. uses AWS S3 for storage of user data. This installation makes use of a Tutor plugin to offload object storage from the Ubuntu file system to AWS S3. It creates a public read-only bucket named of the form prod-abzt-openedx-eu-storage, with write access provided to edxapp so that app-generated static content like user profile images, xblock-generated file content, application badges, e-commerce pdf receipts, instructor grades downloads and so on will be saved to this bucket. This is not only a necessary step for making your application layer ephemeral but it also facilitates the implementation of a CDN (which Terraform implements for you). Terraform additionally implements a completely separate, more secure S3 bucket for archiving your daily data backups of MySQL and MongoDB. Common configuration settings are located here
- CDN. uses AWS Cloudfront as a CDN, publicly acccessible as https://cdn.courses.openedx-devops.abzt.de. Terraform creates Cloudfront distributions for each of your enviornments. These are linked to the respective public-facing S3 Bucket for each environment, and the requisite SSL/TLS ACM-issued certificate is linked. Terraform also automatically creates all Route53 DNS records of form cdn.courses.openedx-devops.abzt.de. Common configuration settings are located here
- Password & Secrets Management uses Kubernetes Secrets in the EKS cluster. Open edX software relies on many passwords and keys, collectively referred to in this documentation simply as, "secrets". For all back services, including all Open edX applications, system account and root passwords are randomly and strongluy generated during automated deployment and then archived in EKS' secrets repository. This methodology facilitates routine updates to all of your passwords and other secrets, which is good practice these days. Common configuration settings are located here
- SSL Certs. Uses AWS Certificate Manager and LetsEncrypt. Terraform creates all SSL/TLS certificates. It uses a combination of AWS Certificate Manager (ACM) as well as LetsEncrypt. Additionally, the ACM certificates are stored in two locations: your aws-region as well as in us-east-1 (as is required by AWS CloudFront). Common configuration settings are located here
- DNS Management uses AWS Route53 hosted zones for DNS management. Terraform expects to find your root domain already present in Route53 as a hosted zone. It will automatically create additional hosted zones, one per environment for production, dev, test and so on. It automatically adds NS records to your root domain hosted zone as necessary to link the zones together. Configuration data exists within several modules but the highest-level settings are located here
- System Access uses AWS Identity and Access Management (IAM) to manage all system users and roles. Terraform will create several user accounts with custom roles, one or more per service.
- Network Design. uses Amazon Virtual Private Cloud (Amazon VPC) based on the AWS account number provided in the global configuration file to take a top-down approach to compartmentalize all cloud resources and to customize the operating enviroment for your Open edX resources. Terraform will create a new virtual private cloud into which all resource will be provisioned. It creates a sensible arrangment of private and public subnets, network security settings and security groups. See additional VPC documentation here
- Proxy Access to Backend Services. uses an Amazon EC2 t2.micro Ubuntu instance publicly accessible via ssh as bastion.courses.openedx-devops.abzt.de:22 using the ssh key specified in the global configuration file. For security as well as performance reasons all backend services like MySQL, Mongo, Redis and the Kubernetes cluster are deployed into their own private subnets, meaning that none of these are publicly accessible. See additional Bastion documentation here. Terraform creates a t2.micro EC2 instance to which you can connect via ssh. In turn you can connect to services like MySQL via the bastion. Common configuration settings are located here. Note that if you are cost conscious then you could alternatively use AWS Cloud9 to gain access to all backend services.
This repository was generated using Cookiecutter. Keep your repository up to date with the latest Terraform code and configuration versions of the Open edX application stack, AWS infrastructure services and api code libraries by occasionally re-generating the Cookiecutter template using this make file.