Instance Schedule Optimiser

Instance Schedule Optimiser is a pattern and web application to ingest and analyse EC2 instance usage patterns at an application level via an "ApplicationId" tag.

You can view the number of requests an application has received and recommendations for when you can turn off your application with the corresponding cost savings if the recommendations are followed.

This project provides data that can be used in conjunction with the Instance Scheduler solution

Architecture

Prerequisites

All code was developed using MacOSX. The following instructions should work on any Unix based system.

If you plan to develop using Windows, please use Windows Subsystem for Linux or a docker container containing a linux base system.

• node - version 20+, use NVM to manage node versions where possible.
• pnpm - a node version manager that reduces space on disk and speeds up installs
• AWS CLI
• CDK version 2
• AWS SAM CLI
• Docker

Getting started

First time setup

Setting up this prototype requires 4 steps:

1. Install the repo dependencies.

   This can be done by running pnpm install in terminal in the root directory

2. Define the configuration

   Infrastructure outlined in the “Integration” section needs to already exist prior to this step. You will then need to set up the configuration for the prototype. All configurations can be found in the ./config.ts.example file. Copy the ./config.ts.example file to ./config.ts, then fill in the missing values using the types and comments outlined in the file.

3. Deploy the cross-account role in each application account

   The solution uses CDK to orchestrate the deployment of resources required by the prototype and requires a role in each application account to allow the inventory snapshot function and pricing function to list all application resources in an account at the time of request. All resources are outlined in the ./libs/infra-app-acn/infra.ts file.

   This can be done by first assuming setting the cli credentials of an application account in your terminal, then boot strap the account by running pnpm bootstrap-app-acn in terminal in the root directory followed by pnpm deploy-app-acn. These steps will need to be done for every account that will be monitored.

4. Deploy the solution

   The main application for monitoring usage can be deployed using CDK as well All resources for the main application are outlined in the ./libs/infra/infra.ts file. To deploy the solution, open a terminal and make sure the monitoring account credentials are set, then run pnpm boostrap-infra to bootstrap the account with CDK followed by pnpm deploy-infra to deploy the infrastructure.

5. Create an initial inventory snapshot

   See the next section on Oboarding new applications and regions

Onboarding new applications and regions

Onboarding a new application account will first require the account to send VPC flow logs and CloudTrail logs to the monitoring account in an S3 bucket designated for receiving logs. The application account will also need to have an IAM role that allows the monitoring account to snapshot or build up an inventory list of the current instances and IP addresses in an account to solve for the “Missing Inventory” problem outlined in the challenges section.

Once you have the application account sending it’s logs to the monitoring account and have created a cross-account role, you will need to open the ./config.ts file and add a new record in the “crossAccountConfig” property of the “config” variable map.

The new record should follow the following standard:

{
  crossAccountRoleName: 'MonitoringAccountDescribeEnvironmentRole',
  awsAccountId: '123456789101',
  regions: ['ap-southeast-2'],
}

The fields required are self-explanatory, you can also update the regions array for an account after you’ve added an account and taking the initial inventory of the that account.

To take an account inventory and onboard it to the application, you will need to first open a terminal with aws credentials configured and the current terminal directory pointed to the root directory of the solution repo. The role that is attached to the credentials will need permission to assume the cross-account role defined in ./config.ts in the property currentAccountCrossAccountRole. Then run the following command to make an account snapshot. pnpm nx create infrastructure-snapshot This will run the code in the ./apps/infrastructure-snapshot/src/index.ts file.

The infrastructure snapshot application will run locally and will need permission to assume a role with the name configured in the config file under the “currentAccountCrossAccountRole” key. The application will assume the role, then get all the account configs from the “crossAccountConfig” array and assume each of those roles outlined by the “crossAccountRoleName” property. It will then get perform several API calls against the EC2 API and Autoscaling API to build up a map of IP addresses and their applications. The resulting map will be saved in the s3 bucket referenced in the config under the “s3LogDataBucketName” property and under the S3 key defined by the “ipSnapshotPath” property.

Running the infrastructure-snapshot command multiple times will only cause it to add additional accounts or regions that didn’t exist before, it will not remove accounts or regions. You do not need to re-run this command when you add new applications to accounts that have already been set up since the new applications will be extracted from the CloudTrail logs.

Application commands quick reference

Create the initial IP address snapshot file

Do this only once at the start, this will create a point in time snapshot of all the ip addresses that are currently in an environment.

pnpm snapshot

Deploy the infrasructure

pnpm deploy-infra

Run the UI application locally

pnpm serve