Skip to content

DevOps Onboarding

Jump to bottom
Rendy Oka edited this page Jan 17, 2018 · 2 revisions

Table of Contents

Welcome to Labrador

To help you get started, go ahead and follow the Onboarding guide below. If something is unclear (or wrong), let us know and we'll help you out!

Facility

  1. Don’t be shy. Come introduce yourself to us if we haven’t said hi to you yet. We’ll give you a quick tour of the facility so you know where to get your caffeine fix or take a shower if you choose to run/bike to work.
  2. Talk to the front desk and fill out a welcome form. They will give you a card that will allow you to enter the building. Additionally, the welcome letter that you will receive contains the WiFi password that you will need.
  3. Talk to Joe Mangion to make sure that your paperwork is in order.

Laptop Setup

Before you started, you need to have your equipment set up. Most likely you will be provided with a MacBook Pro. However, if you would like to use a different machine, let us know and we’ll see if that’s something we can accommodate. It is preferred that everyone is on the same hardware to simplify environment management and software licensing.

  1. Your laptop should already be ready to go, but if not, you may want to reformat it first.
  2. Follow the prompts on the screen and create a username/password for yourself. At some point, it will also ask you for the Internet access information. Use the WiFi password that you received in your welcome letter. Unless there’s already someone with your name here, you can choose your first name as the username.
  3. You may also get a Bluetooth keyboard and/or mouse. Let us know if you need help configuring them.

Application Setup

The next step is to install all the necessary software that you will need to access the internal components of Labrador.

Part 1

  1. Your new Labrador e-mail account should be set up. You can access it a number of different ways, including Gmail console, Mail application, Thunderbird, etc. Try logging in and make sure that you have access and are able to send/receive e-mails.
  2. Next, you can try logging into your Trello account. Check your e-mail and follow the invite to see the Labrador boards.
  3. If you don’t already have a GitHub account, please create one. Once you’re done, let us know and we’ll add you to Labrador’s team. Next, you need to create an SSH key to access this repository. You can find the instructions in the link below. You will need to manually:
    • Open https://github.com/labradorcouk/lab-mac-setup/blob/master/git-keys.sh
    • Start with lines 1-4. The ‘pbcopy’ command copies the content of your id_rsa.pub file to clipboard. You will need to paste it into your SSH configuration by going to Settings -> SSH and GPG keys -> “New SSH key” in GitHub
    • Lines 5 and 7 will update your local ~/.gitconfig and ~/.gitignore files. You can verify/modify the contents of these files using your favorite application (e.g. vim, cat, less, etc.)
  4. Your AWS IAM account should be created by now. If not, talk to us and we’ll create it for you. You will be given an AWS Access Key and Secret Key. Keep these safe and secure! Before you do, you want to put these in your OS environment:
    • Open your ~/.zshrc file and append the following lines at the bottom of the file:
       export AWS_ACCESS_KEY_ID= AKIAIOSFODNN7EXAMPLE
      export AWS_SECRET_ACCESS_KEY=wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY
  5. Let’s start installing your applications. If you are using OS X, check out the following instructions:
    • Open https://github.com/labradorcouk/lab-mac-setup/blob/master/setup.sh
    • Start with lines 3, 5, and 7 to install the right applications for the next few lines.
    • Execute lines 10-52. If the installation breaks where it requires you to enter your password, continue from the next package that’s not yet processed.
    • Execute lines 54-55. Verify that they both work.
    • Execute lines 57-81. Just like before, continue from the next package that’s not yet processed if there’s any interruptions along the way.
    • Optionally, you can run line 83. This cleans up your Dock to help you organize it the way you want it.
    • If you are a ‘vim’ user, you can execute lines 86-87 and 89-93. This step will install pathogen that makes it easy to install plugins and runtime files in separate directory environments. After this is done, you can install the vim bundles by executing lines 94-110.

Part 2

With these applications now installed, you can start configuring them.

  1. Open Slack application and try to log in using the credentials given to you. Talk to us if you don’t have this yet. Once you're in, you should be able to communicate to any of the Labrador pack members.
  2. Open Terminal or iTerm2 and set up the AWS CLI by running the following command. Use the AWS credentials that you received earlier and use eu-west-2 as the default region:
    • aws configure
  3. Lastly, let’s set up your access to the Labrador servers.
    • First, you need to create your prod and dev keys. Use the following command to create them. It’s recommended that a password is associated with these keys:
      ssh-keygen -t rsa -f labrador-dev-yourname
      ssh-keygen -t rsa -f labrador-prod-yourname
    • Each command should give you a public file and private file. Keep the private file secure and don’t lose them. Copy the public files (files ending with .pub) into the dev and prod S3 bucket that should have access to now using these commands:
      aws s3 cp ~/.ssh/labrador-dev-yourname.pub s3://labrador-dev-public-keys-bucket/
      aws s3 cp ~/.ssh/labrador-prod-yourname.pub s3://labrador-prod-public-keys-bucket/
    • After a few minutes, try logging into the dev and prod bastion servers. Check EC2 Management Console to get the IP address or hostname of these servers in case they have recently changed:
      ssh -i ~/.ssh/labrador-dev-yourname ubuntu@dev-hostname
      ssh -i ~/.ssh/labrador-prod-yourname ubuntu@prod-hostname

Labrador Components

There are a lot of moving parts that drives this operation. Here’s a short list of what we have and what we are using it for. Understanding how these components interact with each other will help ensure that you are always mindful of the dependencies between them.

Software repository

  • GitHub -
    This is the full library of our internal codes.

Servers

  • Redash -
    TBA
  • Bastion -
    TBA
  • Ingest proxies -
    TBA
  • OpenVPN -
    TBA

Website

  • Wordpress -
    TBA
  • Sign-up journey -
    TBA
  • User dashboard -
    TBA

Dataflow

It’s good to understand how data is generated, how we process them, and how eventually this processed data will make it back to the customers’ screen and to our long term storage for business analysis.

Ingestion process

TBA

Website reporting

TBA

Internal monitoring

TBA

Understand AWS Services

If this is your first time working with AWS and/or cloud computing, there are some good resources online that you can quickly read to get a sense of what the service does and that will give you an idea of how we are leveraging them.

The links will send you to a introduction page of the service. Feel free to read as much as you find interesting or relevant to what you will be working with here.

  • Cognito -
    When new users sign up for our service, we will need to store and manage all the usernames and passwords securely and reliably. We leverage this service to help us do this without the need to manage our own database server.
    See: https://docs.aws.amazon.com/cognito/latest/developerguide/what-is-amazon-cognito.html (Read “What is Amazon Cognito?”)
  • Route53 -
    The number 53 refers to the UDP port that the DNS service uses to translate web addresses to the numerical IP addresses, and vice versa. Our website and API endpoints used by our Retriever devices are hosted and managed by this service.
    See: https://docs.aws.amazon.com/Route53/latest/DeveloperGuide/Welcome.html (Read “What Is Amazon Route 53?” and its first 4 subsections)
  • API Gateway -
    All Labrador Retriever devices that monitor energy consumptions send data through API calls managed by this service. This service can then forward the data to other services, like Lambda or DynamoDB through Kinesis Firehose.
    See: https://docs.aws.amazon.com/apigateway/latest/developerguide/welcome.html (Read “What Is Amazon API Gateway?” and its subsection)
  • Lambda -
    You can think of this service as a compute engine that’s ready to execute any of your codes without the need of provisioning your own service. We use this service to process all energy consumptions that we receive from the Retriever devices.
    See: https://docs.aws.amazon.com/lambda/latest/dg/welcome.html (Read “What Is AWS Lambda?”)
  • Kinesis Firehose -
    This service acts as a buffer to help aggregate and collect data streams. We use this service as a flow control of all the energy data we receive and outputs it to S3 for long term storage, web reporting, and BI purposes.
    See: https://docs.aws.amazon.com/firehose/latest/dev/what-is-this-service.html (Read “What Is Amazon Kinesis Data Firehose?”)
  • RDS -
    When it comes to relational databases, we leverage this service to store some internal information such as Redash, OpenVPN, and some DynamoDB content. This service is simply a managed MySQL and PosgreSQL databases.
    See: https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/Welcome.html (Read “What Is Amazon RDS?”)
  • DynamoDB -
    The counterpart of a relational database is a NoSQL database and that’s what this service provides. We leverage this service to store all of our energy consumption data as well as Labrador’s current subscribers.
    See: https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/Introduction.html (Read “What Is Amazon DynamoDB?” and the “How It Works” subsection)
  • EC2 -
    Cloud computing is practically synonymous to what this service provides, which is pay-per-use hardware that users that provision. We have several compute instances that run bastion servers, OpenVPN servers, and Redash servers, to name a few.
    See: https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/concepts.html (Read “What Is Amazon EC2?” and its first 2 subsections)
  • S3 -
    For all of our long term storage solution, we leverage this service to ensure high availability and redundancy of our critical data. Any data that is not in transit are stored in a bucket and it can be easily accessed by any AWS service.
    See: https://docs.aws.amazon.com/AmazonS3/latest/dev/Introduction.html (Read “Introduction to S3”)

Clone this wiki locally