Lab 1 (Mon): on boarding
By the end of this lab, everybody should be able to build, test, and run their Limbo Slackbots from their laptops. They should also have experience launching their Slackbots to our Amazon Elastic Container Service cluster, so the bots can run 24x7 in the cloud.
We will use your name@mit.edu email address to communicate with you, and we will use the name part of that address as your username for AWS, Slack, etc. (If you don't have an @mit.edu address, we'll figure out what to do with you live during the class.)
The following diagram illustrates the on-boarding steps:
Note that this diagram includes configuration of Travis CI, which we will not be doing until Wednesday.
Let's walk through the above diagram by swimlane.
Ensure your laptop has Docker and Git installed, the tools you'll need to complete this (and the other) lab(s). For Mac laptops, see these instructions. For Windows PCs, see these instructions.
Each student should have a GitHub account. If you don't have a GitHub account and are not familiar with Git, let an instructor know and we'll get you up to speed.
You'll need to fork the tim77code/limbo repository. The fork button is in the top-left on the repo page, circled in red in the following screenshot:
The next step is to create a local clone of the fork of Limbo you made earlier. First, make a directory for the source code if you don't have one. For example:
mkdir ~/src ; cd ~/src
Next, you need to find the address of your fork. The following screenshot indicates where you can find that address:
(Make sure the repository indicated by the red arrow above is your fork of tim77code/limbo.) We suggest using SSH rather than HTTPS addresses, especially for Mac users. For Windows users who have not used ssh before, it's easier to use HTTPS. Use the address indicated by the pull-down to create a clone using the Git command line tool:
git clone git@github.com:<git-username>/limbo.git limbo-personal
or
git clone https://github.com/<git-username>/limbo.git limbo-personal
Once you create this local clone, change your working directory to be the root of the resulting directory:
cd limbo-personal
The rest of these instructions assume that this directory is your working directory.
Finally, we suggest that you add the class repo as the "upstream" of your local copy:
git remote add upstream https://github.com/tim77code/limbo.git
If this is your first time running git, you should run these commands, substituting your email address and name in the appropriate places:
git config --global user.email "you@example.com"
git config --global user.name "Your Name"
We have written an introduction to Slack, which provides some background information on Slack and what we will be using it for in this class. We encourage you to take a minute to read it. In addition to developing a Slackbot in the course, we will be using Slack as a collaboration tool to help each other out through the course. Post requests for help into the #help channel.
- An instructor will send you email or a URL that will let you log into tim77, the Slack organization we created for this course.
- You can create credentials for your Slackbot by going to this page.
- Save these credentials in a file called
.envinside the Limbo source directory that you just checked out. (This file is used by Docker to define environment variables for the containers it creates.) Use the following template for this file, replacing the Slack token below with the one you just created for your Slackbot:$ cat .env SLACK_TOKEN=xoxb-98861257209-83vaWqDDnVXElHdwPrRLNM
At this point you're ready to locally build and test your fork of Limbo. All build, test, and run commands will be performed inside of docker containers. By performing all actions in Docker, we minimize the system requirements on your dev machine (all you need is Make and Docker), and also we ensure that the build and test environment on your development machine agrees with the build and test environment in our continuous integration and deployment system.
To build Limbo, you simply run
docker-compose build
This builds a Docker images labeled limbo in your local Docker cache. This command doesn't publish the image - it remains local for now. The first run of docker-compose build will take several minutes, because Docker will have to download and build a number of basic layers. Fortunately, Docker caches these layers, so subsequent builds run much faster.
To test the local build, run
docker-compose run test
This command generates progress output about the test files and test cases it's running. When testing is done, it prints a test-coverage reports and then a concluding line that looks like this:
========================== 64 passed in 16.38 seconds ==========================
Windows users: if you see tests failing with errors that say some thing like /usr/bin/env: ‘python\r’: No such file or directory then you will need to run git config --global core.autocrlf input (as documented in the Windows set up instructions), then delete the directory created by git clone, and then run the above commands again starting with git clone. You will find that the docker-compose build step runs much faster on the second iteration, so you haven't lost much time.
You will be typing docker-compose run quite often. On Linux, to reduce the amount of typing, you can define an alias:
$ alias dcr="docker-compose run"
with this alias defined, you can run the test suite by typing:
$ dcr test
In the rest of the write up, where we say to type docker-compose run, you can use this alias instead.
On Windows, you can use Set-Alias:
> Set-Alias dc docker-compose
with this alias defined, you can run the test suite by typing:
> dc run test
In the rest of the write up, where we say to type docker-compose, you can use this alias instead.
You can run your Limbo Slackbot locally on your laptop, which facilitates ad hoc testing. To do so, you have to set the SLACK_TOKEN environment variable defined in the .env file as discussed above. (Alternatively, you can define SLACK_TOKEN in your environment, which will override the value in .env.)
With SLACK_TOKEN set, you can start your Slackbot using Docker:
docker-compose run limbo
When you type the above command, your version of Limbo is started in a process local to your laptop. It uses your SLACK_TOKEN to log into Slack's API and starts to listen for messages. In the Slack UI, make sure you are looking at the tim77 team:
you may see the name of your Slackbot appear in the "Apps" section, with a green circle next to it (e.g., the way limbo-travisci appears in this screenshot). Be warned, however, that it doesn't always show up, so looking for it to appear here is not a reliable way of determining if it's running.
Instead, press the + sign that appears next to "Direct Messages" (circled in red above). This will bring you to the following modal dialog:
Type the name of your Slackbot here, and then hit the "Go" button, and then start interacting with your Slackbot. As a start, you might type !help to see what you can do with your Slackbot. As a slightly more elaborate smoke test, you might type '!calc 2+2` and make sure you get a proper response.
If your 'bot doesn't start correctly, type docker ps -l to get the name Docker assigned to it, and type docker logs NAME to see diagnostic output. (The command docker export -o FILE NAME is also sometimes helpful for debugging.)
docker-compose run limbo does not detach the container from your shell process, so the shell will wait for the process to end. You can hit control-C to kill the container when you're done with your ad hoc testing. If you detach the process into the background, you can kill it by typing using docker ps -l to find the name Docker assigned to it, and docker stop NAME to stop it. Alternatively, you can use the following command line:
docker stop `docker ps -q --filter ancestor=limbo --format="{{.ID}}"`
Ultimately, we want your Slackbot to run in the cloud, not on your laptop. To do that, you will need AWS credentials:
-
Your instructor will create an Amazon Web Services (AWS) Identity and Access Management (IAM) account for you. When the account is created, you will receive an email telling you your username and password for the IAM console.
-
You'll need to change your password when you first sign in.
-
After you sign into AWS for the first time, click on "Services" and then "IAM." You might need to enter "IAM" into the search box to find the service.
-
Search IAM for your username, either by using the IAM search box or by clicking on Users and finding your username.
-
Click on the "Security Credentials" tab. You will see that an AWS Access Key has already been created for you. However, you won't be able to recover the secret part of that key, so just delete it. To get an access key you can actually use, click the "Create access key" button. Follow the directions for obtaining the key and add the credentials to the same file
.envfile you used for your Slack credentials. It should now look something like this:$ cat .env SLACK_TOKEN=xoxb-98861257209-83vaWqDDnVXElHdwPrRLNM AWS_ACCESS_KEY_ID=AKIAJWEHIFHQHE76OCFA AWS_SECRET_ACCESS_KEY=+RPKZdocZxhv0Fq4ElDT3aNp0kwiZEE1nw0XBkVq
(On Windows machines, it's very important that carriage return characters are not accidentally inserted into this file! Similarly, if you download you AWS tokens from Amazon in a CSV file, that CSV file will have carriage-return characters at the end of each line: Be careful not to accidentally copy those into your .env file.)
Our Chatbots are intended to be reliable, always-on services. As such, while it's convenient to test them from our local machines, they need to be pushed to the cloud to be available 24x7.
For 6.S188, we're using AWS Elastic Container Service (ECS) to host our Chatbots. Our Makefile contains targets you can use to start, update, and stop your Chatbot in ECS. These targets require that the environment variables AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY, and SLACK_TOKEN are defined in your .env file (or as variables in your shell's environment).
As a pre-condition to running your 'bot in ECS, you need to create a repository in our AWS Elastic Docker Registry (ECR) to hold your 'bot's images. You do this by running:
docker-compose run ecs make_ecr_repo
This uses your GitHub account name as the name of your ECR repository. Once that repository has been created, you can run
docker-compose run ecs start
to start your 'bot. This command will publish your local limbo:latest Docker image as tim77/<limbo-botname>:latest in ECR, where <limbo-botname> is the Slackbot user name associated by Slack with the API token in the SLACK_TOKEN variable. This command will then start your Chatbot in ECS under the service name <limbo-botname>).
You can update your running Slackbot, you run:
docker-compose run ecs update
You would typically do this after making a change to your code and then rebuilding the image with docker-compose build.
To stop your Slackbot, run:
docker-compose run ecs stop
For those new to AWS, we have prepared a cheat sheet for finding the most useful pages in the AWS Console.