-
Notifications
You must be signed in to change notification settings - Fork 0
AWS Workstation Setup
Here are some instructions for how to set up a workstation for working with AWS. This will allow you to do things like create and connect to AWS EC2 instances, and access files stored on AWS S3.
The main tool for connecting your workstation to AWS is the AWS Command Line Interface. If you have pip installed, then it should be easy:
sudo pip install awscli
On macOS / OS X, you might have python byt not pip. In that case you could use Homebrew to install a regular version of Python that comes with pip:
brew install python
Then the pip command worked as above.
You also need to configure the AWS Command Line Interface to work with our AWS account:
aws configure
It will prompt you for some information:
AWS Access Key ID: ???
AWS Secret Access Key: ???
Default region name: us-east-1
Default output format: json
The first two you have to get from the lab's AWS account administrator. The admin should make you an IAM user account and grant you the permissions you'll need:
- For working with EC2 instances, you need AmazonEC2FullAccess.
- For accessing S3 data, you need AmazonS3FullAccess.
The default region should be us-east-1, and the default output format should be json.
Once your AWS CLI is configured you should be able to run jobs on EC2 instances.
Here's an example script for running a simple calculation on a short-lived EC2 instance.
Once your AWS CLI is configured you should be able to access S3 data.
A nice way to do this is to mount an S3 bucket to your local file system and access it like a regular directory. We can do this with a tool called yas3fs.
When it works, the S3 bucket will look like it's part of the local file system. You'll be able to browse the files and open them in the Finder, Matlab, etc. The data will only be transferred when you actually open a file, so it's OK if you can't fit the whole bucket on your local drive.
Here are instructions for setting up yas3fs on macOS / OS X.
You need to have osxfuse installed. This allows you to mount things on the local file system, without getting involved with the kernel.
Go to the releases page, download the .dmg, and run the installer.
Now you can install "Yet Another S3 File System", aka yas3fs. It should be easy with pip:
sudo pip install yas3fs
Finally, you should be able to mount an S3 bucket. You need the name of the bucket, for example render-toolbox-test. You also need to pick a local folder where the bucket will show up, for example ~/Desktop/render-toolbox-test. You can change either value as needed.
yas3fs s3://render-toolbox-test ~/Desktop/render-toolbox-test --mkdir
You should see the volume in the Finder, and you should be able to browse and open files. Browsing might be a little slow and you might have to wait several seconds for all of the files in a folder to appear. This is expected because S3 is not really a file system and not optimized for browsing folders.
When done, you can unmount the bucket:
umount ~/Desktop/render-toolbox-test
rmdir ~/Desktop/render-toolbox-test
Here are instructions for setting up yas3fs on Linux. This example used Linux Mint.