Skip to content

Commit

Permalink
Updates to the AWS Meme Generator application for AWS re:Invent 2013
Browse files Browse the repository at this point in the history
  • Loading branch information
@fulghum
fulghum committed Nov 21, 2013
1 parent 2a0200b commit 3943739
Show file tree
Hide file tree
Showing 20 changed files with 511 additions and 652 deletions.
30 changes: 12 additions & 18 deletions README.txt
View file
@@ -1,5 +1,5 @@
/*
* Copyright 2012 Amazon Technologies, Inc.
* Copyright 2013 Amazon Technologies, Inc.
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
Expand All @@ -16,7 +16,7 @@
Meme Generator Sample Application
=================================

This sample application demonstrates using S3, DynamoDB, and SQS to run an image processing application with a web front-end on Elastic Beanstalk and a back-end running on EC2. For a demonstration of the application and a discussion of its architecture, please watch the following presentation.
This sample application demonstrates using Amazon S3, Amazon DynamoDB, and Amazon SQS to run an image processing application with a web front-end on AWS Elastic Beanstalk and a back-end running on Amazon EC2. For a demonstration of the application and a discussion of its architecture, please watch the following presentation.

http://www.youtube.com/watch?v=YeRNErD81VA&feature=youtu.be

Expand All @@ -25,30 +25,24 @@
Instruction for setup:

1) Download Eclipse for J2EE developers
2) Download the AWS Toolkit for Eclipse from http://aws.amazon.com/eclipse
3) Install a tomcat runtime environment for Eclipse. The simplest way is to choose Window > Preferences > Server > Runtime Environments. Then add a new Apache Tomcat 7.0 runtime environment, and use the "Download and Install" button to download and install a new version of Apache Tomcat.
2) Install the AWS Toolkit for Eclipse from http://aws.amazon.com/eclipse
3) Install a Tomcat runtime environment for Eclipse. The simplest way is to choose Window > Preferences > Server > Runtime Environments. Then add a new Apache Tomcat 7.0 runtime environment, and use the "Download and Install" button to download and install a new version of Apache Tomcat.
4) Import the 3 projects contained in this sample application into your eclipse workspace. Chose File > Import > Existing projects into workspace. Navigate to the directory containing this file and select the "projects" directory.
5) You should now see the Memes, MemeCommon, and MemeWorker projects in your workspace.
6) Add the AWS SDK for Java classpath container to each of the three projects. Right-click on each project and select Build Path > Configure Build Path... Then, in the "Libraries" tab, select "Add Library" and choose the AWS SDK for Java library. The latest SDK will be downloaded at this point, if you don't have one installed already. The code should all compile at this point. If not, clean all three projects with Project > Clean.
7) For the Memes project, which runs in Tomcat, you will need to add the SDK jar and third-party jars into WebContent/WEB-INF/lib. The simplest way to do this is to drag and drop from the file explorer.
7a) Find the directory where the AWS SDK for Java was downloaded. This defaults to ${home}/aws-java-sdk, but can be configured via preferences. We'll call this directory $SDK.
7b) Drag the $SDK/lib/aws-java-sdk-X.Y.Z.jar file into Memes/WebContent/WEB-INF/lib in Eclipse's file explorer. Eclipse will ask if you want to link or copy the files. Copying is more dependable.
7c) Repeat this process for every jar file in $SDK/third-party/*. You can leave out the aspectj, spring, freemarker, and java-mail libraries, since they aren't used by the sample.
7d) Yes, the above process is very tedious. But we don't want to check an SDK (and all required third-party libs) into source control.
8) Locate the AWSCredentials.properties file for each project, and fill in your access key and secret key
9) In the MemeCommon project, edit src/com/amazonaws/memes/AWSResources.java to configure the name of the S3 bucket to use. The application uses the us-west-1 region by default. To use another region, edit the service endpoint constants in this file.
10) To create all the AWS resources required by the application, run the above file as a Java program (right-click, Run As... > Java application)
5) You should now see the Memes, MemeCommon, and MemeWorker projects in your workspace. If the code isn't compiling, make sure you've installed a Tomcat 7.0 runtime as described in step 3. If the code still doesn't compile, you can try cleaning all three projects with Project > Clean.
6) Locate the AWSCredentials.properties file in the MemeCommon project, and fill in your AWS security credentials.
7) In the MemeCommon project, edit src/com/amazonaws/memes/AWSResources.java to configure the name of the S3 bucket to use. The application uses the us-west-1 region by default. To use another region, edit the service endpoint constants in this file.
8) To create all the AWS resources required by the application, run the above file as a Java program (right-click, Run As... > Java application)

Deploying the Web application:

The application is now ready to deploy to elastic beanstalk (or to a local tomcat server).
The application is now ready to deploy to AWS Elastic Beanstalk (or to a local tomcat server).

To deploy the web application to elastic beanstalk, right-click on the Memes project in the package explorer, then select Run As... > Run on server. In the deployment wizard, you can create a new elastic beanstalk environment to deploy the application onto if you haven't created one yet.
To deploy the web application to AWS Elastic Beanstalk, right-click on the Memes project in the package explorer, then select Run As... > Run on server. In the deployment wizard, you can create a new AWS Elastic Beanstalk environment to deploy the application into if you haven't created one yet.

Running the image processing worker:

The other half of the application is the back-end image processing worker, contained in the MemeWorker project. You can either run it locally or on EC2.
The other half of the application is the back-end image processing worker, contained in the MemeWorker project. You can either run it locally or on Amazon EC2.

To run it locally, just right-click on the MemeWorker class and select Run As ... > Java application.

To run it on EC2, export the MemeWorker project as an executable jar using the File > Export menu. Then copy it to an EC2 host, and invoke the command "java -f <jar name>".
To run it on Amazon EC2, export the MemeWorker project as an executable jar using the File > Export menu. Then copy it to an EC2 host, and invoke the command "java -f <jar name>".
1 change: 1 addition & 0 deletions projects/MemeCommon/.classpath
View file
Expand Up @@ -12,5 +12,6 @@
<attribute name="owner.project.facets" value="java"/>
</attributes>
</classpathentry>
<classpathentry kind="con" path="com.amazonaws.eclipse.sdk.AWS_JAVA_SDK"/>
<classpathentry kind="output" path="bin"/>
</classpath>
2 changes: 1 addition & 1 deletion projects/MemeCommon/src/AwsCredentials.properties
View file
@@ -1,4 +1,4 @@
#Insert your AWS Credentials from http://aws.amazon.com/security-credentials
#Tue Sep 25 14:56:08 PDT 2012
secretKey=CHANGEME
accessKey=CHANGEME
secretKey=CHANGEME
3 changes: 0 additions & 3 deletions projects/MemeCommon/src/META-INF/MANIFEST.MF
View file

This file was deleted.

161 changes: 74 additions & 87 deletions projects/MemeCommon/src/com/amazonaws/memes/AWSResources.java
View file
@@ -1,5 +1,5 @@
/*
* Copyright 2012 Amazon Technologies, Inc.
* Copyright 2012-2013 Amazon Technologies, Inc.
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
Expand All @@ -14,110 +14,97 @@
*/
package com.amazonaws.memes;

import com.amazonaws.AmazonServiceException;
import com.amazonaws.auth.AWSCredentialsProvider;
import com.amazonaws.auth.ClasspathPropertiesFileCredentialsProvider;
import com.amazonaws.services.dynamodb.AmazonDynamoDB;
import com.amazonaws.services.dynamodb.AmazonDynamoDBClient;
import com.amazonaws.services.dynamodb.model.CreateTableRequest;
import com.amazonaws.services.dynamodb.model.DescribeTableRequest;
import com.amazonaws.services.dynamodb.model.KeySchema;
import com.amazonaws.services.dynamodb.model.KeySchemaElement;
import com.amazonaws.services.dynamodb.model.ProvisionedThroughput;
import com.amazonaws.services.dynamodb.model.ScalarAttributeType;
import com.amazonaws.services.dynamodb.model.TableDescription;
import com.amazonaws.services.dynamodb.model.TableStatus;
import com.amazonaws.services.s3.AmazonS3;
import com.amazonaws.regions.Region;
import com.amazonaws.regions.Regions;
import com.amazonaws.services.dynamodbv2.AmazonDynamoDBClient;
import com.amazonaws.services.dynamodbv2.datamodeling.DynamoDBMapper;
import com.amazonaws.services.dynamodbv2.model.AttributeDefinition;
import com.amazonaws.services.dynamodbv2.model.CreateTableRequest;
import com.amazonaws.services.dynamodbv2.model.KeySchemaElement;
import com.amazonaws.services.dynamodbv2.model.KeyType;
import com.amazonaws.services.dynamodbv2.model.ProvisionedThroughput;
import com.amazonaws.services.dynamodbv2.model.ScalarAttributeType;
import com.amazonaws.services.dynamodbv2.util.Tables;
import com.amazonaws.services.s3.AmazonS3Client;
import com.amazonaws.services.s3.model.CreateBucketRequest;
import com.amazonaws.services.sqs.AmazonSQS;
import com.amazonaws.services.sqs.AmazonSQSClient;
import com.amazonaws.services.sqs.model.CreateQueueRequest;

/**
* Configuration for AWS resources. Run to set up your resources.
*
* @author zachmu
* Configuration for AWS resources. Run this class to create your resources for
* the Meme Generator sample application.
*/
public class AWSResources {

public static final String BUCKET_NAME = "CHANGEME";
public static final String S3_BUCKET_NAME = "reinvent-meme-generator";
public static final String SQS_QUEUE_NAME = "reinvent-memes";
public static final String DYNAMODB_TABLE_NAME = "reinvent-memes";

public static final String MACRO_PATH = "macros/";
public static final String FINISHED_PATH = "memes/";
public static final String SQS_QUEUE = "reinvent-memes";
public static final String DYNAMO_TABLE_NAME = "reinvent-memes";

public static final String SQS_ENDPOINT = "sqs.us-west-1.amazonaws.com";
public static final String DYNAMO_ENDPOINT = "dynamodb.us-west-1.amazonaws.com";

public static void main(String[] args) {
createResources();
}

private static void createResources() {
ClasspathPropertiesFileCredentialsProvider provider = new ClasspathPropertiesFileCredentialsProvider();

AmazonSQS sqs = new AmazonSQSClient(provider);
sqs.setEndpoint(SQS_ENDPOINT);
sqs.createQueue(new CreateQueueRequest().withQueueName(SQS_QUEUE));

AmazonS3 s3 = new AmazonS3Client(provider);
if ( !s3.doesBucketExist(BUCKET_NAME) ) {
s3.createBucket(new CreateBucketRequest(BUCKET_NAME));
}

AmazonDynamoDBClient dynamo = new AmazonDynamoDBClient(provider);
dynamo.setEndpoint(DYNAMO_ENDPOINT);
public static final String FINISHED_PATH = "memes/";

if ( !doesTableExist(dynamo, DYNAMO_TABLE_NAME) ) {
dynamo.createTable(new CreateTableRequest()
.withTableName(DYNAMO_TABLE_NAME)
.withProvisionedThroughput(
new ProvisionedThroughput().withReadCapacityUnits(50l).withWriteCapacityUnits(50l))
.withKeySchema(
new KeySchema().withHashKeyElement(new KeySchemaElement().withAttributeName("id")
.withAttributeType(ScalarAttributeType.S))));
waitForTableToBecomeAvailable(dynamo, DYNAMO_TABLE_NAME);
}
}
/*
* The SDK provides several easy to use credentials providers.
* Here we're loading our AWS security credentials from a properties
* file on our classpath.
*/
public static final AWSCredentialsProvider CREDENTIALS_PROVIDER =
new ClasspathPropertiesFileCredentialsProvider();

private static boolean doesTableExist(AmazonDynamoDB dynamo, String tableName) {
try {
TableDescription table = dynamo.describeTable(new DescribeTableRequest().withTableName(tableName))
.getTable();
return "ACTIVE".equals(table.getTableStatus());
} catch ( AmazonServiceException ase ) {
if ( ase.getErrorCode().equals("ResourceNotFoundException") )
return false;
throw ase;
}
/*
* This controls the AWS region used for created resources. You can easily
* deploy applications in any or all of the AWS regions around the world,
* allowing you to provide a lower latency and better experience for your
* customers.
*/
public static final Region REGION = Region.getRegion(Regions.US_WEST_1);

/*
* We construct our clients to access AWS here, so that we can share them
* easily throughout our application.
*/
public static final AmazonS3Client S3 = new AmazonS3Client(CREDENTIALS_PROVIDER);
public static final AmazonSQSClient SQS = new AmazonSQSClient(CREDENTIALS_PROVIDER);
public static final AmazonDynamoDBClient DYNAMODB = new AmazonDynamoDBClient(CREDENTIALS_PROVIDER);
public static final DynamoDBMapper DYNAMODB_MAPPER = new DynamoDBMapper(DYNAMODB, CREDENTIALS_PROVIDER);


static {
/*
* Set any other client options that you need here. For example, if you
* connect to the internet through a proxy, then call setConfiguration
* and pass in a ClientConfiguration object with your proxy settings.
*
* Here we set our region, so that we can keep our data located in the
* same region.
*/
DYNAMODB.setRegion(REGION);
SQS.setRegion(REGION);
}

private static void waitForTableToBecomeAvailable(AmazonDynamoDB dynamo, String tableName) {
System.out.println("Waiting for " + tableName + " to become ACTIVE...");

long startTime = System.currentTimeMillis();
long endTime = startTime + (10 * 60 * 1000);
while ( System.currentTimeMillis() < endTime ) {
try {
Thread.sleep(1000 * 20);
} catch ( Exception e ) {
}
try {
DescribeTableRequest request = new DescribeTableRequest().withTableName(tableName);
TableDescription table = dynamo.describeTable(request).getTable();
if ( table == null )
continue;
public static void main(String[] args) {
String queueUrl = SQS.createQueue(new CreateQueueRequest(SQS_QUEUE_NAME)).getQueueUrl();
System.out.println("Using Amazon SQS Queue: " + queueUrl);

String tableStatus = table.getTableStatus();
System.out.println(" - current state: " + tableStatus);
if ( tableStatus.equals(TableStatus.ACTIVE.toString()) )
return;
} catch ( AmazonServiceException ase ) {
if ( ase.getErrorCode().equalsIgnoreCase("ResourceNotFoundException") == false )
throw ase;
}

if ( !S3.doesBucketExist(S3_BUCKET_NAME) ) {
S3.createBucket(new CreateBucketRequest(S3_BUCKET_NAME));
}
System.out.println("Using Amazon S3 Bucket: " + S3_BUCKET_NAME);


throw new RuntimeException("Table " + tableName + " never went active");
if ( !Tables.doesTableExist(DYNAMODB, DYNAMODB_TABLE_NAME) ) {
System.out.println("Creating new AWS DynamoDB Table...");
DYNAMODB.createTable(new CreateTableRequest()
.withTableName(DYNAMODB_TABLE_NAME)
.withKeySchema(new KeySchemaElement("id", KeyType.HASH))
.withAttributeDefinitions(new AttributeDefinition("id", ScalarAttributeType.S))
.withProvisionedThroughput(new ProvisionedThroughput(50l, 50l)));
}
Tables.waitForTableToBecomeActive(DYNAMODB, DYNAMODB_TABLE_NAME);
System.out.println("Using AWS DynamoDB Table: " + DYNAMODB_TABLE_NAME);
}
}

0 comments on commit 3943739

Please sign in to comment.