Skip to content
GraphQL starter progressive web application (PWA) with Realtime, Offline and AI/ML functionality using AWS AppSync
CSS JavaScript HTML
Branch: master
Clone or download
Fetching latest commit…
Cannot retrieve the latest commit at this time.
Permalink
Type Name Latest commit message Commit time
Failed to load latest commit information.
.github
backend
media
public
src
.gitignore
CODE_OF_CONDUCT.md
CONTRIBUTING.md
LICENSE
README.md
package-lock.json
package.json

README.md

Workshop: Building a React PWA Chat Application

Introduction

This is a Starter React Progressive Web Application (PWA) that uses AWS AppSync to implement offline and real-time capabilities in a chat application with AI/ML features such as image recognition, text-to-speech, language translation, sentiment analysis as well as conversational chatbots developed as part of the re:Invent session Bridging the Gap Between Real Time/Offline and AI/ML Capabilities in Modern Serverless Apps. In the chat app, users can search for users and messages, have conversations with other users, upload images and exchange messages.

Architecture

ChatQL Overview

The application demonstrates GraphQL Mutations, Queries and Subscriptions with AWS AppSync integrating with other AWS Services:

  • Amazon Cognito for user management as well as Authentication and Authorisation (AuthN/Z)
  • Amazon DynamoDB with multiple data sources (Users, Messages, Conversations, ConvoLink)
  • Amazon Elasticsearch data source for full text search on messages and users
  • Amazon S3 for Media Storage
  • AWS Lambda as a Serverless integration layer for connecting to AI Services
    • Amazon Comprehend for sentiment and entity analysis as well as language detection
    • Amazon Rekognition for object, scene and celebrity detection on images
    • Amazon Lex for conversational chatbots
    • Amazon Polly for text-to-speech on messages
    • Amazon Translate for language translation

Getting Started

Prerequisites

Region Selection

Please Note: If you wish to integrate the AI features as part of this workshop. It's recommended that you launch the entire solution in one of these regions:

  • us-east-1
  • us-west-1
  • eu-west-1

At the time of writing this workshop, Amazon Lex is only available in these regions.

Back-end Setup Instructions

Launch a Serverless Chat Application with AWS Amplify

  1. First, clone this repository and navigate to the created folder:

    git clone https://github.com/StefanBuchman/aws-appsync-chat-workshop.git
    cd aws-appsync-chat-workshop
  2. Init the directory as an amplify Javascript app using the React framework:

The amplify init command is a one-time initialization step for your Amplify powered cloud app. You run this once for each project (JavaScript, iOS, or Android) to connect your app with an AWS backend.

amplify init

Example Output:

Amplify Init

Set the region we are deploying resources to:

export AWS_REGION=$(jq -r '.providers.awscloudformation.Region' amplify/#current-cloud-backend/amplify-meta.json)
echo $AWS_REGION

Make sure ALL services are supported in this region or else you'll get errors in the next steps.

  1. Log into the console and head over to CloudFormation.

    You should see a new stack has been created with a:

    • Deployment bucket
    • Two IAM roles
  2. Add an Amazon Cognito User Pool auth resource.

    amplify add auth

    When prompted:

    "Do you want to use the default authentication and security configuration?" Answer: Use the default configuration

    This will create a local resource for a Cognito user pool. We'll push this configuration up to the cloud shortly.

  3. Add an AppSync GraphQL API with Amazon Cognito User Pool for the API Authentication. Follow the default options. When prompted with "Do you have an annotated GraphQL schema?", select "YES" and provide the schema file path backend/schema.graphql

    amplify add api

Amplify will create your API for you according to the schema defined. It will additionally build out the DynamoDB tables and populate the data sources when Step 7 is executed.

Example Output:

Amplify add api

  1. Add S3 Private Storage for Content (Images, audio, video, etc.) to the project with the default options. Select private read/write access for Auth users only:

    amplify add storage
  2. Now it's time to provision your cloud resources based on the local setup and configured features. When asked:

    Do you want to generate code for your newly created GraphQL API, answer "No" as it would overwrite the current custom files in the src/graphql folder.

    amplify push

    Wait for the provisioning to complete. Once done, a src/aws-exports.js file with the resources information is created.

At this point, go check out the resources that have been created as part of the amplify push command:

In your src directory you should have a file called: aws-exports.js. This file contains the references to the series of resources we created in the cloud.

  • AppSync you should see your new API, made up of the Schema and Data Sources.

  • DynamoDB you should see 4 new tables to support the chat application. These were created by Amplify based off of the GraphQL schema.

  • Cognito a new Cognito User Pool should have been created to store credentials for your chat users.

  • S3 there should be a deployment bucket as well as a new bucket to hold media from the chat application.

  • CloudFormation finally, you should see 4 new stacks, a primary and 3 nested stacks. These ultimately provisioned the resouces during init and push. When cleaning up, these stacks will also tear down the provisioned resources.


Testing the chat app before adding AI features

  1. Install the required modules:

    npm install
  2. Execute the following command to install your project package dependencies and run the application locally:

    amplify serve
  3. Access your chat app at http://localhost:3000

  4. Use two different browsers or one in Incognito/InPrivate mode. Sign up at least 2 different users, authenticate with each user to get them registered in the backend Users table.

  5. Search for your new users to start a conversation and test real-time/offline messaging.

  6. Try to send an image, you should be able to go to your S3 bucket where you'll see the file being committed to.

  7. Head back to Cognito and validate you can see your two new users. You'll be able to view their details, validate that they confimed their identity and even initiate a reset of their password.


Add AI Features to your Serverless Chat Application

Getting AI supporting resources

  1. Look up the S3 bucket name created for user storage:

    export USER_FILES_BUCKET=$(sed -n 's/.*"aws_user_files_s3_bucket": "\(.*\)".*/\1/p' src/aws-exports.js)
    echo $USER_FILES_BUCKET
  2. Retrieve the API ID of your AppSync GraphQL endpoint

    export GRAPHQL_API_ID=$(jq -r '.api[(.api | keys)[0]].output.GraphQLAPIIdOutput' ./amplify/#current-cloud-backend/amplify-meta.json)
    echo $GRAPHQL_API_ID
  3. Retrieve the project's deployment bucket and stackname . It will be used for packaging and deployment with SAM

    export DEPLOYMENT_BUCKET_NAME=$(jq -r '.providers.awscloudformation.DeploymentBucketName' ./amplify/#current-cloud-backend/amplify-meta.json)
    export STACK_NAME=$(jq -r '.providers.awscloudformation.StackName' ./amplify/#current-cloud-backend/amplify-meta.json)
    echo $DEPLOYMENT_BUCKET_NAME
    echo $STACK_NAME

Lambda Functions

Now we need to deploy 3 Lambda functions (one for AppSync and two for Lex) and configure the AppSync Resolvers to use Lambda accordingly.

First, we install the npm dependencies for each lambda function. We then package and deploy the changes with SAM.

Please Note: If you have defined an AWS Profile for the AWS CLI remember to add --profile profile-name to the SAM or CLI commands below.

  1. First, head to the Lambda functions under the backend directory. Take a look through the functions and see what they're doing.

  2. Let's get the dependencies installed and the functions packaged.

    cd ./backend/chuckbot-lambda; npm install; cd ../..
    cd ./backend/moviebot-lambda; npm install; cd ../..
    
    sam package --template-file ./backend/deploy.yaml --s3-bucket $DEPLOYMENT_BUCKET_NAME --output-template-file packaged.yaml
    
    export STACK_NAME_AIML="$STACK_NAME-extra-aiml"
    
    sam deploy --template-file ./packaged.yaml --stack-name $STACK_NAME_AIML --capabilities CAPABILITY_IAM --parameter-overrides appSyncAPI=$GRAPHQL_API_ID s3Bucket=$USER_FILES_BUCKET --region $AWS_REGION
  3. Head over to CloudFormation and validate that the stack has been created. Wait for the stack to complete deploying.

    Head over to your new Lambda functions and validate they have been created.

  4. We'll now retrieve the ARN for the Lambda functions. We'll need the ARN's to allow Lex to point to the correct function to fullfil an intent.

    export CHUCKBOT_FUNCTION_ARN=$(aws cloudformation describe-stacks --stack-name  $STACK_NAME_AIML --query "Stacks[0].Outputs" --region $AWS_REGION | jq -r '.[] | select(.OutputKey == "ChuckBotFunction") | .OutputValue')
    
    export MOVIEBOT_FUNCTION_ARN=$(aws cloudformation describe-stacks --stack-name  $STACK_NAME_AIML --query "Stacks[0].Outputs" --region $AWS_REGION | jq -r '.[] | select(.OutputKey == "MovieBotFunction") | .OutputValue')
    
    echo $CHUCKBOT_FUNCTION_ARN
    echo $MOVIEBOT_FUNCTION_ARN

    Your ARN's should look something like the following: arn:aws:lambda:us-east-1:0123456789:function:awstest1-20190208143507-extra-aiml-ChuckBot-TZJDK8UUOJCC

Lex Chatbots

  1. Let's set up Lex. We will create 2 chatbots: ChuckBot and MovieBot.

    Execute the following commands to add permissions so Lex can invoke the chatbot related Lambda functions you created in the previous section:

    aws lambda add-permission --statement-id Lex --function-name $CHUCKBOT_FUNCTION_ARN --action lambda:\* --principal lex.amazonaws.com --region $AWS_REGION
    aws lambda add-permission --statement-id Lex --function-name $MOVIEBOT_FUNCTION_ARN --action lambda:\* --principal lex.amazonaws.com --region $AWS_REGION

    Update the bots intents with the Lambda ARN:

    jq '.fulfillmentActivity.codeHook.uri = $arn' --arg arn $CHUCKBOT_FUNCTION_ARN backend/ChuckBot/intent.json -M > tmp.txt ; cp tmp.txt backend/ChuckBot/intent.json; rm tmp.txt
    jq '.fulfillmentActivity.codeHook.uri = $arn' --arg arn $MOVIEBOT_FUNCTION_ARN backend/MovieBot/intent.json -M > tmp.txt ; cp tmp.txt backend/MovieBot/intent.json; rm tmp.txt

    And, deploy the slot types, intents and bots:

    aws lex-models put-slot-type --cli-input-json file://backend/ChuckBot/slot-type.json --region $AWS_REGION
    aws lex-models put-intent --cli-input-json file://backend/ChuckBot/intent.json --region $AWS_REGION
    aws lex-models put-bot --cli-input-json file://backend/ChuckBot/bot.json --region $AWS_REGION
    aws lex-models put-slot-type --cli-input-json file://backend/MovieBot/slot-type.json --region $AWS_REGION
    aws lex-models put-intent --cli-input-json file://backend/MovieBot/intent.json --region $AWS_REGION
    aws lex-models put-bot --cli-input-json file://backend/MovieBot/bot.json --region $AWS_REGION
  2. Head over to Lex and walk-through what has been created.

    You should see two Lex chatbots (ChuckBot & MovieBot), the bots may still be building. While they are building take a look at:

    • The intents, these are the actions your chatbot users will attempt to have the chatbot undertake. We can take slots (variables) as part of the intent to make the chatbot more dynamic
    • The slots, these are pieces of information we may need to fullfil an intent.
    • Finally, make sure each chatbot has a Lambda function defined in the Fullfilment section. This is what will be executed once when know what our chatbot user is looking for.

Interacting with Chatbots

The chatbots retrieve information online via API calls from Lambda to The Movie Database (TMDb) (MovieBot, which is based on this chatbot sample) and chucknorris.io (ChuckBot)

  1. Execute the following command to install your project package dependencies and run the application locally:

    amplify serve
  2. Access your chat app at http://localhost:3000

  3. Resume your previous conversation or start a new one.

  4. In order to initiate or respond to a chatbot conversation, you need to start the message with either @chuckbot or @moviebot to trigger or respond to the specific bot, for example:

    • @chuckbot Give me a Chuck Norris fact
    • @moviebot Tell me about a movie
  5. Each subsequent response needs to start with the bot handle (@chuckbot or @moviebot) so the app can detect the message is directed to Lex and not to the other user in the same conversation. Both users will be able to view Lex chatbot responses in real-time powered by GraphQL subscriptions.

  6. Alternatively you can start a chatbot conversation from the message drop-down menu:

    • Just selecting ChuckBot will display options for further interaction
    • Send a message with a nothing but a movie name and selecting MovieBot subsequently will retrieve the details about the movie

Interacting with AWS AI Services

The ./backend/ai-lambda/index.js function contains the code to call the higher-order AI services on AWS. Take a look at this function for how it calls and interacts with those services.

The chatbot integrates with this function either through the drop-down menu on a container of text or through clicking on an image.

Amazon Rekognition

To have Amazon Rekognition detect the contents of a photo, either upload a new photo inside of a chat and then click on the photo or select an existing photo by clicking on it.

Amazon Polly

To have Amazon Polly convert text to speach, from the drop-down menu, select under Listen, select Text to Speech to trigger Amazon Polly and listen to messages read out loud.

The voice is determined by the automatically detected source language (supported languages: English, Mandarin, Portuguese, French and Spanish) the mappings can be found in src/components/Message.js.

Amazon Translate

To have Amazon Translate convert the message to another language, select the desired language under Translate in the drop-down menu (supported languages: English, Mandarin, Portuguese, French and Spanish). In the translation pane, click on the microphone icon to listen to the translated message.

Amazon Comprehend

To have the chatbot peform sentiment and entity analysis on a message using Amazon Comprehend, select Sentiment under the Analyze section on the drop-down menu.

Building, Deploying and Publishing with the Amplify CLI

To have your new chat/chatbot service hosted on AWS, follow the steps below.

  1. Execute amplify add hosting from the project's root folder and follow the prompts to create an S3 bucket (DEV) and/or a CloudFront distribution (PROD).

  2. Build, deploy, upload and publish the application with a single command:

    amplify publish

    Note: If you are deploying a CloudFront distribution, be mindful it needs to be replicated across all points of presence globally and it might take up to 15 minutes to do so.

Access your public chat application using the S3 Website Endpoint URL or the CloudFront URL returned by the amplify publish command.

Share the link with friends, sign up some users, and start creating conversations, uploading images, translating, executing text-to-speech in different languages, performing sentiment analysis and exchanging messages. Be mindful PWAs require SSL, in order to test PWA functionality access the CloudFront URL (HTTPS) from a mobile device and add the site to the mobile home screen.

AWS Amplify Console

The AWS Amplify Console allows you to build a complete CI/CD pipeline directly from your GitHub repository.

The AWS Amplify Console is a continuous deployment and hosting service for mobile web applications. The AWS Amplify Console makes it easier for you to rapidly release new features, helps you avoid downtime during application deployment, and handles the complexity of simultaneously updating the frontend and backend of your applications.

In order to get up and running using AWS Amplify Console

  1. Fork this repository into your own GitHub account and clone it

  2. Install the Amplify CLI with multienv support:

    npm install -g @aws-amplify/cli@multienv

    More info here.

  3. Repeat Steps 3 to 6 from the Back-end Setup Instructions in the previous section.

    Do not perform step 7 (amplify push).

  4. Commit the changes to your forked repository. A new folder amplify will be commited with the project details.

  5. Connect your repository to the Amplify Console as per the instructions here, making sure the name of the branch in your repository matches the name of the environment configured on amplify init (i.e. master).

When prompted with "We detected a backend created with the Amplify Framework. Would you like Amplify Console to deploy these resources with your frontend?", select "YES" and provide or create an IAM role with appropriate permissions to build the backend resources

  1. Wait for the build, deployment and verification steps

At this point you have an usable serverless chat application with no AI features. The next steps are only needed to deploy and configure the integration with services that provide image recognition, text-to-speech, language translation, sentiment analysis as well as conversational chatbots. From here you can skip to step 8 if there's no interest to setup the AI integration.


  1. Now perform steps 7 to 12 from the Back-end Setup Instructions and continue with the instructions to setup the Lex chatbot.

  2. Access your app from the hosted site generated by the Amplify Console(https://master.xxxxxxxx.amplifyapp.com)

Clean Up

To clean up the project, you can simply delete the bots, delete the stack created by the SAM CLI:

aws lex-models delete-bot --name `jq -r .name backend/ChuckBot/bot.json` --region $AWS_REGION
aws lex-models delete-bot --name `jq -r .name backend/MovieBot/bot.json` --region $AWS_REGION
aws lex-models delete-intent --name `jq -r .name backend/ChuckBot/intent.json` --region $AWS_REGION
aws lex-models delete-intent --name `jq -r .name backend/MovieBot/intent.json` --region $AWS_REGION
aws lex-models delete-slot-type --name `jq -r .name backend/ChuckBot/slot-type.json` --region $AWS_REGION
aws lex-models delete-slot-type --name `jq -r .name backend/MovieBot/slot-type.json` --region $AWS_REGION

aws cloudformation delete-stack --stack-name $STACK_NAME_AIML --region $AWS_REGION

and use:

amplify delete

to delete the resources created by the Amplify CLI.

Thank You!

We hope you have enjoyed this workshop!

If you have any suggestions on how we can make it better please let us know.

You can’t perform that action at this time.