AWS Small Business Hackathon

• Introduction
• Demo
• Design Architecture
• Fronend Repository
• Install Prerequisites
• Setup Development Environment
• Testing Application Locally
• Deploy your App onto AWS
• Liscense

Team Darwin

Chia-Hua Peng - As an AWS Architect and Backend developer, configure AWS resources using CloudFormation, writing modules that interfaces with AWS resources, and ports the API Gateway with Elasticsearch. Also worked on an alternative solution using Lex chatbot.

Martin Maza - As a Fronend Developer developed the entire frontend app and visualizes the trained data in tables and scatter plots.

Vitalie Manzul - Data Scientist working with Sagemaker, model training, and make data analysis design decisions.

Nikolay Sorokin - Data Scientist working with AWS Comprehend and trained customized data.

Introduction

Use Serverless Technology, Elasticsearch, and Amazon Machine Learning and AI Services to determine if a company will make it to Crunchbase top 50k list based on it’s projects funded by SBIR.

This is the backend API Core. The frontend code is in another repository.

Preview Demo

Frontend with a search engine and a filter table.

Visualizing the award and probability of success in a scatter plot.

A chatbot which may interface with messaging platform like Slack or Facebook Messenger.

Chatbot - Slack Integration

You can also you Kibina to search the trained database.

Youtube Link

Design Architecture

Frontend Repository

AWS-SBA Frontend Development in React JS

AWS Services and Resources Utilized

Compute

• Lambda: serverless compute
  • Search Api: Interfaces with Elasticsearch and Api Gateway.
  • Lex Hook: Interfaces with Amazon Lex. Retrieves information from Elasticsearch and have the bot sending a response over the lex response card.

Storage and Analytics

• S3: Stores the document file objects.
• Elasticsearch: Search enginge database.

Machine Learning

• Comprehend: Detects the key phrases in the context.
• Comprehend Customized: Trained customized data using comprehend.
• Lex: Natrual language processing service that interfaces the app backend with a messaging platform.
• SageMaker: Trained probability of success and insert result into Elasticsearch

Management and Governance

• CloudFormation: maintain and deploy the AWS infrastructure as code
• AWS SAM: build the next stage CloudFormation template

Install Prerequisites

• Python 3.6
• Pipenv

  $ pip3 install pipenv --upgrade
  

• AWS CLI

  $ pip install awscli --upgrade --user
  

Setup Development Environment

Configure the AWS CLI credentials

AWS Configure command is the fastest way to set up your AWS CLI installation.

$ aws configure
AWS Access Key ID [None]: AKIAIOSFODNN7EXAMPLE
AWS Secret Access Key [None]: wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY
Default region name [None]: us-east-1
Default output format [None]: json

Setup Virtual Environment with Pipenv

Pipenv is a dependency management tool for python, which creates and manages a virtualenv for your dependencies.

Install the development package in your virtual environment:

$ export PIPENV_VENV_IN_PROJECT=true && pipenv install --three --dev

Enter your virtual environment:

$ pipenv shell

To exit the virtual environment:

$ exit

Testing Application Locally

Configure a local Elasticsearch instance for testing

You can install Elasticsearch 6.3 locally or use Docker to simulate a Elasticsearch enviornment.

Create an Elasticsearch and Kibana instance from Docker

$ docker run --name esml-es-test -d -p 9200:9200 -p 5601:5601 nshou/elasticsearch-kibana

Debug and test your Elasticsearch using Kibana from http://localhost:5601

Deploy your App onto AWS

Build your application from the makefile

A makefile is provided to help migrating your python source code and dependencies in pipenv to AWS Lambda.

$ make

Deploy the infrastructure from AWS SAM and CloudFormation template

This project uses AWS Serverless Application Models and CloudFormation templates to configure and maintain the AWS infrastructure.

Deploy using "dev" deployment stage:

$ ./deploy.sh

or giving a different deployment stage name by:

$ ./deploy.sh <deployment-stage>

A deploy.sh shell script is provided to help building the next stage CloudFormation template from template.yaml and creating your CloudFormation stack on AWS.

Adding sample sagemaker output

Move 'example-sagemaker-output.json' to the src folder, rename the file to 'dummy-output.json', and run json_es_import.py from the src/ directory.

Importing Amazon Lex

CloudFormation does not yet have an Amazon Lex resource and property types. An Amazon Lex template json is provided in the lex/ directory.

Add Lex Hook Lambda Arn to the template file:

After your CloudFormation stack status turns to CREATE_COMPLETE or UPDATE_COMPLETE, go to your AWS lambda console and find the lex_hook function ARN. Add the lambda ARN to the specified section in the Amazon Lex template.

Import template file to Amazon Lex

Follow the instructions here to import Lex template to the cloud from the console or from the CLI.

Integrating your bot with Slack or other messaging platform

Follow the instructions here to integrate your bot with external messaging platform.

Liscense

MIT License