Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Room Makeover Demo #573

Merged
merged 37 commits into from
May 22, 2024
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
37 commits
Select commit Hold shift + click to select a range
2af1279
Initial Room generation demo code
gfaires Feb 28, 2024
0408c6b
Bug fixes and minor refactorings
gfaires Mar 1, 2024
5265da6
Small refactoring
gfaires Mar 1, 2024
3455784
Cloudformation for deployment, plus small refactoring
gfaires Mar 11, 2024
1665bcb
Add CORS policy and IAM policy for image bucket
gfaires Mar 12, 2024
7b5830d
Automating inference deployment
gfaires Mar 19, 2024
1c9dd99
Refactoring of image analyzer and update to inference endpoint
gfaires Mar 22, 2024
c11c729
Change prompt to use conjugate prompt
gfaires Mar 22, 2024
6ed1a70
Update conjugate prompt with weights
gfaires Mar 23, 2024
b9d9e7f
Refactor
gfaires Mar 26, 2024
38e9084
Change folder names
gfaires Mar 26, 2024
835c9a6
Refactorings and bug fixes
gfaires Mar 27, 2024
39d7213
Add room gallery support with thumbnails
gfaires Apr 6, 2024
ba8a135
Product image processing lambda
gfaires Apr 10, 2024
39ccf90
renaming
gfaires Apr 11, 2024
cb9133f
s3 operations batch job and cloudformation
gfaires Apr 12, 2024
d82ec9f
Step function setup for product pre-processing
gfaires Apr 13, 2024
7579a9b
Bug fixing cloudformation
gfaires Apr 16, 2024
d93a6d8
Handle throttling errors
gfaires Apr 16, 2024
52a50ea
Frontend and template fixes
gfaires Apr 16, 2024
3f9dfdb
Move copy images lambda function to Cloudfront stack
gfaires Apr 16, 2024
b7433ed
Accidentally changed output name
gfaires Apr 16, 2024
f79d8af
Update inference model
gfaires Apr 17, 2024
9e394be
Redirect to login page if not authenticated
gfaires Apr 25, 2024
f3c12ef
Merge branch 'master' into roomgen
gfaires Apr 25, 2024
e2a625f
Fix deployment when Room Maker demo not selected.
gfaires Apr 29, 2024
a494086
Merge branch 'master' into roomgen
gfaires May 1, 2024
e0de5d9
Remove region hardcoding for interacting with Bedrock
gfaires May 1, 2024
c9396c3
Update DLC support for applicable regions. Update documentation
gfaires May 2, 2024
ba536a0
Merge branch 'master' into roomgen
gfaires May 2, 2024
014b753
Implement Autoscaling
gfaires May 7, 2024
6fa19b3
Fix room gallery styling
gfaires May 7, 2024
5c1bd2d
Support GitHub as code source
gfaires May 8, 2024
dbcf19e
Update Lambda timeouts for getting captions and embeddings
gfaires May 14, 2024
bc7897f
Merge branch 'master' into roomgen
gfaires May 15, 2024
901dba2
Merge branch 'master' into roomgen
gfaires May 17, 2024
3d1b93c
Merge branch 'master' into roomgen
gfaires May 20, 2024
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -40,3 +40,5 @@ workshop/datagenerator/*
workshop/requirements.txt
workshop/1-Personalization/generate_interactions_personalize.py
workshop/1-Personalization/generate_interactions_personalize_offers.py
dist
*.code-workspace
7 changes: 4 additions & 3 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -51,9 +51,9 @@ The Retail Demo Store has been tested in the AWS regions indicated in the deploy
| ------------- | ------------- | ------------- |
| US East (N. Virginia) | us-east-1 | Fully supported |
| US West (Oregon) | us-west-2 | Fully supported |
| Europe (Ireland) | eu-west-1 | Partial support (personalized product descriptions and thematic similar product descriptions not supported) |
| Europe (Frankfurt) | eu-central-1 | Partial support (thematic similar product descriptions not supported) |
| Asia Pacific (Tokyo) | ap-northeast-1 | Partial support (personalized product descriptions not supported) |
| Europe (Ireland) | eu-west-1 | Partial support (personalized product descriptions, thematic similar product descriptions and room makeover not supported) |
| Europe (Frankfurt) | eu-central-1 | Partial support (thematic similar product descriptions and room makeover not supported) |
| Asia Pacific (Tokyo) | ap-northeast-1 | Partial support (personalized product descriptions and room makeover not supported) |
| Asia Pacific (Sydney) | ap-southeast-2 | Partial support (personalized product descriptions and thematic similar product descriptions not supported) |

> [Note]
Expand Down Expand Up @@ -113,6 +113,7 @@ When deploying Retail Demo Store, enter the number as a parameter. The number sh

#### Amazon Bedrock Demos
Amazon Bedrock users need to request access to models before they are available for use. To demo Personalised Product Descriptions make sure to enable access to Anthropic's Claude v2 via the Amazon Bedrock Management Console. Instructions on enabling model access can be found [here](https://docs.aws.amazon.com/bedrock/latest/userguide/model-access.html). Amazon Bedrock is currently available in 5 regions globally, US East (N. Virginia), US West (Oregon), Asia Pacific (Tokyo), Europe (Frankfurt) and Asia Pacific (Singapore).
To demo the Room Makeover, enable model access to Anthropic Claude 3 Haiku and Amazon Titan Multimodal Embeddings G1.

## Using the Retail Demo Store Web Application

Expand Down
27 changes: 27 additions & 0 deletions aws/cloudformation-templates/apigateway-deploy.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,27 @@
---
AWSTemplateFormatVersion: 2010-09-09

Description: >
This template deploys the Retail Demo Store API Gateway

Parameters:
ApiGatewayId:
Type: String

Resources:

HttpApiStage:
Type: 'AWS::ApiGatewayV2::Stage'
Properties:
StageName: '$default'
ApiId: !Ref ApiGatewayId
AutoDeploy: true
AccessLogSettings:
DestinationArn: !GetAtt AccessLogs.Arn
Format: '{ "requestId":"$context.requestId", "ip": "$context.identity.sourceIp", "requestTime":"$context.requestTime", "httpMethod":"$context.httpMethod","routeKey":"$context.routeKey", "status":"$context.status","protocol":"$context.protocol", "integrationStatus": $context.integrationStatus, "integrationLatency": $context.integrationLatency, "responseLength":"$context.responseLength", "authorizerError": "$context.authorizer.error" }'

AccessLogs:
Type: AWS::Logs::LogGroup
Properties:
RetentionInDays: 30
LogGroupName: !Sub "/${AWS::StackName}/APIAccessLogs"
26 changes: 7 additions & 19 deletions aws/cloudformation-templates/apigateway.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -686,25 +686,13 @@ Resources:
- - integrations
- !Ref LocationServiceIntegration

HttpApiStage:
Type: 'AWS::ApiGatewayV2::Stage'
DependsOn:
- GetProduct
Properties:
StageName: '$default'
ApiId: !Ref HttpAPI
AutoDeploy: true
AccessLogSettings:
DestinationArn: !GetAtt AccessLogs.Arn
Format: '{ "requestId":"$context.requestId", "ip": "$context.identity.sourceIp", "requestTime":"$context.requestTime", "httpMethod":"$context.httpMethod","routeKey":"$context.routeKey", "status":"$context.status","protocol":"$context.protocol", "integrationStatus": $context.integrationStatus, "integrationLatency": $context.integrationLatency, "responseLength":"$context.responseLength", "authorizerError": "$context.authorizer.error" }'

AccessLogs:
Type: AWS::Logs::LogGroup
Properties:
RetentionInDays: 30
LogGroupName: !Sub "/${AWS::StackName}/APIAccessLogs"

Outputs:
APIEndpoint:
Description: "API Gateway endpoint URL"
Value: !GetAtt HttpAPI.ApiEndpoint
Value: !GetAtt HttpAPI.ApiEndpoint
ApiGatewayAuthorizer:
Description: API Gateway Authorizer id
Value: !Ref LambdaAuthorizer
ApiGatewayId:
Description: API Gateway Id
Value: !Ref HttpAPI
24 changes: 24 additions & 0 deletions aws/cloudformation-templates/base/_template.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -106,6 +106,21 @@ Parameters:
Type: String
Description: ACM Certificate ARN for CloudFront WebUICDN

ResourceBucketImages:
Type: String
Description: >
(Leave empty in most cases) only specify if your product images are stored in a different bucket

ResourceBucketImagesPrefix:
Type: String
Description: >
(Leave empty in most cases) Images bucket prefix only specify if your product images are stored in a different bucket

SourceImagesPackage:
Type: String
Description : full url to the images.tar.gz adress
Default: https://code.retaildemostore.retail.aws.dev/images.tar.gz

Resources:
# Authentication
Authentication:
Expand Down Expand Up @@ -259,6 +274,11 @@ Resources:
CleanupBucketLambdaArn: !Ref CleanupBucketLambdaArn
LoggingBucketName: !GetAtt Buckets.Outputs.LoggingBucketName
ACMCertificateArn: !Ref ACMCertificateArn
ResourceBucketImages: !Ref ResourceBucketImages
ResourceBucketImagesPrefix: !Ref ResourceBucketImagesPrefix
SourceImagesPackage: !Ref SourceImagesPackage
ResourceBucket: !Ref ResourceBucket
ResourceBucketRelativePath: !Ref ResourceBucketRelativePath

# Personalize Resources
Personalize:
Expand Down Expand Up @@ -299,6 +319,10 @@ Outputs:
Description: Authentication Cognito Identity Pool Id.
Value: !GetAtt Authentication.Outputs.IdentityPoolId

CognitoAuthorizedRole:
Description: Authentication Cognito Authorized Role name
Value: !GetAtt Authentication.Outputs.CognitoAuthorizedRole

StackBucketName:
Description: Stack Bucket
Value: !GetAtt Buckets.Outputs.StackBucketName
Expand Down
5 changes: 4 additions & 1 deletion aws/cloudformation-templates/base/authentication.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -262,4 +262,7 @@ Outputs:
Value: !Ref UserPoolClient
IdentityPoolId:
Description: Identity Pool Id
Value: !Ref IdentityPool
Value: !Ref IdentityPool
CognitoAuthorizedRole:
Description: Cognito Authorized Role
Value: !Ref CognitoAuthorizedRole
160 changes: 160 additions & 0 deletions aws/cloudformation-templates/base/cloudfront.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -14,10 +14,35 @@ Parameters:
ACMCertificateArn:
Type: String
Description: ACM Certificate ARN for CloudFront WebUICDN

ResourceBucketImages:
Type: String
Description: >
(Leave empty in most cases) only specify if your product images are stored in a different bucket

ResourceBucketImagesPrefix:
Type: String
Description: >
(Leave empty in most cases) Images bucket prefix only specify if your product images are stored in a different bucket

SourceImagesPackage:
Type: String
Description : full url to the images.tar.gz adress
Default: https://code.retaildemostore.retail.aws.dev/images.tar.gz

ResourceBucket:
Type: String
Description: S3Bucket Bucket where the Resources are stored (cloudformation, images, lambda code)

ResourceBucketRelativePath:
Type: String
Description: S3Bucket Path where the Resources are stored (cloudformation, images, lambda code) (i.e. path/path2), can be empty if resources are at the root of the bucket. MUST contain trailing /


Conditions:
ACMCertificateArnExists:
!Not [!Equals [!Ref ACMCertificateArn, '']]
CustomResourceBucketImages: !Not [ !Equals [!Ref ResourceBucketImages, ""] ]

Resources:

Expand Down Expand Up @@ -194,6 +219,141 @@ Resources:
OriginRequestPolicyId: !Ref UIOriginRequestPolicy
ResponseHeadersPolicyId: 60669652-455b-4ae9-85a4-c4c02393f86c

CopyImagesLambdaFunction:
Type: 'AWS::Lambda::Function'
Properties:
Description: 'Retail Demo Store deployment utility function that copies catalog images from staging bucket to Web UI bucket'
Code:
ZipFile: |
import boto3
import cfnresponse
import tarfile
import urllib.request
import tempfile
import os

def handler(event, context):
print(event)

response_data = {}
response_status = cfnresponse.SUCCESS

try:
source_bucket_name = event['ResourceProperties']['SourceBucket']
source_path = event['ResourceProperties']['SourceBucketPath']
target_bucket_name = event['ResourceProperties']['TargetBucket']
source_images_package = event['ResourceProperties']['SourceImagesPackage']

if event['RequestType'] == 'Create' or event['RequestType'] == 'Update':
os.makedirs('/tmp/images', exist_ok = True)
filename, _ = urllib.request.urlretrieve( source_images_package , '/tmp/images/images.tar.gz')
print(filename)
file = tarfile.open('/tmp/images/images.tar.gz')
file.extractall('/tmp/images/')
file.close()
#remove images.tar.gz

os.remove('/tmp/images/images.tar.gz')

s3 = boto3.resource('s3')
target_bucket = s3.Bucket(target_bucket_name)

# For all files in tmpdirname
for root, dirs, files in os.walk('/tmp/images'):
for filename in files:
local_path = os.path.join(root, filename)
relative_path = os.path.relpath(local_path, '/tmp/images')
s3_path = os.path.join('images/', relative_path)
# Upload file to target bucket
target_bucket.upload_file(local_path, s3_path)

response_data['Message'] = "Resource creation succeeded"
elif event['RequestType'] == 'Delete':
response_data['Message'] = "Resource deletion succeeded"

except Exception as e:
print("Error: " + str(e))
response_status = cfnresponse.FAILED
response_data['Message'] = "Resource {} failed: {}".format(event['RequestType'], e)

cfnresponse.send(event, context, response_status, response_data)

Handler: index.handler
Runtime: python3.12
EphemeralStorage:
Size: 2048
Timeout: 900
MemorySize: 1024
Role: !GetAtt CopyImagesLambdaExecutionRole.Arn

CustomCopyImagesLambdaFunction:
Type: Custom::CopyImagesToWebUI
Properties:
ServiceToken: !GetAtt CopyImagesLambdaFunction.Arn
SourceBucket:
!If [
CustomResourceBucketImages,
!Ref ResourceBucketImages,
!Ref ResourceBucket
]
SourceBucketPath:
!If [
CustomResourceBucketImages,
!Ref ResourceBucketImagesPrefix,
!Ref ResourceBucketRelativePath
]
TargetBucket: !Ref WebUIBucket
SourceImagesPackage: !Ref SourceImagesPackage

CopyImagesLambdaExecutionRole:
Type: 'AWS::IAM::Role'
Properties:
AssumeRolePolicyDocument:
Version: 2012-10-17
Statement:
- Effect: Allow
Principal:
Service: lambda.amazonaws.com
Action:
- 'sts:AssumeRole'
Policies:
- PolicyName: LoggingPolicy
PolicyDocument:
Version: 2012-10-17
Statement:
- Effect: Allow
Action:
- logs:CreateLogGroup
- logs:CreateLogStream
- logs:PutLogEvents
Resource: '*'
- PolicyName: S3
PolicyDocument:
Version: 2012-10-17
Statement:
- Fn::If:
- CustomResourceBucketImages
- Effect: Allow
Action:
- s3:List*
- s3:GetObject
- s3:PutObject
Resource:
- !Sub arn:${AWS::Partition}:s3:::${ResourceBucketImages}/*
- !Sub arn:${AWS::Partition}:s3:::${ResourceBucketImages}
- !Sub arn:${AWS::Partition}:s3:::${WebUIBucket}
- !Sub arn:${AWS::Partition}:s3:::${WebUIBucket}/*
- Effect: Allow
Action:
- s3:List*
- s3:GetObject
- s3:PutObject
Resource:
- !Sub arn:${AWS::Partition}:s3:::${ResourceBucket}/*
- !Sub arn:${AWS::Partition}:s3:::${ResourceBucket}
- !Sub arn:${AWS::Partition}:s3:::${WebUIBucket}
- !Sub arn:${AWS::Partition}:s3:::${WebUIBucket}/*

Outputs:
# Web UI outputs
WebUIBucketName:
Expand Down
Original file line number Diff line number Diff line change
@@ -0,0 +1,3 @@
RepositoryName: retaildemostore-src
BranchName: main
PollForSourceChanges: false
8 changes: 8 additions & 0 deletions aws/cloudformation-templates/codepipeline-config-github.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,8 @@
Owner:
Ref: GitHubUser
Repo:
Ref: GitHubRepo
Branch:
Ref: GitHubBranch
OAuthToken:
Ref: GitHubToken
Loading
Loading