Skip to content
Permalink
Branch: master
Find file Copy path
Find file Copy path
Fetching contributors…
Cannot retrieve contributors at this time
43 lines (38 sloc) 2.43 KB

README

It's the infrastructure code of an AWS Batch application that will schedule the long-running computing tasks in a cost effective best effect.

Prerequisites

How to deploy batch demo app

# install dependencies of lambda functions
pip3 install -r assets/job-splitter/requirements.txt --target ./assets/job-splitter/
pip3 install -r assets/task-receiver/requirements.txt --target ./assets/task-receiver/
pip3 install -r assets/job-api/requirements.txt --target ./assets/job-api/

cdk deploy BatchAppStack
# or expose restful APIs via API Gateway
./node_modules/aws-cdk/bin/cdk deploy -cApiMode=restapi
# or secure the endpoints of API Gateway by all IAM users in current account
./node_modules/aws-cdk/bin/cdk deploy -cApiMode=restapi -c Auth=iam

Below resources would be created for this stack,

  • Lambda Function TaskReceiver to receive the job request behind a public ALB or API gateway based on the given ApiMode option
  • SQS queue will be created for receiving the job request messages
  • A batch job definition, job queue and compute environments created for computing the tasks. The batch definition will prefer to launch the spot instances for the jobs
  • Lambda Function JobSplitter will split the job requests to sub tasks, which submit the jobs to batch's job queue
  • A DynamoDB table would be created for presisting the job info and result of computing
  • A Lambda function JobAPI exposes the job data via restful API behind ALB

Usage

Schedule a long-running batch task behind ALB

curl -X PUT http://<alb hostname>:9999/v1/new-task  -H 'content-type:application/json' -d '{"job_id": "my-batch-job-id-xxsss22", "universe": ["000002.XSHE", "000004.XSHE"]}'
# Send request with AWS v4 signature
aws sts assume-role --role-arn arn:aws:iam::<accountid>:role/BatchAppStack-API-Execute-Role --role-session-name session1
# send request in Postman with AWS signature or generate v4 signature mannually per doc https://docs.aws.amazon.com/general/latest/gr/signature-version-4.html

Fetch the job status or result behind ALB

curl http://<alb hostname>:9999/v1/tasks/<job_id>
You can’t perform that action at this time.