Skip to content

Commit 931e623

Browse files
ryfeusryfeus
committed
Added Batch and Fargate examples
1 parent b3e2d1f commit 931e623

File tree

8 files changed

+739
-4
lines changed

8 files changed

+739
-4
lines changed

‎.gitignore‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,2 @@
1+
2+
.DS_Store

‎README.md‎

Lines changed: 42 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -3,11 +3,15 @@ Configuration with AWS step functions and lambdas which initiates processing fro
33

44
## Motivation
55

6-
Currently AWS Step Function service provides the best way to unite together serverless processing and cluster processing. Serverless processing can be extremely useful for the cases when you need to have a very good scalability or cheap processing. But for the long or CPU heavy processing you may still need to use cluster. The current version of step function expects cluster to make requests to the Step Function activity. For clusters with extremely expensive machines it may not be the best way for handling so this project allows to transform activity state into active processing trigger. This could be extremely convenient to use with AWS Batch or AWS Fargate.
6+
Currently AWS Step Function service provides the best way to unite together serverless processing and cluster processing. Serverless processing can be extremely useful for the cases when you need to have a very good scalability or cheap processing. But for the long or CPU heavy processing you may still need to use cluster. The current version of step function expects cluster (except for ECS and Batch) to make requests to the Step Function activity. For clusters with extremely expensive machines it may not be the best way for handling so this project allows to transform activity state into active processing trigger. Examples here also include AWS Batch and AWS Fargate
77

88
**TL;DR**: Project allows to initiate processing from AWS Step function instead of making state requests towards it.
99

10-
## Scheme
10+
## Examples
11+
12+
### AWS Lambda example
13+
14+
#### Scheme
1115

1216
![Image](https://s3.amazonaws.com/ryfeus-blog/images/stepFunction.png)
1317

@@ -20,12 +24,46 @@ Currently AWS Step Function service provides the best way to unite together serv
2024
6. Processing lambda conducts processing part and return token and output json to activity.
2125
7. Step functions provides output json from the activity as output
2226

27+
28+
### AWS Fargate example
29+
30+
#### How to deploy
31+
32+
1. Push docker image into ECR (https://docs.aws.amazon.com/AmazonECR/latest/userguide/docker-push-ecr-image.html)
33+
2. Update serverless.yml with image name and instance parameters
34+
3. Install and configure serverless (https://serverless.com/framework/docs/providers/aws/guide/installation/)
35+
4. Install plugins serverless-step-functions and serverless-pseudo-parameters
36+
5. Run ```serverless deploy```
37+
38+
#### Scheme
39+
40+
![Image](https://s3.amazonaws.com/ryfeus-blog/images/FargateScheme.png)
41+
42+
### AWS Batch example
43+
44+
#### How to deploy
45+
46+
1. Push docker image into ECR (https://docs.aws.amazon.com/AmazonECR/latest/userguide/docker-push-ecr-image.html)
47+
2. Update serverless.yml with image name and instance parameters
48+
3. Install and configure serverless (https://serverless.com/framework/docs/providers/aws/guide/installation/)
49+
4. Install plugins serverless-step-functions and serverless-pseudo-parameters
50+
5. Run ```serverless deploy```
51+
52+
#### Scheme
53+
54+
![Image](https://s3.amazonaws.com/ryfeus-blog/images/BatchScheme.png)
55+
2356
## Tools
2457

2558
Project utilises serverless framework with plugin for step functions for orchestration and deployment of the application.
2659

60+
## References
61+
62+
- https://github.com/nathanpeck/aws-cloudformation-fargate
63+
- https://gist.github.com/lizrice/5889f33511aab739d873cb622688317e
64+
2765
## Next steps
2866

29-
- Add AWS Batch processing example
30-
- Add AWS Fargate processing example
67+
- [x] Add AWS Batch processing example
68+
- [x] Add AWS Fargate processing example
3169
- Create example for deep learning/machine learning training processes

‎aws-batch/index.py‎

Lines changed: 8 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,8 @@
1+
def handlerMap(event,context):
2+
return event
3+
4+
def handlerReduce(event,context):
5+
return event
6+
7+
def handlerBranch(event,context):
8+
return 'Hello world'

‎aws-batch/serverless.yml‎

Lines changed: 189 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,189 @@
1+
service: StepFuncBatch
2+
3+
frameworkVersion: ">=1.2.0 <2.0.0"
4+
5+
provider:
6+
name: aws
7+
region: us-east-1
8+
runtime: python3.6
9+
memorySize: 128
10+
timeout: 10
11+
12+
package:
13+
exclude:
14+
- node_modules/**
15+
16+
functions:
17+
branch:
18+
handler: index.handlerBranch
19+
map:
20+
handler: index.handlerMap
21+
reduce:
22+
handler: index.handlerReduce
23+
24+
25+
stepFunctions:
26+
stateMachines:
27+
HelloWorldStepFunction:
28+
events:
29+
- http:
30+
path: startFunction
31+
method: GET
32+
name: ${self:service}-StepFunction
33+
definition:
34+
StartAt: StartStepF
35+
States:
36+
StartStepF:
37+
Type: Task
38+
Resource: arn:aws:lambda:#{AWS::Region}:#{AWS::AccountId}:function:${self:service}-${opt:stage}-map
39+
Next: BatchStep
40+
BatchStep:
41+
Type: Task
42+
Resource: arn:aws:states:::batch:submitJob.sync
43+
Parameters:
44+
JobName: TestStep
45+
JobDefinition: "#{JobDefinition}"
46+
JobQueue: "#{JobQueue}"
47+
ContainerOverrides:
48+
Command:
49+
- 'app/exec'
50+
Next: Parallel
51+
Parallel:
52+
Type: Parallel
53+
Next: EndStepF
54+
Branches:
55+
- StartAt: Branch1
56+
States:
57+
Branch1:
58+
Type: Task
59+
Resource: arn:aws:lambda:#{AWS::Region}:#{AWS::AccountId}:function:${self:service}-${opt:stage}-branch
60+
End: True
61+
- StartAt: Branch2
62+
States:
63+
Branch2:
64+
Type: Task
65+
Resource: arn:aws:lambda:#{AWS::Region}:#{AWS::AccountId}:function:${self:service}-${opt:stage}-branch
66+
End: True
67+
EndStepF:
68+
Type: Task
69+
Resource: arn:aws:lambda:#{AWS::Region}:#{AWS::AccountId}:function:${self:service}-${opt:stage}-reduce
70+
End: true
71+
72+
plugins:
73+
- serverless-step-functions
74+
- serverless-pseudo-parameters
75+
76+
resources:
77+
Parameters:
78+
ImageUrl:
79+
Type: String
80+
Default: stepfunctiontest:latest
81+
Description: The url of a docker image that contains the application process that
82+
will handle the traffic for this service
83+
Resources:
84+
ComputeEnvironment:
85+
Type: AWS::Batch::ComputeEnvironment
86+
Properties:
87+
Type: MANAGED
88+
ServiceRole: arn:aws:iam::#{AWS::AccountId}:role/service-role/AWSBatchServiceRole
89+
ComputeEnvironmentName: '#{AWS::StackName}-ComputeEnvironment'
90+
ComputeResources:
91+
MaxvCpus: 128
92+
SecurityGroupIds:
93+
- !Ref BatchSecurityGroup
94+
Type: SPOT
95+
SpotIamFleetRole: arn:aws:iam::#{AWS::AccountId}:role/aws-ec2-spot-fleet-tagging-role
96+
Subnets:
97+
- !Ref PublicSubnet1
98+
MinvCpus: 0
99+
InstanceRole: ecsInstanceRole
100+
InstanceTypes:
101+
- c5.large
102+
Tags: {"Name": "Batch Instance - #{AWS::StackName}"}
103+
DesiredvCpus: 0
104+
State: ENABLED
105+
JobQueue:
106+
Type: AWS::Batch::JobQueue
107+
Properties:
108+
ComputeEnvironmentOrder:
109+
- Order: 1
110+
ComputeEnvironment: !Ref ComputeEnvironment
111+
State: ENABLED
112+
Priority: 1
113+
JobQueueName: '#{AWS::StackName}-JobQueue'
114+
JobDefinition:
115+
Type: "AWS::Batch::JobDefinition"
116+
Properties:
117+
Type: Container
118+
ContainerProperties:
119+
Command:
120+
- ls
121+
Memory: 128
122+
Vcpus: 1
123+
Image: '#{AWS::AccountId}.dkr.ecr.#{AWS::Region}.amazonaws.com/#{ImageUrl}'
124+
JobDefinitionName: '#{AWS::StackName}-JobDefinition'
125+
RetryStrategy:
126+
Attempts: 1
127+
PubPrivateVPC:
128+
Type: 'AWS::EC2::VPC'
129+
Properties:
130+
CidrBlock: 172.31.0.0/16
131+
Tags:
132+
- Key: Name
133+
Value: !Join [_, [!Ref 'AWS::StackName']]
134+
PublicSubnet1:
135+
Type: 'AWS::EC2::Subnet'
136+
Properties:
137+
VpcId: !Ref PubPrivateVPC
138+
AvailabilityZone: '#{AWS::Region}a'
139+
CidrBlock: 172.31.48.0/20
140+
MapPublicIpOnLaunch: true
141+
InternetGateway:
142+
Type: 'AWS::EC2::InternetGateway'
143+
Properties:
144+
Tags:
145+
- Key: Name
146+
Value: !Join [_, [!Ref 'AWS::StackName']]
147+
- Key: Network
148+
Value: Public
149+
GatewayToInternet:
150+
Type: 'AWS::EC2::VPCGatewayAttachment'
151+
Properties:
152+
VpcId: !Ref PubPrivateVPC
153+
InternetGatewayId: !Ref InternetGateway
154+
PublicRouteTable:
155+
Type: 'AWS::EC2::RouteTable'
156+
Properties:
157+
VpcId: !Ref PubPrivateVPC
158+
Tags:
159+
- Key: Network
160+
Value: Public
161+
PublicRoute:
162+
Type: 'AWS::EC2::Route'
163+
DependsOn: GatewayToInternet
164+
Properties:
165+
RouteTableId: !Ref PublicRouteTable
166+
DestinationCidrBlock: 0.0.0.0/0
167+
GatewayId: !Ref InternetGateway
168+
PublicSubnet1RouteTableAssociation:
169+
Type: 'AWS::EC2::SubnetRouteTableAssociation'
170+
Properties:
171+
SubnetId: !Ref PublicSubnet1
172+
RouteTableId: !Ref PublicRouteTable
173+
NatGateway:
174+
Type: "AWS::EC2::NatGateway"
175+
DependsOn: NatPublicIP
176+
Properties:
177+
AllocationId: !GetAtt NatPublicIP.AllocationId
178+
SubnetId: !Ref PublicSubnet1
179+
NatPublicIP:
180+
Type: "AWS::EC2::EIP"
181+
DependsOn: PubPrivateVPC
182+
Properties:
183+
Domain: vpc
184+
BatchSecurityGroup:
185+
Type: AWS::EC2::SecurityGroup
186+
Properties:
187+
GroupDescription: Allow all ports
188+
VpcId:
189+
Ref: PubPrivateVPC

‎aws-fargate/index.py‎

Lines changed: 8 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,8 @@
1+
def handlerMap(event,context):
2+
return event
3+
4+
def handlerReduce(event,context):
5+
return event
6+
7+
def handlerBranch(event,context):
8+
return 'Hello world'

0 commit comments

Comments
 (0)