Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat(stepfunctions-tasks): task construct to call RunJob on ECS #8451

Merged
merged 33 commits into from
Jun 30, 2020
Merged
Show file tree
Hide file tree
Changes from 29 commits
Commits
Show all changes
33 commits
Select commit Hold shift + click to select a range
bd88a21
set up ecs base class
shivlaks Jun 9, 2020
277fc95
wip
shivlaks Jun 9, 2020
c0ba4c7
add integ tests, update README
shivlaks Jun 9, 2020
40fb7bf
add anchor links in README
shivlaks Jun 9, 2020
d2511e2
mark old implementations as deprecated
shivlaks Jun 9, 2020
28f5eac
Merge branch 'master' into shivlaks/sfn-ecs-tasks
shivlaks Jun 9, 2020
f6e3f0d
PR feedback
shivlaks Jun 18, 2020
96feec1
Merge branch 'master' into shivlaks/sfn-ecs-tasks
shivlaks Jun 22, 2020
c63756f
add support for specifying fargate platform version
shivlaks Jun 23, 2020
0c7b263
drop run task as the current organization is favorable to implementin…
shivlaks Jun 23, 2020
da6ee70
PR feedback - single RunTask class with bind pattern to add fargate/e…
shivlaks Jun 24, 2020
49dbaaa
remove unused task base
shivlaks Jun 24, 2020
23de662
Merge branch 'master' into shivlaks/sfn-ecs-tasks
shivlaks Jun 25, 2020
15693b3
update sfn.Data -> sfn.JsonPath
shivlaks Jun 25, 2020
9078d97
Apply suggestions from code review
shivlaks Jun 25, 2020
ad4da0d
move validation into bind method
shivlaks Jun 26, 2020
1525f3e
modify incorrect default doc
shivlaks Jun 26, 2020
3c7e3bb
reorder methods for easier readability
shivlaks Jun 26, 2020
8bfe251
grab parameters from the bind when rendering task
shivlaks Jun 26, 2020
bedce79
Apply suggestions from code review
shivlaks Jun 26, 2020
6323fae
Apply suggestions from code review
shivlaks Jun 26, 2020
e9b10be
use customer provided security group when supplied
shivlaks Jun 26, 2020
11fe124
Merge branch 'shivlaks/sfn-ecs-tasks' of https://github.com/aws/aws-c…
shivlaks Jun 26, 2020
062ec0b
add LaunchTargetBindOptions
shivlaks Jun 26, 2020
bae8051
modify securityGroup to be securityGroups of type ec2.ISecurityGroup[]
shivlaks Jun 26, 2020
54baf07
drop unnecessary Lazy call for task/execution role
shivlaks Jun 26, 2020
d660d0c
make platformVersion required as we don't want to box ourselves in by…
shivlaks Jun 28, 2020
b123be8
update docstring
shivlaks Jun 28, 2020
f35ec4b
remove errant trailing whitespace
shivlaks Jun 29, 2020
c167ad9
mark renderTask as internal
shivlaks Jun 29, 2020
0179875
Merge branch 'master' into shivlaks/sfn-ecs-tasks
shivlaks Jun 29, 2020
98af020
add comment to describe behaviour of placement constraints
shivlaks Jun 30, 2020
f37e936
Merge branch 'master' into shivlaks/sfn-ecs-tasks
mergify[bot] Jun 30, 2020
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Jump to
Jump to file
Failed to load files.
Diff view
Diff view
112 changes: 94 additions & 18 deletions packages/@aws-cdk/aws-stepfunctions-tasks/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -37,6 +37,8 @@ This module is part of the [AWS Cloud Development Kit](https://github.com/aws/aw
- [UpdateItem](#updateitem)
- [ECS](#ecs)
shivlaks marked this conversation as resolved.
Show resolved Hide resolved
- [RunTask](#runtask)
- [EC2](#ec2)
- [Fargate](#fargate)
- [EMR](#emr)
- [Create Cluster](#create-cluster)
- [Termination Protection](#termination-protection)
Expand Down Expand Up @@ -308,31 +310,105 @@ Step Functions supports [ECS/Fargate](https://docs.aws.amazon.com/step-functions

[RunTask](https://docs.aws.amazon.com/step-functions/latest/dg/connect-ecs.html) starts a new task using the specified task definition.

#### EC2

The EC2 launch type allows you to run your containerized applications on a cluster
of Amazon EC2 instances that you manage.

When a task that uses the EC2 launch type is launched, Amazon ECS must determine where
to place the task based on the requirements specified in the task definition, such as
CPU and memory. Similarly, when you scale down the task count, Amazon ECS must determine
which tasks to terminate. You can apply task placement strategies and constraints to
customize how Amazon ECS places and terminates tasks. Learn more about [task placement](https://docs.aws.amazon.com/AmazonECS/latest/developerguide/task-placement.html)

The following example runs a job from a task definition on EC2
shivlaks marked this conversation as resolved.
Show resolved Hide resolved

```ts
import * as ecs from '@aws-cdk/aws-ecs';
nija-at marked this conversation as resolved.
Show resolved Hide resolved
import * as tasks from '@aws-cdk/aws-stepfunctions-tasks';
import * as sfn from '@aws-cdk/aws-stepfunctions';

// See examples in ECS library for initialization of 'cluster' and 'taskDefinition'
const vpc = ec2.Vpc.fromLookup(stack, 'Vpc', {
isDefault: true,
});

new ecs.RunEcsFargateTask({
cluster,
taskDefinition,
containerOverrides: [
{
containerName: 'TheContainer',
environment: [
{
name: 'CONTAINER_INPUT',
value: JsonPath.stringAt('$.valueFromStateData'),
}
]
}
]
const cluster = new ecs.Cluster(stack, 'Ec2Cluster', { vpc });
cluster.addCapacity('DefaultAutoScalingGroup', {
instanceType: new ec2.InstanceType('t2.micro'),
vpcSubnets: { subnetType: ec2.SubnetType.PUBLIC },
});

const taskDefinition = new ecs.TaskDefinition(stack, 'TD', {
compatibility: ecs.Compatibility.EC2,
});

taskDefinition.addContainer('TheContainer', {
image: ecs.ContainerImage.fromRegistry('foo/bar'),
memoryLimitMiB: 256,
});

const runTask = new tasks.EcsRunTask(stack, 'Run', {
integrationPattern: sfn.IntegrationPattern.RUN_JOB,
cluster,
shivlaks marked this conversation as resolved.
Show resolved Hide resolved
taskDefinition,
launchTarget: new tasks.EcsEc2LaunchTarget({
placementStrategies: [
ecs.PlacementStrategy.spreadAcrossInstances(),
ecs.PlacementStrategy.packedByCpu(),
ecs.PlacementStrategy.randomly(),
],
placementConstraints: [
ecs.PlacementConstraint.memberOf('blieptuut')
],
}),
});
```

#### Fargate

AWS Fargate is a serverless compute engine for containers that works with Amazon
Elastic Container Service (ECS). Fargate makes it easy for you to focus on building
your applications. Fargate removes the need to provision and manage servers, lets you
specify and pay for resources per application, and improves security through application
isolation by design. Learn more about [Fargate](https://aws.amazon.com/fargate/)

The Fargate launch type allows you to run your containerized applications without the need
to provision and manage the backend infrastructure. Just register your task definition and
Fargate launches the container for you.

The following example runs a job from a task definition on Fargate

```ts
import * as ecs from '@aws-cdk/aws-ecs';
import * as tasks from '@aws-cdk/aws-stepfunctions-tasks';
import * as sfn from '@aws-cdk/aws-stepfunctions';

const vpc = ec2.Vpc.fromLookup(stack, 'Vpc', {
isDefault: true,
});

fargateTask.connections.allowToDefaultPort(rdsCluster, 'Read the database');
const cluster = new ecs.Cluster(stack, 'FargateCluster', { vpc });

new sfn.Task(this, 'CallFargate', {
task: fargateTask
const taskDefinition = new ecs.TaskDefinition(stack, 'TD', {
memoryMiB: '512',
cpu: '256',
compatibility: ecs.Compatibility.FARGATE,
});

const containerDefinition = taskDefinition.addContainer('TheContainer', {
image: ecs.ContainerImage.fromRegistry('foo/bar'),
memoryLimitMiB: 256,
});

const runTask = new tasks.EcsRunTask(stack, 'RunFargate', {
integrationPattern: sfn.IntegrationPattern.RUN_JOB,
cluster,
taskDefinition,
containerOverrides: [{
containerDefinition,
environment: [{ name: 'SOME_KEY', value: sfn.JsonPath.stringAt('$.SomeKey') }],
}],
launchTarget: new tasks.EcsFargateLaunchTarget(),
});
```

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -41,6 +41,8 @@ export interface RunEcsEc2TaskProps extends CommonEcsRunTaskProps {

/**
* Run an ECS/EC2 Task in a StepFunctions workflow
*
* @deprecated - replaced by `EcsEc2RunTask`
*/
export class RunEcsEc2Task extends EcsRunTaskBase {
constructor(props: RunEcsEc2TaskProps) {
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -40,6 +40,8 @@ export interface RunEcsFargateTaskProps extends CommonEcsRunTaskProps {

/**
* Start a service on an ECS cluster
*
* @deprecated - replaced by `EcsFargateRunTask`
*/
export class RunEcsFargateTask extends EcsRunTaskBase {
constructor(props: RunEcsFargateTaskProps) {
Expand Down
Original file line number Diff line number Diff line change
@@ -1,3 +1,5 @@
import { ContainerDefinition } from '@aws-cdk/aws-ecs';

/**
* A list of container overrides that specify the name of a container
* and the overrides it should receive.
Expand All @@ -6,7 +8,7 @@ export interface ContainerOverride {
/**
* Name of the container inside the task definition
*/
readonly containerName: string;
readonly containerDefinition: ContainerDefinition;
shivlaks marked this conversation as resolved.
Show resolved Hide resolved

/**
* Command to run inside the container
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -87,7 +87,7 @@ export class EcsRunTaskBase implements ec2.IConnectable, sfn.IStepFunctionsTask
}

for (const override of this.props.containerOverrides || []) {
const name = override.containerName;
const name = override.containerDefinition.containerName;
if (!cdk.Token.isUnresolved(name)) {
const cont = this.props.taskDefinition.node.tryFindChild(name);
if (!cont) {
Expand Down Expand Up @@ -191,7 +191,7 @@ function renderOverrides(containerOverrides?: ContainerOverride[]) {
const ret = new Array<any>();
for (const override of containerOverrides) {
ret.push({
Name: override.containerName,
Name: override.containerDefinition.containerName,
Command: override.command,
Cpu: override.cpu,
Memory: override.memoryLimit,
Expand Down