feat(stepfunctions-tasks): run batch job #6396

ayush987goyal · 2020-02-21T16:13:13Z

Commit Message

feat(stepfunctions-tasks): run batch job (#6396)

Adding implementation to run batch job from step functions based on docs

closes #6467

End Commit Message

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache-2.0 license

aws-cdk-automation · 2020-02-21T16:33:49Z

AWS CodeBuild CI Report

CodeBuild project: AutoBuildProject6AEA49D1-qxepHUsryhcu
Commit ID: c7e8f56
Result: FAILED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

aws-cdk-automation · 2020-02-21T17:34:39Z

AWS CodeBuild CI Report

CodeBuild project: AutoBuildProject6AEA49D1-qxepHUsryhcu
Commit ID: e405aa0
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

aws-cdk-automation · 2020-02-22T15:24:03Z

AWS CodeBuild CI Report

CodeBuild project: AutoBuildProject6AEA49D1-qxepHUsryhcu
Commit ID: bf9ddd4
Result: FAILED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

aws-cdk-automation · 2020-02-22T15:25:51Z

AWS CodeBuild CI Report

CodeBuild project: AutoBuildProject6AEA49D1-qxepHUsryhcu
Commit ID: 0727eff
Result: FAILED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

aws-cdk-automation · 2020-02-22T16:57:20Z

AWS CodeBuild CI Report

CodeBuild project: AutoBuildProject6AEA49D1-qxepHUsryhcu
Commit ID: 6fd567d
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

aws-cdk-automation · 2020-02-24T19:32:19Z

AWS CodeBuild CI Report

CodeBuild project: AutoBuildProject6AEA49D1-qxepHUsryhcu
Commit ID: b24ad8f
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

aws-cdk-automation · 2020-02-24T19:53:38Z

AWS CodeBuild CI Report

CodeBuild project: AutoBuildProject6AEA49D1-qxepHUsryhcu
Commit ID: 6e16a67
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

nija-at

Once batch has a more stable API, we can use the interfaces from batch instead of working with names and ARNs

Why wait? Why not start using them straight away?

packages/@aws-cdk/aws-stepfunctions/README.md

ayush987goyal · 2020-02-26T15:43:56Z

On the same note, I had a doubt on naming this task. I was confused between calling it InvokeBatchJob v/s RunBatchJob. What do you think should be the appropriate name?

aws-cdk-automation · 2020-02-26T19:01:03Z

AWS CodeBuild CI Report

CodeBuild project: AutoBuildProject6AEA49D1-qxepHUsryhcu
Commit ID: 74ab577
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

nija-at · 2020-02-27T10:55:35Z

On the same note, I had a doubt on naming this task. I was confused between calling it InvokeBatchJob v/s RunBatchJob. What do you think should be the appropriate name?

Let's switch to RunBatchJob.

Pull request has been modified.

aws-cdk-automation · 2020-02-27T15:05:30Z

AWS CodeBuild CI Report

CodeBuild project: AutoBuildProject6AEA49D1-qxepHUsryhcu
Commit ID: 508b3b9
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

aws-cdk-automation · 2020-02-27T16:30:57Z

AWS CodeBuild CI Report

CodeBuild project: AutoBuildProject6AEA49D1-qxepHUsryhcu
Commit ID: 1b322e7
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

packages/@aws-cdk/aws-stepfunctions-tasks/lib/run-batch-job.ts

packages/@aws-cdk/aws-stepfunctions-tasks/test/integ.run-batch-job.ts

Pull request has been modified.

aws-cdk-automation · 2020-02-28T19:40:55Z

AWS CodeBuild CI Report

CodeBuild project: AutoBuildProject6AEA49D1-qxepHUsryhcu
Commit ID: 156f794
Result: FAILED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

aws-cdk-automation · 2020-02-28T19:50:03Z

AWS CodeBuild CI Report

CodeBuild project: AutoBuildProject6AEA49D1-qxepHUsryhcu
Commit ID: 65bca62
Result: FAILED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

aws-cdk-automation · 2020-02-28T20:47:31Z

AWS CodeBuild CI Report

CodeBuild project: AutoBuildProject6AEA49D1-qxepHUsryhcu
Commit ID: bc15bd9
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

nija-at

Thanks for the updated revision. I have a few more comments based on a second iteration on this. It's starting to come together and looks close. 😊

nija-at · 2020-03-04T15:46:22Z

packages/@aws-cdk/aws-stepfunctions-tasks/lib/run-batch-job.ts

+        // Resource-level access control is not supported by Batch
+        // https://docs.aws.amazon.com/step-functions/latest/dg/batch-iam.html
+        new iam.PolicyStatement({
+          resources: ['*'],


I found this - https://docs.aws.amazon.com/batch/latest/userguide/batch-supported-iam-actions-resources.html.

Is it possible that AWS batch has added supported for resource-level permissions and that the step functions page is stale?

I am not sure how it would work out with the current batch implementation anyway. It does not expose the revision of a jobDefinition.
https://docs.aws.amazon.com/cdk/api/latest/docs/@aws-cdk_aws-batch.JobDefinition.html

I'm not sure I'm fully following your last comment.

I was thinking we could do -

resources: [ `${this.props.jobdefinition.jobDefinitionArn}/*` ]

This would be better than just *.

packages/@aws-cdk/aws-stepfunctions-tasks/lib/run-batch-job.ts

packages/@aws-cdk/aws-stepfunctions-tasks/test/integ.run-batch-job.ts

packages/@aws-cdk/aws-stepfunctions-tasks/lib/run-batch-job.ts

packages/@aws-cdk/aws-stepfunctions-tasks/test/integ.run-batch-job.ts

Pull request has been modified.

aws-cdk-automation · 2020-03-04T19:26:47Z

AWS CodeBuild CI Report

CodeBuild project: AutoBuildProject6AEA49D1-qxepHUsryhcu
Commit ID: a9f93aa
Result: FAILED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

nija-at

Mostly good. I'm still a bit worried about the * permission, which I think we can do better, but not sure.

Some minor code clean up for readability.

This is very close to the line. Thanks for iterating on this!

nija-at · 2020-03-09T17:58:53Z

packages/@aws-cdk/aws-stepfunctions-tasks/lib/run-batch-job.ts

+        // Resource-level access control is not supported by Batch
+        // https://docs.aws.amazon.com/step-functions/latest/dg/batch-iam.html
+        new iam.PolicyStatement({
+          resources: ['*'],


I'm not sure I'm fully following your last comment.

I was thinking we could do -

resources: [ `${this.props.jobdefinition.jobDefinitionArn}/*` ]

This would be better than just *.

packages/@aws-cdk/aws-stepfunctions-tasks/lib/run-batch-job.ts

packages/@aws-cdk/aws-stepfunctions-tasks/test/integ.run-batch-job.ts

ayush987goyal · 2020-03-10T03:50:01Z

Based on the permission configuration mentioned in this doc, we have to either provide the job-definition with the revision or none at all.

Job Definition

arn:aws:batch:region:account:job-definition/*

arn:aws:batch:region:account:job-definition/definition-name:revision

So based on this, I am going ahead with these permission instead of '*'

	new iam.PolicyStatement({
          resources: [
            Stack.of(_task).formatArn({
              service: 'batch',
              resource: 'job-definition',
              resourceName: '*'
            }),
            this.props.jobQueue.jobQueueArn
          ],
          actions: ['batch:SubmitJob']
        }),

That way we are limiting the permission to a specific queue but all the job-definitions. Once we have revision provided by Batch, we can probably update the permission to that specific revision.

Pull request has been modified.

aws-cdk-automation · 2020-03-10T05:26:14Z

AWS CodeBuild CI Report

CodeBuild project: AutoBuildProject6AEA49D1-qxepHUsryhcu
Commit ID: 407757a
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

nija-at · 2020-03-10T08:53:10Z

Based on the permission configuration mentioned in this doc, we have to either provide the job-definition with the revision or none at all.

Oops, yes you're correct. I mis-read the doc. This is fine to start off with.

mergify · 2020-03-10T08:56:08Z

Thank you for contributing! Your pull request will be updated from master and then merged automatically (do not update manually, and be sure to allow changes to be pushed to your fork).

aws-cdk-automation · 2020-03-10T09:59:53Z

AWS CodeBuild CI Report

CodeBuild project: AutoBuildProject6AEA49D1-qxepHUsryhcu
Commit ID: 22b3107
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

mergify · 2020-03-10T10:00:18Z

Thank you for contributing! Your pull request will be updated from master and then merged automatically (do not update manually, and be sure to allow changes to be pushed to your fork).

ayush987goyal requested a review from nija-at February 21, 2020 16:13

ayush987goyal mentioned this pull request Feb 21, 2020

Support all Step Function task resources #1314

Closed

7 tasks

SomayaB assigned nija-at Feb 21, 2020

nija-at changed the title ~~feat(sfn-tasks): add invoke-batch-job task~~ feat(stepfunctions-tasks): invoke batch job Feb 26, 2020

nija-at previously requested changes Feb 26, 2020

View reviewed changes

packages/@aws-cdk/aws-stepfunctions/README.md Outdated Show resolved Hide resolved

ayush987goyal requested a review from nija-at February 27, 2020 02:43

ayush987goyal changed the title ~~feat(stepfunctions-tasks): invoke batch job~~ feat(stepfunctions-tasks): run batch job Feb 27, 2020

nija-at previously requested changes Feb 28, 2020

View reviewed changes

ayush987goyal requested a review from nija-at February 28, 2020 19:22

nija-at previously requested changes Mar 4, 2020

View reviewed changes

ayush987goyal requested a review from nija-at March 4, 2020 19:09

nija-at previously requested changes Mar 9, 2020

View reviewed changes

ayush987goyal added 15 commits March 10, 2020 07:47

feat(sfn-tasks): add invoke-batch-job task

16d8278

choir(sfn): add documentation

e5c4a6e

feat(sfn-tasks): add all supported properties of batch

85c91c7

fix(sfn-tasks): make camelCase properties

3db78da

feat(sfn-tasks): use batch resources and add readme snippet

21244cc

choir(sfn-tasks): rename InvokeBatchJob to RunBatchJob

7491e38

choir(aws-snf): update readme

d07ccc7

fix(sfn-tasks): update public api and add env check

1ddef8c

fix(sfn-tasks): remove unused policy

2b03f30

choir: add file missed in last commit

9294cd6

fix: make environment ts index signature

19ed37b

fix(sfn-tasks): update the implementation

2ae13c4

fix: add validation for dependsOn

d51b8cb

choir: update the usage of JobQueue

0ba4399

choir: update readme

4680681

Update permission

407757a

ayush987goyal requested a review from nija-at March 10, 2020 04:45

nija-at approved these changes Mar 10, 2020

View reviewed changes

Merge branch 'master' into pr/invoke-batch

22b3107

mergify bot merged commit de1a36b into aws:master Mar 10, 2020

ayush987goyal deleted the pr/invoke-batch branch March 10, 2020 10:00

snyk-bot mentioned this pull request Apr 10, 2020

[Snyk] Upgrade aws-cdk from 1.28.0 to 1.30.0 ammobinDOTca/ammobin-cdk#2

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat(stepfunctions-tasks): run batch job #6396

feat(stepfunctions-tasks): run batch job #6396

ayush987goyal commented Feb 21, 2020 •

edited by nija-at

Loading

aws-cdk-automation commented Feb 21, 2020

aws-cdk-automation commented Feb 21, 2020

aws-cdk-automation commented Feb 22, 2020

aws-cdk-automation commented Feb 22, 2020

aws-cdk-automation commented Feb 22, 2020

aws-cdk-automation commented Feb 24, 2020

aws-cdk-automation commented Feb 24, 2020

nija-at left a comment

ayush987goyal commented Feb 26, 2020

aws-cdk-automation commented Feb 26, 2020

nija-at commented Feb 27, 2020

aws-cdk-automation commented Feb 27, 2020

aws-cdk-automation commented Feb 27, 2020

aws-cdk-automation commented Feb 28, 2020

aws-cdk-automation commented Feb 28, 2020

aws-cdk-automation commented Feb 28, 2020

nija-at left a comment

nija-at Mar 4, 2020

ayush987goyal Mar 4, 2020

nija-at Mar 9, 2020

aws-cdk-automation commented Mar 4, 2020

nija-at left a comment •

edited

Loading

nija-at Mar 9, 2020

ayush987goyal commented Mar 10, 2020

aws-cdk-automation commented Mar 10, 2020

nija-at commented Mar 10, 2020

mergify bot commented Mar 10, 2020

aws-cdk-automation commented Mar 10, 2020

mergify bot commented Mar 10, 2020

feat(stepfunctions-tasks): run batch job #6396

feat(stepfunctions-tasks): run batch job #6396

Conversation

ayush987goyal commented Feb 21, 2020 • edited by nija-at Loading

Commit Message

End Commit Message

aws-cdk-automation commented Feb 21, 2020

AWS CodeBuild CI Report

aws-cdk-automation commented Feb 21, 2020

AWS CodeBuild CI Report

aws-cdk-automation commented Feb 22, 2020

AWS CodeBuild CI Report

aws-cdk-automation commented Feb 22, 2020

AWS CodeBuild CI Report

aws-cdk-automation commented Feb 22, 2020

AWS CodeBuild CI Report

aws-cdk-automation commented Feb 24, 2020

AWS CodeBuild CI Report

aws-cdk-automation commented Feb 24, 2020

AWS CodeBuild CI Report

nija-at left a comment

Choose a reason for hiding this comment

ayush987goyal commented Feb 26, 2020

aws-cdk-automation commented Feb 26, 2020

AWS CodeBuild CI Report

nija-at commented Feb 27, 2020

aws-cdk-automation commented Feb 27, 2020

AWS CodeBuild CI Report

aws-cdk-automation commented Feb 27, 2020

AWS CodeBuild CI Report

aws-cdk-automation commented Feb 28, 2020

AWS CodeBuild CI Report

aws-cdk-automation commented Feb 28, 2020

AWS CodeBuild CI Report

aws-cdk-automation commented Feb 28, 2020

AWS CodeBuild CI Report

nija-at left a comment

Choose a reason for hiding this comment

nija-at Mar 4, 2020

Choose a reason for hiding this comment

ayush987goyal Mar 4, 2020

Choose a reason for hiding this comment

nija-at Mar 9, 2020

Choose a reason for hiding this comment

aws-cdk-automation commented Mar 4, 2020

AWS CodeBuild CI Report

nija-at left a comment • edited Loading

Choose a reason for hiding this comment

nija-at Mar 9, 2020

Choose a reason for hiding this comment

ayush987goyal commented Mar 10, 2020

aws-cdk-automation commented Mar 10, 2020

AWS CodeBuild CI Report

nija-at commented Mar 10, 2020

mergify bot commented Mar 10, 2020

aws-cdk-automation commented Mar 10, 2020

AWS CodeBuild CI Report

mergify bot commented Mar 10, 2020

ayush987goyal commented Feb 21, 2020 •

edited by nija-at

Loading

nija-at left a comment •

edited

Loading