feat(stepfunctions-tasks): add EKS call to SFN-tasks #12779

NovakGu · 2021-01-29T17:33:05Z

Taking over the ownership of original PR #11738

feat(stepfunctions-tasks): support for EKS Call

Implementation

Update package @aws-cdk/aws-stepfunctions-tasks to include support for EKS Call

API as per documentation here:
https://docs.aws.amazon.com/step-functions/latest/dg/connect-eks.html

Includes support for the following Amazon EKS API calls:
eks:call

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache-2.0 license

gitpod-io · 2021-01-29T17:33:08Z

mergify · 2021-01-29T17:33:43Z

Title does not follow the guidelines of Conventional Commits. Please adjust title before merge.

daisuke-yoshimoto · 2021-01-31T22:01:10Z

@shivlaks
Would you please review this PR?

shivlaks · 2021-02-01T17:13:06Z

packages/@aws-cdk/aws-stepfunctions-tasks/lib/eks/call.ts

+  /**
+   * Name of the cluster
+   */
+  readonly clusterName: string;


why not take in an ICluster and take properties out of it?
question: are step functions executions of EKS restricted to the account in which the state machine resides?

this is available as clusterName in an ICluster

Will use EKS L2 Construct instead in next commit.

shivlaks · 2021-02-01T17:13:29Z

packages/@aws-cdk/aws-stepfunctions-tasks/lib/eks/call.ts

+  /**
+   * API endpoint to communicate with your cluster
+   */
+  readonly endpoint: string;


this is clusterEndpoint from an ICluster

Will use EKS L2 Construct instead in next commit.

shivlaks · 2021-02-01T17:14:18Z

packages/@aws-cdk/aws-stepfunctions-tasks/lib/eks/call.ts

+  /**
+   * Base 64 encoded certificate data required to communicate with your cluster
+   */
+  readonly certificateAuthority: string;


this is available as clusterCertificateAuthorityData from an ICluster

Will use EKS L2 Construct instead in next commit.

shivlaks · 2021-02-01T17:17:30Z

packages/@aws-cdk/aws-stepfunctions-tasks/lib/eks/call.ts

+/**
+ * Method type of a EKS call
+ */
+export enum MethodType {


choose a more specific name. It doesn't quite communicate the intent of the enum... maybe something like HttpMethods or some appropriate qualifier that is less general from a naming perspective

Will Change to HttpMethods in next commit.

packages/@aws-cdk/aws-stepfunctions-tasks/lib/index.ts

shivlaks · 2021-02-01T17:18:22Z

packages/@aws-cdk/aws-stepfunctions-tasks/test/eks/call.test.ts

+import * as cdk from '@aws-cdk/core';
+import { EksCall, MethodType } from '../../lib/eks/call';
+
+describe('Call an EKS endpoint', () => {


missing tests for invalid service integration pattern (i.e. an error is thrown if RUN_JOB is selected)

Will add tests for run a job pattern in next commit.

shivlaks · 2021-02-01T17:21:06Z

packages/@aws-cdk/aws-stepfunctions-tasks/test/eks/integ.call.ts

+
+const app = new cdk.App();
+const stack = new cdk.Stack(app, 'aws-stepfunctions-tasks-eks-call-integ');


this test is not self contained. Please include verification steps, but more importantly create the EKS cluster as a part of the integration test.

The test should create a state machine that can be executed successfully. Right now it has a dependency that an EKS job already exists called clusterName, but this is not self-contained.

Verification steps should look something like this

Modified integ test in the latest commit

Pull request has been modified.

NovakGu · 2021-02-04T02:49:41Z

Commit: b831188

Description:

Refactor call.ts to use EKS L2 Construct
Add more coverage test for call.ts
Modified integ test for call.ts, created EKS cluster as part of the test
Addressed other comments from previous commit
Rebase local branch with master

Testing:

Deploy integ test to personal account, manually verified the state machine created can be executed and finish with SUCCEEDED
Run Coverage test locally with npm test
yarn integ eks/integ.call.js

…ected.json

daisuke-yoshimoto · 2021-02-06T06:30:03Z

@shivlaks

It looks like the fix is complete. Would you please re-review it?

shivlaks

getting closer! - would also love a review/input from @iliapolo who is the maintainer of the eks module in the cdk

shivlaks · 2021-02-09T17:46:52Z

packages/@aws-cdk/aws-stepfunctions-tasks/lib/eks/call.ts

+  ];
+
+  protected readonly taskMetrics?: sfn.TaskMetricsConfig;
+  protected readonly taskPolicies?: iam.PolicyStatement[];


probably worth a comment why task policies are not being set in this integration

Added a comment in latest commit.

shivlaks · 2021-02-09T17:48:17Z

packages/@aws-cdk/aws-stepfunctions-tasks/test/eks/call.test.ts

+    requestBody: sfn.TaskInput.fromObject({
+      RequestBody: 'requestBody',
+    }),


should the type of requestBody be sfn.TaskInput ? it's currently not configured in that way.

Yes, that's a good point. Changed the requestBody to be sfn.TaskInput

shivlaks · 2021-02-09T17:50:21Z

packages/@aws-cdk/aws-stepfunctions-tasks/lib/eks/call.ts

+  readonly httpPath: string;
+
+  /**
+   * Query Parameters part of HTTP request


nit: would be helpful to include doc links and or purpose of query parameters. the current doc strings don't add much information from the parameter names

Query parameters would look something like this: https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.19/#-strong-write-operations-cronjob-v1beta1-batch-strong- I'm not sure if it would be helpful to include an example such as this

gotcha, I agree. maybe using it in the readme example would be helpful for users though.

packages/@aws-cdk/aws-stepfunctions-tasks/lib/eks/call.ts

shivlaks · 2021-02-09T17:57:30Z

packages/@aws-cdk/aws-stepfunctions-tasks/lib/eks/call.ts

+   * Request body part of HTTP request
+   * @default - No request body
+   */
+  readonly requestBody?: { [key: string]: any };


this does not support paths: i.e. how could you supply $.blah
please change the type to sfn.TaskInput and also add a test for it.

the same applies to any parameters that support state input json path.

Changed to sfn.TaskInput and added more coverage tests

Pull request has been modified.

NovakGu · 2021-02-09T22:19:20Z

Updating branch to re-trigger build

iliapolo · 2021-02-17T21:53:58Z

packages/@aws-cdk/aws-stepfunctions-tasks/lib/eks/call.ts

+        CertificateAuthority: this.props.cluster.clusterCertificateAuthorityData,
+        Endpoint: this.props.cluster.clusterEndpoint,


Are these parameters required for successfully making the call? Note that if ICluster is imported, these properties might not be defined, and will throw an exception.

I would extract these and do:

try { const ca = this.props.cluster.clusterCertificateAuthorityData; } catch (err) { // if its required, throw an error here with the right context // if its not required, just pass }

Thanks for the feedback, I'll add validation for ClusterName, ClusterEndpoint and clusterCertificateAuthorityData, as well as coverage tests for these cases

With some testing I think the error thrown from the eks module has the valid context in our case.

for eks:call I'm extracting clusterName, clusterEndpoint, and clusterCertificateAuthorityData from ICluster, for example if clusterEndpoint is not defined, it will throw exception with message "clusterEndpoint" is not defined for this imported cluster, which is sufficient to show the context of the error.

All three are required to make the call, I'll just add coverage test to reflect this case

@shivlaks Do you think "clusterEndpoint" is not defined for this imported cluster is sufficient or more details needs to be added such as Cluster Endpoint is not defined for this imported cluster, this field is required for eks:call task.

@NovakGu that seems sensible. let's keep it simple. I generally like the format of <validation error><remediation (if any)>

i.e.
The clusterEndpoint property must be specified when using an imported Cluster.

@shivlaks @iliapolo
Added validation for clusterEndpoint and clusterCertificateAuthorityData.
clusterName is required when import a cluster or creating a new cluster so I'm not double checking it here.

After changes

if an given cluster does not have cluster Endpoint it will throw with error
The "clusterEndpoint" property must be specified when using an imported Cluster.

If an given cluster does not have certAuth it will throw with error
The "clusterCertificateAuthorityData" property must be specified when using an imported Cluster.

…ests

shivlaks

LGTM. added a do-not-merge so @iliapolo can also take a look in case he had any additional feedback.

iliapolo · 2021-02-21T14:11:03Z

LGTM

daisuke-yoshimoto · 2021-02-22T04:36:26Z

@NovakGu @shivlaks @iliapolo
Is this PR not yet merged?

Sumeet Badyal and others added 2 commits January 28, 2021 22:17

feat(stepfunctions-tasks): add EKS call to SFN-tasks

ea41033

feat(stepfunctions-tasks): addressed comments from previous commit

d6d55e5

NovakGu changed the title ~~Guchangh/eks call~~ feat(stepfunctions-tasks): add EKS call to SFN-tasks Jan 29, 2021

github-actions bot assigned shivlaks Jan 29, 2021

github-actions bot added the @aws-cdk/aws-stepfunctions-tasks label Jan 29, 2021

NovakGu marked this pull request as ready for review January 29, 2021 18:11

NovakGu mentioned this pull request Jan 29, 2021

feat(stepfunctions-tasks): add EKS call to SFN-tasks #11738

Closed

shivlaks previously requested changes Feb 1, 2021

View reviewed changes

Merge branch 'master' into guchangh/eks-call

fb639aa

NovakGu added 2 commits February 3, 2021 18:43

Add coverage/integ test & refactor using EKS L2 Construct

b831188

Merge branch 'master' into guchangh/eks-call

26150b4

Fix typo in README and failed build due to execution role name in exp…

db84626

…ected.json

shivlaks previously requested changes Feb 9, 2021

View reviewed changes

Address comments and nits

6f545ed

Merge branch 'master' into guchangh/eks-call

5796a70

shivlaks requested a review from iliapolo February 10, 2021 16:33

iliapolo reviewed Feb 17, 2021

View reviewed changes

add validation for clusterEndpoint and clusterCertAuth and coverage t…

1130675

…ests

shivlaks added the pr/do-not-merge This PR should not be merged at this time. label Feb 18, 2021

shivlaks previously approved these changes Feb 18, 2021

View reviewed changes


		const app = new cdk.App();
		const stack = new cdk.Stack(app, 'aws-stepfunctions-tasks-eks-call-integ');

		CertificateAuthority: this.props.cluster.clusterCertificateAuthorityData,
		Endpoint: this.props.cluster.clusterEndpoint,

feat(stepfunctions-tasks): add EKS call to SFN-tasks #12779

feat(stepfunctions-tasks): add EKS call to SFN-tasks #12779

Conversation

NovakGu commented Jan 29, 2021 • edited Loading

Includes support for the following Amazon EKS API calls: eks:call

gitpod-io bot commented Jan 29, 2021 • edited Loading

mergify bot commented Jan 29, 2021

daisuke-yoshimoto commented Jan 31, 2021

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

NovakGu commented Feb 4, 2021 • edited Loading

daisuke-yoshimoto commented Feb 6, 2021

shivlaks left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

NovakGu commented Feb 9, 2021

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

NovakGu Feb 18, 2021 • edited Loading

Choose a reason for hiding this comment

shivlaks left a comment

Choose a reason for hiding this comment

iliapolo commented Feb 21, 2021

daisuke-yoshimoto commented Feb 22, 2021

NovakGu commented Jan 29, 2021 •

edited

Loading

Includes support for the following Amazon EKS API calls:
eks:call

gitpod-io bot commented Jan 29, 2021 •

edited

Loading

NovakGu commented Feb 4, 2021 •

edited

Loading

NovakGu Feb 18, 2021 •

edited

Loading