Add FSx for Lustre AutoImport support #158

gkao123 · 2020-10-01T18:45:39Z

Is this a bug fix or adding new feature?
This pull request adds support for AutoImport, a new FSxL feature. For more details, please view: https://docs.aws.amazon.com/fsx/latest/LustreGuide/autoimport-data-repo.html

What testing is done?
End to end testing

k8s-ci-robot · 2020-10-01T18:45:42Z

Thanks for your pull request. Before we can look at your pull request, you'll need to sign a Contributor License Agreement (CLA).

📝 Please follow instructions at https://git.k8s.io/community/CLA.md#the-contributor-license-agreement to sign the CLA.

It may take a couple minutes for the CLA signature to be fully registered; after that, please reply here with a new comment and we'll verify. Thanks.

If you've already signed a CLA, it's possible we don't have your GitHub username or you're using a different email address. Check your existing CLA data and verify that your email is set on your git commits.
If you signed the CLA as a corporation, please sign in with your organization's credentials at https://identity.linuxfoundation.org/projects/cncf to be authorized.
If you have done the above and are still having issues with the CLA being reported as unsigned, please log a ticket with the Linux Foundation Helpdesk: https://support.linuxfoundation.org/
Should you encounter any issues with the Linux Foundation Helpdesk, send a message to the backup e-mail support address at: login-issues@jira.linuxfoundation.org

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

k8s-ci-robot · 2020-10-01T18:45:47Z

Welcome @gkao123!

It looks like this is your first PR to kubernetes-sigs/aws-fsx-csi-driver 🎉. Please refer to our pull request process documentation to help your PR have a smooth ride to approval.

You will be prompted by a bot to use commands during the review process. Do not be afraid to follow the prompts! It is okay to experiment. Here is the bot commands documentation.

You can also check if kubernetes-sigs/aws-fsx-csi-driver has its own contribution guidelines.

You may want to refer to our testing guide if you run into trouble with your tests not passing.

If you are having difficulty getting your pull request seen, please follow the recommended escalation practices. Also, for tips and tricks in the contribution process you may want to read the Kubernetes contributor cheat sheet. We want to make sure your contribution gets all the attention it needs!

Thank you, and welcome to Kubernetes. 😃

k8s-ci-robot · 2020-10-01T18:45:47Z

Hi @gkao123. Thanks for your PR.

I'm waiting for a kubernetes-sigs member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should join the org to skip this step.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

wongma7 · 2020-10-06T11:26:52Z

/ok-to-test

wongma7 · 2020-10-06T11:30:10Z

examples/kubernetes/dynamic_provisioning_s3/README.md

  s3ImportPath: s3://ml-training-data-000
  s3ExportPath: s3://ml-training-data-000/export
  deploymentType: SCRATCH_2
 ```
 * subnetId - the subnet ID that the FSx for Lustre filesystem should be created inside.
 * securityGroupIds - a common separated list of security group IDs that should be attached to the filesystem.
+* autoImportPolicy - the policy FSx will follow that determines how the filesystem is automatically updated with changes made in the linked data repository. For list of acceptable policies, please view the official FSx for Lustre documentation


I tried looking for the documentation myself but couldn't find it. I found https://docs.aws.amazon.com/fsx/latest/LustreGuide/autoimport-data-repo.html but not the strings that each policy corresponds to. Is it OK to link to documentation here or is it a moving target?

I can link to this documentation: https://docs.aws.amazon.com/fsx/latest/APIReference/API_CreateFileSystemLustreConfiguration.html!

Yeah that would be perfect thanks

wongma7 · 2020-10-06T11:30:51Z

lgtm I just have one nitpick about the doc on what values are valid because I couldn't find it myself ...

/approve

wongma7 · 2020-10-06T11:32:34Z

tests/e2e/dynamic_provisioning_test.go

@@ -158,6 +158,7 @@ var _ = Describe("[fsx-csi-e2e] Dynamic Provisioning with s3 data repository", f
 						Parameters: map[string]string{
 							"subnetId":         subnetId,
 							"securityGroupIds": strings.Join(securityGroupIds, ","),
+							"autoImportPolicy": "NEW_CHANGED"


think a comma is missing here.

Also might need to run gofmt, the "verify" test complained about it

wongma7 · 2020-10-08T23:11:28Z

go.mod

@@ -1,7 +1,7 @@
 module github.com/kubernetes-sigs/aws-fsx-csi-driver

 require (
-	github.com/aws/aws-sdk-go v1.29.9
+	github.com/aws/aws-sdk-go v1.35.6


try also to run 'go mod tidy', i think go.sum should have been updated too

Ran go mod tidy and updated the pr with go.sum. I also verified go make still works.

wongma7 · 2020-10-08T23:24:59Z

/lgtm
/approve

k8s-ci-robot · 2020-10-08T23:25:08Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: gkao123, wongma7

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~OWNERS~~ [wongma7]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

gkao123 · 2020-10-08T23:52:51Z

Hi @wongma7, the e2e tests seem to have failed due to the following error:

failed to created filesystem CreateFileSystem failed: ServiceLimitExceeded: Account 607362164682 can have at most 100800 GiB of total Lustre storage across file systems. Please contact AWS support to request a limit increase for this account.

This seems unrelated to my code change. How should we resolve this error?

wongma7 · 2020-10-08T23:58:31Z

I'm cleaning up the test account, some old file systems are still in Deleting state, I'll retest when they're deleted

wongma7 · 2020-10-08T23:59:11Z

/retest

should be enough room already

gkao123 · 2020-10-09T00:00:40Z

I'm cleaning up the test account, some old file systems are still in Deleting state, I'll retest when they're deleted

Thank you!

wongma7 · 2020-10-09T02:05:41Z

One of the test failed, not sure if it's a flake because it does involve importing from S3: "Dynamic Provisioning with s3 data repository should create a volume on demand with s3 as data repository"
https://prow.k8s.io/view/gs/kubernetes-jenkins/pr-logs/pull/kubernetes-sigs_aws-fsx-csi-driver/158/pull-aws-fsx-csi-driver-e2e/1314354691160674304

/retest

gkao123 · 2020-10-11T20:58:06Z

/retest

gkao123 · 2020-10-11T21:36:43Z

@wongma7 I noticed in https://prow.k8s.io/view/gs/kubernetes-jenkins/pr-logs/pull/kubernetes-sigs_aws-fsx-csi-driver/158/pull-aws-fsx-csi-driver-e2e/1314354691160674304 that the resources are not being successfully deleted after the test is completed. Do you know if this could cause the test failure?

gkao123 · 2020-10-11T21:43:57Z

/retest

tests/e2e/dynamic_provisioning_test.go

wongma7 · 2020-10-12T23:43:36Z

/lgtm
tests shouldn't affect each other, but we do reuse the same fsx volume for all tests since it would take too long to create one for each test case
I guess the failure was a flake.

k8s-ci-robot added cncf-cla: no Indicates the PR's author has not signed the CNCF CLA. needs-ok-to-test Indicates a PR that requires an org member to verify it is safe to test. labels Oct 1, 2020

k8s-ci-robot requested review from d-nishi and wongma7 October 1, 2020 18:45

k8s-ci-robot added the size/S Denotes a PR that changes 10-29 lines, ignoring generated files. label Oct 1, 2020

gkao123 force-pushed the fsx_autoimport_pr branch from 5985697 to a932e67 Compare October 1, 2020 18:51

k8s-ci-robot added cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. and removed cncf-cla: no Indicates the PR's author has not signed the CNCF CLA. labels Oct 1, 2020

gkao123 force-pushed the fsx_autoimport_pr branch from a932e67 to 93ac0ff Compare October 1, 2020 19:07

k8s-ci-robot added ok-to-test Indicates a non-member PR verified by an org member that is safe to test. and removed needs-ok-to-test Indicates a PR that requires an org member to verify it is safe to test. labels Oct 6, 2020

wongma7 reviewed Oct 6, 2020

View reviewed changes

k8s-ci-robot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Oct 6, 2020

wongma7 reviewed Oct 6, 2020

View reviewed changes

gkao123 force-pushed the fsx_autoimport_pr branch from 93ac0ff to 8c82a3d Compare October 8, 2020 23:04

k8s-ci-robot added size/M Denotes a PR that changes 30-99 lines, ignoring generated files. and removed size/S Denotes a PR that changes 10-29 lines, ignoring generated files. labels Oct 8, 2020

wongma7 reviewed Oct 8, 2020

View reviewed changes

gkao123 force-pushed the fsx_autoimport_pr branch from 8c82a3d to c659501 Compare October 8, 2020 23:13

k8s-ci-robot assigned wongma7 Oct 8, 2020

k8s-ci-robot added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Oct 8, 2020

gkao123 force-pushed the fsx_autoimport_pr branch from c659501 to 1f43285 Compare October 11, 2020 22:32

k8s-ci-robot removed the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Oct 11, 2020

Add FSx for Lustre AutoImport support

44cc44b

gkao123 force-pushed the fsx_autoimport_pr branch from 1f43285 to 44cc44b Compare October 11, 2020 23:25

wongma7 reviewed Oct 12, 2020

View reviewed changes

tests/e2e/dynamic_provisioning_test.go Show resolved Hide resolved

k8s-ci-robot added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Oct 12, 2020

k8s-ci-robot merged commit 6b66b59 into kubernetes-sigs:master Oct 12, 2020

gkao123 deleted the fsx_autoimport_pr branch October 12, 2020 23:45

philxiao mentioned this pull request Aug 17, 2021

Update OWNER file to include myself kubernetes/community#5952

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add FSx for Lustre AutoImport support #158

Add FSx for Lustre AutoImport support #158

gkao123 commented Oct 1, 2020

k8s-ci-robot commented Oct 1, 2020

k8s-ci-robot commented Oct 1, 2020

k8s-ci-robot commented Oct 1, 2020

wongma7 commented Oct 6, 2020

wongma7 Oct 6, 2020

gkao123 Oct 8, 2020

wongma7 Oct 8, 2020

wongma7 commented Oct 6, 2020

wongma7 Oct 6, 2020

wongma7 Oct 8, 2020

gkao123 Oct 8, 2020

wongma7 commented Oct 8, 2020

k8s-ci-robot commented Oct 8, 2020

gkao123 commented Oct 8, 2020

wongma7 commented Oct 8, 2020

wongma7 commented Oct 8, 2020

gkao123 commented Oct 9, 2020

wongma7 commented Oct 9, 2020

gkao123 commented Oct 11, 2020

gkao123 commented Oct 11, 2020

gkao123 commented Oct 11, 2020

wongma7 commented Oct 12, 2020

Add FSx for Lustre AutoImport support #158

Add FSx for Lustre AutoImport support #158

Conversation

gkao123 commented Oct 1, 2020

k8s-ci-robot commented Oct 1, 2020

k8s-ci-robot commented Oct 1, 2020

k8s-ci-robot commented Oct 1, 2020

wongma7 commented Oct 6, 2020

wongma7 Oct 6, 2020

Choose a reason for hiding this comment

gkao123 Oct 8, 2020

Choose a reason for hiding this comment

wongma7 Oct 8, 2020

Choose a reason for hiding this comment

wongma7 commented Oct 6, 2020

wongma7 Oct 6, 2020

Choose a reason for hiding this comment

wongma7 Oct 8, 2020

Choose a reason for hiding this comment

gkao123 Oct 8, 2020

Choose a reason for hiding this comment

wongma7 commented Oct 8, 2020

k8s-ci-robot commented Oct 8, 2020

gkao123 commented Oct 8, 2020

wongma7 commented Oct 8, 2020

wongma7 commented Oct 8, 2020

gkao123 commented Oct 9, 2020

wongma7 commented Oct 9, 2020

gkao123 commented Oct 11, 2020

gkao123 commented Oct 11, 2020

gkao123 commented Oct 11, 2020

wongma7 commented Oct 12, 2020