Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

feat(executor): Auto create s3 bucket if not present. Closes #3586 #4574

Merged
merged 27 commits into from Dec 3, 2020
Merged

feat(executor): Auto create s3 bucket if not present. Closes #3586 #4574

merged 27 commits into from Dec 3, 2020

Conversation

alexcapras
Copy link
Contributor

@alexcapras alexcapras commented Nov 20, 2020 •
edited by alexec

Checklist:

Closes #3586

@alexcapras alexcapras force-pushed the auto-create-s3bucket branch 2 times, most recently from a163e0d to 5c90d05 Compare November 20, 2020 14:37
@alexcapras alexcapras marked this pull request as ready for review November 20, 2020 14:38
@alexcapras alexcapras marked this pull request as draft November 20, 2020 14:38
@alexcapras alexcapras marked this pull request as ready for review November 20, 2020 14:50
@alexec
Copy link
Contributor

alexec commented Nov 20, 2020

Assigned to @jessesuen to approve proto changes.

@alexec
Copy link
Contributor

alexec commented Nov 20, 2020

@alexcapras can you please link to the issue and previous PR?

@alexcapras
Copy link
Contributor Author

@alexcapras can you please link to the issue and previous PR?

Issue: #3586
Previous PR: https://github.com/argoproj/argo/pull/4098/files

@alexcapras alexcapras changed the title feat(executor): Auto create s3 bucket if not present. feat(executor): Auto create s3 bucket if not present. Closes #3586 Nov 20, 2020
jessesuen
jessesuen approved these changes Dec 1, 2020
View reviewed changes
Copy link
Member

@jessesuen jessesuen left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

re-approving the proto changes.

@simster7 simster7 assigned alexec and unassigned jessesuen Dec 1, 2020
alexec
alexec requested changes Dec 2, 2020
View reviewed changes
Copy link
Contributor

@alexec alexec left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Changes requested.

@alexec
Copy link
Contributor

alexec commented Dec 2, 2020

Good work! You have some failing jobs that need remedy and you also need to sync with master and we are ready to merge!

@alexcapras alexcapras force-pushed the auto-create-s3bucket branch 3 times, most recently from 891126d to c4cf9f4 Compare December 2, 2020 21:16
Boolman and others added 9 commits December 2, 2020 22:09
@Boolman @alexcapras
fix(server): Add list sa and create secret to argo-server roles. …
25617a2 
…Closes argoproj#4526 (argoproj#4514)

Signed-off-by: boolman <boolman@gmail.com>
Signed-off-by: Alex Capras <alexcapras@gmail.com>
@arghya88 @alexcapras
feat(controller): make sso timeout configurable via cm (argoproj#4494)
deab963 
Signed-off-by: Arghya Sadhu <arghya88@gmail.com>
Signed-off-by: Alex Capras <alexcapras@gmail.com>
@arghya88 @alexcapras
fix(controller): support float for param value (argoproj#4490)
1a78713 
Signed-off-by: Arghya Sadhu <arghya88@gmail.com>
Signed-off-by: Alex Capras <alexcapras@gmail.com>
@simster7 @alexcapras
docs: Add "Argo Workflows in 5 min" link to README (argoproj#4533)
704b087 
Signed-off-by: Simon Behar <simbeh7@gmail.com>
Signed-off-by: Alex Capras <alexcapras@gmail.com>
@alexec @alexcapras
fix(ui): Do not auto-reload doc.location. Fixes argoproj#4530 (argopr…
aa232fc 
…oj#4535)

Signed-off-by: Alex Collins <alex_collins@intuit.com>
Signed-off-by: Alex Capras <alexcapras@gmail.com>
@sarabala1979 @alexcapras
fix(test): fix TestWFDefaultWithWFTAndWf flakiness (argoproj#4538)
4d94e07 
Signed-off-by: Saravanan Balasubramanian <sarabala1979@gmail.com>
Signed-off-by: Alex Capras <alexcapras@gmail.com>
@davidgibbons @alexcapras
fix: Missing arg lines caused files not to copy into containers (argo…
25bc7fa 
…proj#4542)

Signed-off-by: David Gibbons <david.gibbons@vevo.com>
Signed-off-by: Alex Capras <alexcapras@gmail.com>
@simster7 @alexcapras
fix: Validate metric key names (argoproj#4540)
5f7f4d0 
Signed-off-by: Simon Behar <simbeh7@gmail.com>
Signed-off-by: Alex Capras <alexcapras@gmail.com>
@alexec @alexcapras
fix(controller): Correct default port logic (argoproj#4547)
c6c3f3f 
Signed-off-by: Alex Collins <alex_collins@intuit.com>
Signed-off-by: Alex Capras <alexcapras@gmail.com>
L3Nerd and others added 6 commits December 2, 2020 22:09
@L3Nerd @alexcapras
fix: executor/pns containerid prefix fix (argoproj#4555)
958ca6c 
Signed-off-by: Lennart Kindermann <lennart.kindermann@cloudseeds.de>
Signed-off-by: Alex Capras <alexcapras@gmail.com>
@terrytangyuan @alexcapras
chore: Remove unused image build and push hooks (argoproj#4539)
c37450b 
Signed-off-by: terrytangyuan <terrytangyuan@gmail.com>
Signed-off-by: Alex Capras <alexcapras@gmail.com>
@secretlifeof @alexcapras
docs: Clean-up examples. Fixes argoproj#4124 (argoproj#4128)
1a0cebb 
Signed-off-by: github@finnesand.no <github@finnesand.no>

feat(ui): Add Template/Cron workflow filter to workflow page. Closes argoproj#4532 (argoproj#4543)

Signed-off-by: Tianchu Zhao <evantczhao@gmail.com>

feat(executor): Auto create s3 bucket if not present.

Signed-off-by: Alex Capras <alexcapras@gmail.com>

Apply codegen

Signed-off-by: Alex Capras <alexcapras@gmail.com>

Add argo-e2e label to test wf

Signed-off-by: Alex Capras <alexcapras@gmail.com>

chore: Updated stress test YAML (argoproj#4569)

Signed-off-by: Alex Collins <alex_collins@intuit.com>

docs: Updated kubectl apply command in manifests README (argoproj#4577)

Signed-off-by: Stefan Gloutnikov <stefan@gloutnikov.com>

feat(controller): Make MAX_OPERATION_TIME configurable. Close argoproj#4239 (argoproj#4562)

Signed-off-by: Alex Collins <alex_collins@intuit.com>

docs: Fix a typo in example (argoproj#4590)

Signed-off-by: Takayoshi Nishida <takayoshi.nishida@gmail.com>

feat(controller): Retry transient offload errors. Resolves argoproj#4464 (argoproj#4482)

Signed-off-by: Alex Collins <alex_collins@intuit.com>

fix(server): use the correct name when downloading artifacts (argoproj#4579)

Signed-off-by: Daniel Herman <dherman@factset.com>

fix(server): serve artifacts directly from disk to support large artifacts (argoproj#4589)

Signed-off-by: Daniel Herman <dherman@factset.com>

fix(executor): Handle sidecar killing in a process-namespace-shared pod (argoproj#4575)

Signed-off-by: Daisuke Taniwaki <daisuketaniwaki@gmail.com>

docs: Add JSON schema for IDE validation (argoproj#4581)

Signed-off-by: Paul Brabban <paul.brabban@gmail.com>

refactor: Use polling model for workflow phase metric (argoproj#4557)

Signed-off-by: Simon Behar <simbeh7@gmail.com>

Addressing reviewers comments

Signed-off-by: Alex Capras <alexcapras@gmail.com>

Addressing reviewers comments

docs: Minor typo fix (argoproj#4610)

Signed-off-by: Paavo Pokkinen <paavo.pokkinen@vaimo.com>

fix(controller): Prevent tasks with names starting with digit to use either 'depends' or 'dependencies' (argoproj#4598)

Signed-off-by: terrytangyuan <terrytangyuan@gmail.com>

fix(docs): Bring minio chart instructions up to date (argoproj#4586)

Signed-off-by: Ranga Krishnan <ranga@bei.re>

fix(executor): Fixed waitMainContainerStart returning prematurely. Closes argoproj#4599 (argoproj#4601)

Signed-off-by: fsiegmund <siegmund@slb.com>

feat(controller): Enhanced artifact repository ref. See argoproj#3184 (argoproj#4458)

Signed-off-by: Alex Collins <alex_collins@intuit.com>

fix: Null check pagination variable (argoproj#4617)

Signed-off-by: Simon Behar <simbeh7@gmail.com>

fix: Perform fields filtering server side (argoproj#4595)

Signed-off-by: Simon Behar <simbeh7@gmail.com>

fix(server): Correct webhook event payload marshalling. Fixes argoproj#4572 (argoproj#4594)

Signed-off-by: Alex Collins <alex_collins@intuit.com>

feat(ui): Add columns--narrower-height to AttributeRow (argoproj#4371)

fix: Fix TestCleanFieldsExclude (argoproj#4625)

Signed-off-by: Simon Behar <simbeh7@gmail.com>

fix(argo-server): fix global variable validation error with reversed dag.tasks (argoproj#4369)

Signed-off-by: chenyu.zheng <chenyu.zheng@hulu.com>

fix: derive jsonschema and fix up issues, validate examples dir… (argoproj#4611)

Signed-off-by: Paul Brabban <paul.brabban@gmail.com>

fix(ui): Reference secrets in EnvVars. Fixes argoproj#3973  (argoproj#4419)

Signed-off-by: Alejandro Tejera <aletepe@gmail.com>

fix(ui): Fix Snyk issues (argoproj#4631)

Signed-off-by: Alex Collins <alex_collins@intuit.com>

feat(executor): More informative log when executors do not support output param from base image layer (argoproj#4620)

Signed-off-by: terrytangyuan <terrytangyuan@gmail.com>

Codegen patch. Signed off by alexcapras@gmail.com

Codegen patch. Signed off by alexcapras@gmail.com

Delete test.patch

Signed-off-by: Alex Capras <alexcapras@gmail.com>
@alexcapras
Codegen patch. Signed off by alexcapras@gmail.com
35db05b 
Signed-off-by: Alex Capras <alexcapras@gmail.com>
@alexcapras
Codegen patch. Signed off by alexcapras@gmail.com
61dfcac 
Signed-off-by: Alex Capras <alexcapras@gmail.com>
@alexec @alexcapras
Delete test.patch
480a06e 
Signed-off-by: Alex Capras <alexcapras@gmail.com>
@alexcapras alexcapras marked this pull request as draft December 2, 2020 22:25
@alexcapras alexcapras marked this pull request as ready for review December 2, 2020 22:47
@alexcapras
Remove disabled example
542f7c7 
Signed-off-by: Alex Capras <alexcapras@gmail.com>
@alexcapras
Use argoproj/argosay:v2
b66ea22 
Signed-off-by: Alex Capras <alexcapras@gmail.com>
@alexcapras
Dont fail if bucket already exists
588e316 
Signed-off-by: Alex Capras <alexcapras@gmail.com>
@alexec
Copy link
Contributor

alexec commented Dec 3, 2020

TestOutputArtifactS3BucketCreationEnabled failed?

@alexcapras
Removing error handling logic if makeBucket call fails
289548e 
Signed-off-by: Alex Capras <alexcapras@gmail.com>
@alexcapras
Copy link
Contributor Author

alexcapras commented Dec 3, 2020 •
edited

TestOutputArtifactS3BucketCreationEnabled failed?

@alexec Addressed this in my latest commit (289548e). Only drawback here is that if the user has enabled createBucketIfNotPresent and the MakeBucket call fails for a reason (other than the bucket already exists, e.g. insufficient perms), we will try to create it again in the ExponentialBackoff loop. (Because the bucketCreation will fail but we will still try to Put the artifact).

If this is not an issue then i think it should be ok now

@alexcapras
Fix linting
d90f2f6 
Signed-off-by: Alex Capras <alexcapras@gmail.com>
@alexcapras
Rever to argosay:v1
507347b 
Signed-off-by: Alex Capras <alexcapras@gmail.com>
@alexcapras alexcapras marked this pull request as draft December 3, 2020 00:59
@alexcapras
Fix test/linting issues
2f8a511 
Signed-off-by: Alex Capras <alexcapras@gmail.com>
@alexcapras alexcapras marked this pull request as ready for review December 3, 2020 01:27
@alexec alexec merged commit 94be7da into argoproj:master Dec 3, 2020
@alexec alexec added this to the v3.0 milestone Dec 3, 2020
@alexec
Copy link
Contributor

alexec commented Dec 3, 2020

Merged. This is currently targeted for v3.0. That'll be some time away, maybe Jan. I'll look into backporting it to v2.12

@alexcapras alexcapras deleted the auto-create-s3bucket branch December 3, 2020 17:16
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@alexec alexec alexec approved these changes

@jessesuen jessesuen jessesuen approved these changes

Assignees

@alexec alexec

Labels
None yet
Projects
None yet
Milestone
v3.0
Development

Successfully merging this pull request may close these issues.

Create default S3 bucket if not present