Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

fix(executor): Optional input artifacts. Fixes #2990 #3019

Merged
merged 13 commits into from May 14, 2020
Merged

fix(executor): Optional input artifacts. Fixes #2990 #3019

merged 13 commits into from May 14, 2020
+243 −79

Conversation

alexec
Copy link
Contributor

@alexec alexec commented May 12, 2020 •
edited

Checklist:

  • Either (a) I've created an enhancement proposal and discussed it with the community, (b) this is a bug fix, or (c) this is a chore.
  • The title of the PR is (a) conventional, (b) states what changed, and (c) suffixes the related issues number. E.g. "fix(controller): Updates such and such. Fixes #1234".
  • I've signed the CLA.
  • I have written unit and/or e2e tests for my change. PRs without these are unlikely to be merged.
  • My builds are green. Try syncing with master if they are not.
  • My organization is added to USERS.md.

Changes

  • Each artifact repository that can output an artifact can now support optional input artifacts.
  • HTTP also - as this was easy to test.

Testing

Name Change How
Artifactory Translate 404 error Ask @jessesuen or community
GCS Bug fixes. Translate ErrObjectNotExists Ask author @whynowy
Git - N/A
HDFS Translate ErrNotExists Ask @sarabala1979 and @dtaniwaki
HTTP Translate exit code 22 Automated
OSS Bug fix Ask @merlintang
Raw - N/A
S3 Translate error Automated

Prototypes

https://bit.ly/argo-wf-protos

@alexec alexec linked an issue May 12, 2020 that may be closed by this pull request
Cannot use optional with input artifacts/S3 #2990
Closed
4 tasks
@alexec alexec changed the title fix: Allow optional input artifacts. Fixes #2990 fix(executor): Allow optional input artifacts. Fixes #2990 May 12, 2020
@alexec alexec marked this pull request as ready for review May 12, 2020 23:06
@alexec alexec requested a review from sarabala1979 May 12, 2020 23:06
@alexec alexec added this to the v2.10 milestone May 13, 2020
@alexec alexec changed the title fix(executor): Allow optional input artifacts. Fixes #2990 fix(executor): Optional input artifacts. Fixes #2990 May 13, 2020
@merlintang
Copy link
Contributor

/lgtm

@alexec alexec mentioned this pull request May 13, 2020
Merged
simster7
simster7 reviewed May 13, 2020
View reviewed changes
workflow/executor/executor.go
@@ -164,6 +164,10 @@ func (we *WorkflowExecutor) LoadArtifacts() error {
tempArtPath := artPath + ".tmp"
err = artDriver.Load(&art, tempArtPath)
if err != nil {
if art.Optional && errors.IsCode(errors.CodeNotFound, err) {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Currently Artifact.Optional means "not specified". With this we also enhance it to mean "not found" as well. Is this what we want or should we distinguish between the two?

Some users might expect different behaviors with each of these.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Lets discus at stand-up tomorrow, I'm having trouble getting my head around how "optional" could not be interpreted as "optional" - where as something specified in the spec can be inferred to be "not specified". It's like words have new meanings.

Copy link
Contributor Author

@alexec alexec May 13, 2020 •
edited

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more. 

// Make Artifacts optional, if Artifacts doesn't generate or exist
Optional bool `json:"optional,omitempty" protobuf:"varint,8,opt,name=optional"`

Hmmm....

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

On further thought I agree that the two meanings are essentially the same.

"Optional" should be taken from the point of view of the container that receives the artifact. The reason as to why it's not there should not matter.

@alexec alexec requested a review from simster7 May 13, 2020 22:39
@sonarcloud
Copy link

sonarcloud bot commented May 14, 2020

Kudos, SonarCloud Quality Gate passed!

Bug A 0 Bugs
Vulnerability A 0 Vulnerabilities (and Security Hotspot 0 Security Hotspots to review)
Code Smell A 1 Code Smell

55.8% 55.8% Coverage
0.0% 0.0% Duplication

dtaniwaki
dtaniwaki approved these changes May 14, 2020
View reviewed changes
Copy link
Member

@dtaniwaki dtaniwaki left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you!

workflow/artifacts/hdfs/hdfs.go
@@ -169,7 +169,14 @@ func (driver *ArtifactDriver) Load(inputArtifact *wfv1.Artifact, path string) er
}
}

return hdfscli.CopyToLocal(driver.Path, path)
err = hdfscli.CopyToLocal(driver.Path, path)
if err != nil {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

👍

@alexec alexec merged commit 202d4ab into argoproj:master May 14, 2020
@alexec alexec deleted the fix-2990 branch May 14, 2020 16:19
@alexec alexec mentioned this pull request Jul 16, 2020
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@simster7 simster7 simster7 approved these changes

@dtaniwaki dtaniwaki dtaniwaki approved these changes

@sarabala1979 sarabala1979 Awaiting requested review from sarabala1979

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
v2.10
Development

Successfully merging this pull request may close these issues.

Cannot use optional with input artifacts/S3
4 participants
@alexec @merlintang @simster7 @dtaniwaki