New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
feat(workflow): support for workflow definition files #3176
Conversation
1b17950
to
423aa0c
Compare
dd1a34e
to
0074aef
Compare
bc8c416
to
85264a6
Compare
c7522db
to
79daadd
Compare
a43d41e
to
4ef78fe
Compare
51e6d1d
to
2e11444
Compare
renku/core/util/os.py
Outdated
@@ -234,7 +241,7 @@ def hash_file(path: Union[Path, str], hash_type: str = "sha256") -> Optional[str | |||
return hash_file_descriptor(f, hash_type) | |||
|
|||
|
|||
def hash_str(content: str, hash_type: str = "sha256") -> str: | |||
def hash_string(content: str, hash_type: str = "md5") -> str: |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This isn't actually used anywhere? The function above it uses sha256 as well, and the docstring of this function still says sha256.
Was there a reason to change this?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I removed this function since we have it in the git history.
md5
is a bit faster than sha256
and since this is used just for hashing (and not cryptography) and we might use it a lot, I've replaced the hash function.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
but wouldn't that cause issues with existing hashes?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Right. I've used md5
for new hash functions and this is the only old one that I've replaced. It's not used anywhere so no harm done.
bccfb7d
to
adb7445
Compare
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thank you for addressing all the changes, looks great now! Only two minor suggestions plus the one mistake I made that you found 🙈
92341b1
to
e6f572f
Compare
bafb439
to
fbe852a
Compare
fbe852a
to
03892e9
Compare
Description
Workflow definition file pitch:
renku run workflow-file.yml
andrenku workflow show workflow-file.yml
.Every existing file that has a
.yml
or.yaml
extension and is not executable is considered to be a workflow file. Also, valid YAML files that contain a dictionary are considered to be a workflow file. We can also force running a workflow file by passing the--file
flag.Some Decisions
workflow-file.pre-processing
). It is calledqualified name
. This is done to make sure that workflow file steps will have a unique name which is required for the Renku workflow machinery to function correctly.renku status
works as intended.renku status
shows if a workflow file has changed and its generation. Note thatrenku update
doesn't re-run workflow files. Users need to run the workflow file to bring its generations up to date.WorkflowFileCompositePlan
; We stored the plan with all steps. We can later find out if a subset of steps were executed by looking at the number of activities that are stored in theWorkflowFileActivityCollection
.Demo Project
A demo project is available at https://gitlab.dev.renku.ch/mohammad.alisafaee/workflow-file-demo.git.