Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Implement conditional workflow steps #14846

Merged
merged 36 commits into from
Jan 19, 2023

Conversation

mvdbeek
Copy link
Member

@mvdbeek mvdbeek commented Oct 24, 2022

This PR implements executing tool and subworkflow steps conditionally, based on the result of a javascript expression. The workflow editor will for now only allow the simplest expression, which is a connection to a boolean parameter (from an expression tool or a workflow input).
An example gxformat2 workflow looks like this:

class: GalaxyWorkflow
inputs:
  should_run:
    type: boolean
  some_file:
    type: data
steps:
  cat1:
    tool_id: cat1
    in:
      input1: some_file
      should_run: should_run
    when: $(inputs.should_run)

If when evaluates to true the step will run, if it evaluates to false it will not run.
This is also implemented for mapped over tool and workflow steps; if an output is skipped it will be produce a null value. You can think of this as an optional output. That means you can use the pickValue expression tool to pick for instance the first created output for two mutually exclusive processing steps.

Includes a whole new infrastructure for displaying scheduling warnings, failures and cancellations:

Screenshot 2023-01-16 at 22 23 19

Remaining work:

  • Update the workflow editor
  • Some visual indicator for partial collections in the history ?

How to test the changes?

(Select all options that apply)

  • I've included appropriate automated tests.
  • This is a refactoring of components with existing test coverage.
  • Instructions for manual testing are as follows:
    1. [add testing steps and prerequisites here if you didn't write automated tests covering all your changes]

License

  • I agree to license these and all my past contributions to the core galaxy codebase under the MIT license.

@mvdbeek
Copy link
Member Author

mvdbeek commented Nov 18, 2022

I think the backend is ready for a review ... I would probably split out the UI work into another PR (I should wrap up the first part of the workflow editor overhaul, that would make this easier).

@mvdbeek mvdbeek marked this pull request as ready for review November 18, 2022 09:43
@github-actions github-actions bot added this to the 23.0 milestone Nov 18, 2022
@dannon dannon requested a review from jmchilton December 8, 2022 15:32

hda_references = []

def to_cwl(value):
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is so complicated and we're only basically testing boolean parameters right?

What if an integer is sent - what happens? A dataset? Etc...

Can we have a test that the workflow dies if an invalid expression is set in there?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I've included a bunch of tests and I've also added "invocation messages", see the screenshot in the description ... I think we're now displaying (almost?) all of the possible workflow errors in the API and UI.

@mvdbeek mvdbeek force-pushed the galaxy_conditionals branch 5 times, most recently from 33ee880 to 195dd2f Compare January 15, 2023 17:32
@mvdbeek
Copy link
Member Author

mvdbeek commented Jan 17, 2023

Test errors are unrelated

if value["class"] == "File":
# This is going to re-file -> HDA this each iteration I think, not a good
# implementation.
return progress.raw_to_galaxy(value)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is nothing right? How did we let this in 😆. It seems it hasn't caused any harm yet!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

4 participants