Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Implement expression tools and non-data tool outputs. #7556

Merged
merged 15 commits into from Mar 20, 2019

Conversation

Projects
None yet
4 participants
@jmchilton
Copy link
Member

commented Mar 18, 2019

PR #6925 introduced a GUI for connecting non-data (e.g. integer, boolean, color, etc..) workflow input parameter to tool input parameters (the backend for this was originally added in #1306). That ideally was just the beginning of work toward using such values in structured ways in workflows.

This PR extends tool output handling to allow producing of non-data parameters. These can serve as a source for non-data values in workflows the same work workflow input parameters can. To make such values more easy to produce, this PR also introduces Galaxy expression tools - mirroring functionality regularly used in CWL. These small JavaScript-based tools that consume inputs just like a regular Galaxy tool but that produce dictionary of non-data values.

I think these expressions will be maximally useful when paired with format 2 workflows once we allow users to load arbitrary tools (I make the case more in full here), but I outline important near term potential uses there as well.

Because there is always a checklist in my PR descriptions:

  • Tool definition language and plumbing and datatype for expressing expressions as jobs.
  • Allow connecting expression tools to parameters in workflows, will delay evaluation of workflow so calculated value
  • Example test expression tools for testing and demonstration.

@galaxybot galaxybot added this to the 19.05 milestone Mar 18, 2019

@jmchilton jmchilton changed the title Implement expression tools and non-data tool outputs. [WIP] Implement expression tools and non-data tool outputs. Mar 19, 2019

@mvdbeek
Copy link
Member

left a comment

Absolutely awesome, can't wait to use this!

Show resolved Hide resolved lib/galaxy/tools/__init__.py Outdated
Show resolved Hide resolved lib/galaxy_ext/expressions/handle_job.py Outdated
Show resolved Hide resolved lib/galaxy/tools/parameters/wrapped_json.py Outdated

@jmchilton jmchilton force-pushed the jmchilton:expression_tools branch from 83542e3 to 88a9b06 Mar 19, 2019

@jmchilton

This comment has been minimized.

Copy link
Member Author

commented Mar 19, 2019

And I force pushed over the Github changes :). Sorry, going to do that without credit.

jmchilton added some commits Feb 7, 2018

Implement expression tools and non-data tool outputs.
PR #6925 introduced a GUI for connecting non-data (e.g. integer, boolean, color, etc..) workflow input parameter to tool input parameters (the backend for this was originally added in #1306). That ideally was just the beginning of work toward using such values in structured ways in workflows.

This PR extends tool output handling to allow producing of non-data parameters. These can serve as a source for non-data values in workflows the same work workflow input parameters can. To make such values more easy to produce, this PR also introduces Galaxy expression tools - mirroring functionality regularly used in CWL. These small JavaScript-based tools that consume inputs just like a regular Galaxy tool but that produce dictionary of non-data values.

I think these expressions will be maximally useful when paired with format 2 workflows once we allow users to load arbitrary tools (I make the case more in full here #7545 (comment)), but I outline some potential uses there as well.

Because there is always a checklist in my PR descriptions:

- Tool definition language and plumbing and datatype for expressing expressions as jobs.
- Allow connecting expression tools to parameters in workflows, will delay evaluation of workflow so calculated value
- Example test expression tools for testing and demonstration.
Rollback expression utilities for operation tools.
Some extras in here for group and filter tool that never made it out of CWL branch and into Galaxy - clean those up for getting expression tools into Galaxy.

@jmchilton jmchilton force-pushed the jmchilton:expression_tools branch 2 times, most recently from ab78f16 to 0aee7bb Mar 19, 2019

@jmchilton jmchilton force-pushed the jmchilton:expression_tools branch from 0aee7bb to f427a74 Mar 19, 2019

@mvdbeek

This comment has been minimized.

Copy link
Member

commented Mar 20, 2019

This is going to be so awesome:
expression_tool

We should probably ship with a tool that just takes a text file and parses that into integer / float/ text / bool/ color, this would be useful for workflows that have to compute parameters, and I think it's not too difficult for users.

Fix enumerating over output
There's no iteritems in python 3. We could
use six.iteritems which calls iteritems on py2 and
items on py3, but I don't expect these dicts to be large
enough that there would be any noticeable difference.
@mvdbeek

This comment has been minimized.

Copy link
Member

commented Mar 20, 2019

I opened a PR against your branch with some fixes I found while testing this

mvdbeek and others added some commits Mar 20, 2019

Allow connecting ExpressionTool outputs to inputs of correct type
This means we can now connect these tools in the workflow editor.
Fix test_workflow_resume_with_mapped_over_input test
self.dataset_populator._get_contents_request(history_id=history_id).json()
returns the dataset collection as the first element.
We do need a HDA here though, so we'd just randomly pick
a HDA with the HDCA ID. So if that dataset was deleted by chance the
test would fail. Ouch.
@jmchilton

This comment has been minimized.

Copy link
Member Author

commented Mar 20, 2019

It is green 😄 - and a million thanks for the workflow test case fix - no wonder it was sort of consistently failing during my history import/export testing but in such an odd way.

@jmchilton jmchilton changed the title [WIP] Implement expression tools and non-data tool outputs. Implement expression tools and non-data tool outputs. Mar 20, 2019

@jmchilton jmchilton added status/review and removed status/WIP labels Mar 20, 2019

@mvdbeek
Copy link
Member

left a comment

Fantastic, 19.05 is going to be a great release!

@martenson
Copy link
Member

left a comment

very exciting, I've noted two minor points

Show resolved Hide resolved lib/galaxy/tools/xsd/galaxy.xsd Outdated
Show resolved Hide resolved scripts/common_startup.sh

@martenson martenson merged commit e65604b into galaxyproject:dev Mar 20, 2019

7 checks passed

api test Build finished. 478 tests run, 1 skipped, 0 failed.
Details
continuous-integration/travis-ci/pr The Travis CI build passed
Details
framework test Build finished. 213 tests run, 0 skipped, 0 failed.
Details
get_code_and_test Workflow: get_code_and_test
Details
integration test Build finished. 297 tests run, 19 skipped, 0 failed.
Details
selenium test Build finished. 151 tests run, 2 skipped, 0 failed.
Details
toolshed test Build finished. 578 tests run, 0 skipped, 0 failed.
Details

@nsoranzo nsoranzo deleted the jmchilton:expression_tools branch Mar 20, 2019

@mvdbeek mvdbeek referenced this pull request Apr 19, 2019

Closed

Expression tool(s) for workflow parameterization #7774

4 of 4 tasks complete
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.