Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add InteractiveTools. #7494

Merged
merged 153 commits into from Aug 22, 2019
Merged
Changes from 1 commit
Commits
Show all changes
153 commits
Select commit Hold shift + click to select a range
261eb7f
Add RealTimeTools. Currently working for docker in local runner.
blankenberg Mar 8, 2019
edeca76
Flake8
blankenberg Mar 8, 2019
22f6d48
Update lib/galaxy/datatypes/realtime.py
nsoranzo Mar 8, 2019
a7c9ac0
fix
blankenberg Mar 9, 2019
b413771
fix
blankenberg Mar 9, 2019
acbee59
Make stop button in Active RealTimeTool list do something.
blankenberg Mar 13, 2019
593bffb
better handling of missing containers
blankenberg Mar 14, 2019
e853f4c
merge dev into realtimetools
blankenberg Mar 15, 2019
054f1fc
Update access to url_for().
blankenberg Mar 15, 2019
696c2cf
import ordering
blankenberg Mar 15, 2019
dcdbee1
Only display view result message when view is available.
blankenberg Mar 16, 2019
2bf465d
Add a ttl cache aware mapper to key_type_token_mapping.py.
blankenberg Mar 20, 2019
2570cb1
Remove entries from realtimetool sqlite db when job has ended.
blankenberg Mar 20, 2019
9274f7d
Simplify RealTimeTool model, associated entry points directly with job.
blankenberg Mar 20, 2019
82704fd
merge dev
blankenberg Mar 20, 2019
27fe34f
remove debug statement
blankenberg Mar 21, 2019
10c71c3
merge dev
blankenberg Mar 21, 2019
e037148
comma
blankenberg Mar 21, 2019
16eb46b
Merge branch 'dev' of https://github.com/galaxyproject/galaxy into re…
blankenberg Mar 21, 2019
91f5f6f
fix for displaying realtimetool grid on refresh.
blankenberg Mar 21, 2019
672d3c3
Merge branch 'dev' of https://github.com/galaxyproject/galaxy into re…
blankenberg Mar 21, 2019
e8da308
Ports for yaml tools.
blankenberg Mar 22, 2019
464aa35
update tools xsd
blankenberg Mar 22, 2019
fcd4c0f
Merge branch 'dev' of https://github.com/galaxyproject/galaxy into re…
blankenberg Mar 22, 2019
523f648
over zealous delete key
blankenberg Mar 22, 2019
d64ba78
flake
blankenberg Mar 22, 2019
c960113
flake
blankenberg Mar 22, 2019
cf314d2
naming update
blankenberg Mar 22, 2019
2d55589
move realtime rpc script to /scripts/realtime/
blankenberg Mar 22, 2019
9ee4b9c
move realtime rpc script to /scripts/realtime/
blankenberg Mar 22, 2019
0a6e4bb
Revert "move realtime rpc script to /scripts/realtime/"
blankenberg Mar 22, 2019
c484330
rename rtt db migration
blankenberg Mar 27, 2019
7f175ac
Merge branch 'dev' of https://github.com/galaxyproject/galaxy into re…
blankenberg Mar 27, 2019
73e81ac
Merge branch 'dev' of https://github.com/galaxyproject/galaxy into re…
blankenberg Apr 2, 2019
3e94402
rename realtimetool migration to higher version
blankenberg Apr 2, 2019
417d777
Merge branch 'dev' of https://github.com/galaxyproject/galaxy into re…
blankenberg Apr 3, 2019
2a26d8c
Merge branch 'dev' of https://github.com/galaxyproject/galaxy into re…
blankenberg Apr 3, 2019
054e36e
change version number of realtimetools migration
blankenberg Apr 8, 2019
4bfd4d1
Merge branch 'dev' of https://github.com/galaxyproject/galaxy into re…
blankenberg Apr 8, 2019
2c4f15e
Merge branch 'dev' of https://github.com/galaxyproject/galaxy into re…
blankenberg Apr 8, 2019
557135b
update migrate version of realtimetools
blankenberg Apr 8, 2019
5fdcea3
Merge branch 'dev' of https://github.com/galaxyproject/galaxy into re…
blankenberg Apr 8, 2019
ae312d7
Merge branch 'dev' of https://github.com/galaxyproject/galaxy into re…
blankenberg Apr 16, 2019
ee680d7
Remove inspection check, only do ports
blankenberg Apr 16, 2019
7a5a987
Remove unneeded RealTimeTool datatype.
blankenberg Apr 16, 2019
2fa9ff1
No more inspect, no more json needed.
blankenberg Apr 16, 2019
f164db2
Merge branch 'dev' of https://github.com/blankenberg/galaxy into real…
blankenberg May 9, 2019
e5947b9
Update rtt migrate version number
blankenberg May 9, 2019
97e8a4f
js-lint fix
blankenberg May 9, 2019
1fdfdd3
remove extra _
blankenberg May 10, 2019
ed66427
Merge branch 'dev' of https://github.com/galaxyproject/galaxy into re…
blankenberg May 10, 2019
5fd9ef4
Merge branch 'dev' of https://github.com/galaxyproject/galaxy into re…
blankenberg May 13, 2019
3d829c4
Update RealTimeTool entry point definitions.
blankenberg May 13, 2019
0c94350
Update xsd
blankenberg May 13, 2019
ac7d6fd
Enable templating of RealTime tools name, port, url.
blankenberg May 13, 2019
7376cc9
Add docs and example job config.
blankenberg May 13, 2019
d4fa635
Merge branch 'dev' of https://github.com/galaxyproject/galaxy into re…
blankenberg May 13, 2019
13a5524
Remove unused import
blankenberg May 13, 2019
718f5f0
change cellxgene Container to work with the IE tools
bgruening May 5, 2019
3c56b59
make entry_point_url parsing more robust
bgruening May 5, 2019
ba6fc19
add initial condor runner support
bgruening May 5, 2019
0a15a46
add initial test tools for a interactive Vis
bgruening May 5, 2019
67e8ad3
add initial Jupyter notebook tool, this tool supports loading a noteb…
bgruening May 5, 2019
dd3839a
fix bug in setting the --net option during Docker startup
bgruening May 5, 2019
09397f8
Update example RealTimeTools to use current syntax.
blankenberg May 13, 2019
b060b72
Merge pull request #5 from blankenberg/realtimetools-bg
blankenberg May 13, 2019
13c3f77
flake8 for lib/galaxy/jobs/runners/condor.py
blankenberg May 13, 2019
b7072d8
mock
blankenberg May 13, 2019
2866b5f
Update docs
blankenberg May 13, 2019
2eb2106
Ports to test.
blankenberg May 13, 2019
6cc43da
Doc formatting
blankenberg May 13, 2019
5ee6e65
More doc info
blankenberg May 14, 2019
b0baaab
notes on resource consumption; and condor
blankenberg May 14, 2019
5f6f7df
Merge branch 'dev' of https://github.com/galaxyproject/galaxy into re…
blankenberg May 14, 2019
1c12c74
update docs
blankenberg May 15, 2019
138e904
More doc formatting
blankenberg May 15, 2019
7c1e009
Merge branch 'dev' of https://github.com/galaxyproject/galaxy into re…
blankenberg May 15, 2019
133d8b7
missing t
blankenberg May 15, 2019
d1f1391
Merge branch 'dev' of https://github.com/galaxyproject/galaxy into re…
blankenberg May 20, 2019
87fed7e
re.sub on cellxgene input name
blankenberg May 20, 2019
9d93823
Simplify set_name stuff in container.
jmchilton May 20, 2019
9136e29
Only set container if container exists, fixes tests failing due to Mo…
blankenberg May 20, 2019
033a776
Merge branch 'dev' of https://github.com/galaxyproject/galaxy into re…
blankenberg May 20, 2019
98f7fd7
Merge branch 'dev' of https://github.com/galaxyproject/galaxy into rt…
blankenberg May 21, 2019
5cc0d80
Fix tool_util import
blankenberg May 21, 2019
6e01ca8
Change default realtime prefix to something that isn't broken.
jmchilton May 21, 2019
58cbb50
Refactor realtime entry point handling toward API usage.
jmchilton May 20, 2019
821778f
Move computing result_view for RTT to client.
jmchilton May 21, 2019
03e9a6f
Integration testing for RTT API hooks.
jmchilton May 20, 2019
ba41c5b
Rebase into intergration testsing..
jmchilton May 22, 2019
1126d68
Rework client polling & entry point index API for potential reuse in …
jmchilton May 22, 2019
75c5785
Make realtime tool grid actual links, poll on active.
jmchilton May 22, 2019
0dfe134
Realtime Tools: Monitor for entrypoint ports in job script.
jmchilton May 22, 2019
70989ef
Proper container killing when running jobs.
jmchilton May 22, 2019
e7d4265
Add missing container_monitor package to previous commits.
jmchilton May 28, 2019
b00cff0
Merge branch 'dev' of https://github.com/galaxyproject/galaxy into re…
blankenberg May 30, 2019
57ac6bd
Lowercase import Vue from "vue";
blankenberg May 30, 2019
1ce76c2
flake8
blankenberg May 30, 2019
e194ba4
jslint
blankenberg May 30, 2019
afc048d
flake
blankenberg May 30, 2019
2ed75af
Update driver_util.py
blankenberg May 31, 2019
d05a3c9
Add container_montior_command to MockJobWrapper
blankenberg Jun 4, 2019
178c1ce
montior --> monitor
blankenberg Jun 4, 2019
7b4c1d7
juypter --> jupyter
blankenberg Jun 6, 2019
ccc4fc4
Merge branch 'dev' of https://github.com/galaxyproject/galaxy into re…
blankenberg Jun 6, 2019
540b2b3
Merge branch 'dev' of https://github.com/galaxyproject/galaxy into re…
blankenberg Jun 17, 2019
f231e42
Merge branch 'dev' of https://github.com/galaxyproject/galaxy into re…
blankenberg Jul 5, 2019
4f347e0
Change RealTimeTool to InteractiveTool in UI.
blankenberg Jul 5, 2019
eb22876
Change realtime tagset to entry_points.
blankenberg Jul 5, 2019
588eb42
Merge pull request #7 from blankenberg/interactivetools
blankenberg Jul 5, 2019
35055ed
interactive
blankenberg Jul 6, 2019
9301550
Fix for check_for_entry_points when file exists, but is not fully pop…
blankenberg Jul 7, 2019
f0fce80
Pass through tool defined environment_variables to Docker.
blankenberg Jul 7, 2019
e4dd821
Merge branch 'dev' of https://github.com/galaxyproject/galaxy into re…
blankenberg Jul 7, 2019
102f935
from galaxy.web.base.controller import is now from galaxy.webapps.bas…
blankenberg Jul 7, 2019
75bc929
Add 404 when InteractiveTool route doesn't exist or can't be accessed
blankenberg Jul 7, 2019
0d6a66f
Flake
blankenberg Jul 7, 2019
361ed34
Add ability to strip whitespace from tool environment_variables.
blankenberg Jul 7, 2019
fd7f0d3
Fix for fetching strip if not defined
blankenberg Jul 8, 2019
56aab88
Merge branch 'dev' of https://github.com/galaxyproject/galaxy into re…
blankenberg Jul 11, 2019
e04e03d
Don't set user in example interactive tool job_conf.xml
blankenberg Jul 25, 2019
8aebb64
Merge branch 'dev' of https://github.com/galaxyproject/galaxy into re…
blankenberg Jul 25, 2019
04b1e8f
add a bunch of ITs
bgruening Aug 11, 2019
3689d85
some small changes
bgruening Aug 11, 2019
29e1173
Add API_KEY, GALAXY_URL, and fix HISTORY_ID envars
blankenberg Aug 12, 2019
717d949
Merge pull request #72 from blankenberg/patch-1
bgruening Aug 12, 2019
5655ace
Fix websockets in interactive_tool_bam_iobio.xml
blankenberg Aug 12, 2019
986b0ba
Merge pull request #73 from blankenberg/patch-1
bgruening Aug 13, 2019
22edfcc
fixes for askomics IT
abretaud Aug 13, 2019
01c23ce
Merge pull request #75 from abretaud/it_askoko
bgruening Aug 13, 2019
0e5892c
Update galaxy.yml.realtime example with outputs_to_working_directory:…
blankenberg Aug 13, 2019
5ed5ff6
Merge branch 'dev' of https://github.com/galaxyproject/galaxy into re…
blankenberg Aug 13, 2019
24c7c76
remove complexity from ipython wrapper, not needed for this version
bgruening Aug 14, 2019
2fd4c13
enhance documentation for developers
bgruening Aug 14, 2019
c34fce8
add some more comments
bgruening Aug 14, 2019
0ba4ca5
cleanup comments
blankenberg Aug 14, 2019
764e28f
Merge branch 'dev' of https://github.com/galaxyproject/galaxy into re…
blankenberg Aug 14, 2019
6953bfa
update migration script version
blankenberg Aug 14, 2019
8444da7
Merge pull request #8 from bgruening/it
blankenberg Aug 14, 2019
c3c4aef
Merge branch 'dev' of https://github.com/galaxyproject/galaxy into re…
blankenberg Aug 20, 2019
4bda4c9
Make InteractiveTools enable-able.
blankenberg Aug 20, 2019
ab6e934
minor typo
blankenberg Aug 20, 2019
00e575f
interactive_tool_juypter_notebook.xml -> interactive_tool_jupyter_not…
blankenberg Aug 20, 2019
41d7155
Add requires_domain to entry_point. Default is False. Currently only …
blankenberg Aug 20, 2019
f16da01
Set requires_domain="True" for all examples.
blankenberg Aug 20, 2019
7250162
Add warning on backwards compatibility of __app__ in future.
blankenberg Aug 20, 2019
34ec3bf
FIXME note on user's API key in tool.
blankenberg Aug 20, 2019
a3167bd
Update galaxy tool XSD.
blankenberg Aug 20, 2019
55c9297
more realtime to interactivetools name changes
blankenberg Aug 20, 2019
d7e99c7
more realtimetool --> interactivetool
blankenberg Aug 20, 2019
7732fae
more realtimetool --> interactivetool
blankenberg Aug 20, 2019
753d258
more realtimetool --> interactivetool
blankenberg Aug 20, 2019
aea9ed2
Fix get_hex for uuid4 in python3
blankenberg Aug 21, 2019
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Jump to
Jump to file
Failed to load files.
Diff view
Diff view
73 changes: 37 additions & 36 deletions test/functional/tools/interactive_tool_juypter_notebook.xml
@@ -1,6 +1,6 @@
<tool id="interactive_tool_jupyter_notebook" tool_type="interactive" name="Interactive Jupyter Notebook" version="0.1">
<requirements>
<container type="docker">bgruening/docker-jupyter-notebook:ie2</container>
<container type="docker">quay.io/bgruening/docker-jupyter-notebook:ie2</container>
</requirements>
<entry_points>
<entry_point name="Jupyter Interactive Tool">
Expand All @@ -9,14 +9,9 @@
</entry_point>
</entry_points>
<environment_variables>
<environment_variable name="HISTORY_ID" strip="True">
#if $jupyter_notebook:
${__app__.security.encode_id($jupyter_notebook.history_id)}
#else:
${__app__.security.encode_id($out_file1.history_id)}
#end if
</environment_variable>
<environment_variable name="REMOTE_HOST">TODO</environment_variable>
<environment_variable name="HISTORY_ID" strip="True">${__app__.security.encode_id($jupyter_notebook.history_id)}</environment_variable>
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We really need a way to inject this and something comparable to galaxy_infrastructure_url without __app__. Willing to implement higher-level primitives for encoded history ID and inject galaxy_infrastructure_url with a per-job destination override option (81b5ca5). Or at least a warning in every sample that uses __app__ that we will break backward compatibility in the near future

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Added warning in 7250162 and 34ec3bf

<environment_variable name="REMOTE_HOST">${__app__.config.galaxy_infrastructure_url}</environment_variable>
<environment_variable name="GALAXY_WEB_PORT">8080</environment_variable>
<environment_variable name="GALAXY_URL">${__app__.config.galaxy_infrastructure_url}</environment_variable>
<environment_variable name="API_KEY" strip="True">
#if $__user__:
Expand All @@ -33,67 +28,73 @@
mkdir -p ./jupyter/outputs/ &&
mkdir -p ./jupyter/data &&

#for $i, $input in enumerate($inputs):
#set $cleaned_name = re.sub('[^\w\-\.]', '_', str($input.element_identifier))
ln -sf '$input' './jupyter/data/${cleaned_name}_${i}' &&
#end for
#set $cleaned_name = re.sub('[^\w\-\.]', '_', str($input.element_identifier))
ln -sf '$input' './jupyter/data/${cleaned_name}' &&

## change into the directory where the notebooks are located
cd ./jupyter/ &&
export PATH=/home/jovyan/.local/bin:\$PATH &&

#if $mode.mode_select == 'scratch':
## copy default notebook
cp '$__tool_directory__/default_notebook.ipynb' ./ipython_galaxy_notebook.ipynb &&
/startup.sh
jupyter trust ./ipython_galaxy_notebook.ipynb &&
jupyter lab --no-browser --NotebookApp.shutdown_button=True &&
cp ./ipython_galaxy_notebook.ipynb '$jupyter_notebook'

#else:
#set $cleaned_name = re.sub('[^\w\-\.]', '_', str($input.element_identifier))
cp '$mode.ipynb' ./${cleaned_name}.ipynb &&
jupyter trust ./${cleaned_name}.ipynb &&

#for $i, $ipynb in enumerate($mode.ipynbs):
#set $cleaned_name = re.sub('[^\w\-\.]', '_', str($input.element_identifier))
cp '$ipynb' ./${cleaned_name}_${i}.ipynb &&
#end for
#if $mode.run_it:
jupyter nbconvert --to notebook --execute --output ./ipython_galaxy_notebook.ipynb --allow-errors ./*.ipynb &&
mv ./ipython_galaxy_notebook.ipynb $jupyter_notebook
#else:
/startup.sh
jupyter lab --no-browser --NotebookApp.shutdown_button=True &&
#end if

cp ./ipython_galaxy_notebook.ipynb '$jupyter_notebook'
#end if
]]>
</command>
<inputs>

<conditional name="mode">
<param argument="mode_select" type="select" label="Do you already have a notebook?" help="If not, no problem we will provide you with a default one.">
<param name="mode_select" type="select" label="Do you already have a notebook?" help="If not, no problem we will provide you with a default one.">
<option value="scratch">Start with a fresh notebook</option>
<option value="previous">Load a previous notebook</option>
</param>
<when value="scratch"/>
<when value="previous">
<param name="ipynbs" type="data" format="ipynb" multiple="true" label="IPython Notebook"/>
<param name="ipynb" type="data" format="ipynb" label="IPython Notebook"/>
<param name="run_it" type="boolean" truevalue="true" falsevalue="false" label="Execute notebook and return a new one."
help="This option is useful in workflows when you just want to execute a notebook and not dive into the webfrontend."/>
</when>
</conditional>

<param name="inputs" type="data" optional="true" multiple="true" label="Include data into the environment"/>
<param name="input" type="data" optional="true" label="Include data into the environment"/>

</inputs>
<outputs>
<data name="out_file1" format="txt">
<filter>(mode['mode_select'] == 'previous' and not mode['run_it']) or mode['mode_select'] == 'scratch'</filter>
</data>
<data name="jupyter_notebook" format="ipynb" label="Executed Notebook">
<filter>mode['mode_select'] == 'previous'</filter>
<filter>mode['run_it']</filter>
</data>
<!--data auto_format="true" name="collected_container_data">
<discover_datasets pattern="__designation__" directory="./jupyter/outputs/" visible="true" recurse="true" />
</data-->
<data name="jupyter_notebook" format="ipynb" label="Executed Notebook"></data>
</outputs>
<tests>
<test expect_num_outputs="1">
<param name="mode" value="previous" />
<param name="ipynb" value="test.ipynb" />
<param name="run_it" value="true" />
<output name="jupyter_notebook" file="test.ipynb" ftype="ipynb"/>
</test>
</tests>
<help>
Interactive tool.
The Jupyter Notebook is an open-source web application that allows you to create and share documents that contain live code, equations,
visualizations and narrative text. Uses include: data cleaning and transformation, numerical simulation, statistical modeling, data visualization,
machine learning, and much more.

Galaxy offers you to use Jupyter Notebooks directly in Galaxy accessing and interacting with Galaxy datasets as you like. A very common use-case is to
do the heavy lifting and data reduction steps in Galaxy and the plotting and more `interactive` part on smaller datasets in Jupyter.

You can start with a new Jupyter notebook from scratch or load an already existing one, e.g. from your collegue and execute it on your dataset.
If you have a defined input dataset you can even execute a Jupyter notebook in a workflow, given that the notebook is writing the output back to the history.

You can import data into the notebook via a predefined `get()` function and write results back to Galaxy with a `put()` function.
</help>
</tool>