Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Updated path-based interactive tools with entry point path injection, support for ITs with relative links, shortened URLs, doc and config updates including Podman job_conf #16795

Merged
merged 18 commits into from
Nov 16, 2023
Merged
Show file tree
Hide file tree
Changes from 15 commits
Commits
Show all changes
18 commits
Select commit Hold shift + click to select a range
cf3fe88
Initial implementation of entry_point_path inject
sveinugu Jul 14, 2023
05f8de5
Updated JupyTool, Open Refine, Guacamole Desktop, and RStudio tools t…
sveinugu Jul 14, 2023
d6a5a6d
Added 'label', 'requires_path_in_url' and 'requires_path_in_header' c…
sveinugu Aug 27, 2023
51db303
Using label instead of name to identify entrypoint
sveinugu Oct 4, 2023
b0e13e5
Added "requires_path_in_url" and "requires_path_in_header" attributes…
sveinugu Oct 5, 2023
73100de
Shortened all interactive tool URLs incl encoded ids of entry points
sveinugu Oct 4, 2023
bf422d3
Removed "interactivetools_shorten_url"
sveinugu Oct 4, 2023
da5a945
Updated config sample files and interactive tools documentation
sveinugu Oct 5, 2023
09020ed
Removed info about default pools value as this is currently incorrect…
sveinugu Oct 6, 2023
54098be
Added simple job_conf.yml files for interactive tools, including conf…
sveinugu Oct 5, 2023
d3c2838
Added note about lack of support for data_dir resolution of "interact…
sveinugu Oct 6, 2023
09a09d6
Fixed linting, import sorting and unrelated mypy error (due to deprec…
sveinugu Oct 6, 2023
7ca529e
Update gravity requirement to 1.0.4, waiting for required gravity + g…
sveinugu Oct 15, 2023
499c122
Cleanup/refactor of URL construction in InteractiveToolManager (as re…
sveinugu Oct 16, 2023
e9c43b9
Lint fix
sveinugu Nov 15, 2023
98f0667
Less secluded docker containers
sveinugu Nov 15, 2023
ed3b288
Revert removal of docs for default pools argument
sveinugu Nov 15, 2023
2506598
Added note on the removal of interactivetools_shorten_url
sveinugu Nov 15, 2023
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Jump to
Jump to file
Failed to load files.
Diff view
Diff view
15 changes: 13 additions & 2 deletions config/galaxy.yml.interactivetools
Original file line number Diff line number Diff line change
Expand Up @@ -2,18 +2,29 @@ gravity:
gx_it_proxy:
enable: true
port: 4002

#handlers:
# handler:
# processes: 3
# pools:
# - job-handlers
# - workflow-schedulers

galaxy:
interactivetools_enable: true
interactivetools_map: database/interactivetools_map.sqlite

# outputs_to_working_directory will provide you with a better level of isolation. It is highly recommended to set
# this parameter with InteractiveTools.
outputs_to_working_directory: true
interactivetools_map: database/interactivetools_map.sqlite

# `galaxy_infrastructure_url` needs to be reachable from IT containers.
# For local development you can map arbitrary hostnames. See `job_conf.xml.interactivetools`
# For local development you can map arbitrary hostnames. See `job_conf.yml.interactivetools`
# for an example.
# In the local development case you should use the `http` protocol (e.g http://localhost:8080) to access
# your Galaxy, so saving notebooks doesn't fail due to invalid certificates.
galaxy_infrastructure_url: http://localhost:8080

# Do not set the following 2 options if you are using an upstream proxy server like nginx
interactivetools_upstream_proxy: false
interactivetools_proxy_host: localhost:4002
58 changes: 58 additions & 0 deletions config/job_conf.yml.interactivetools
Original file line number Diff line number Diff line change
@@ -0,0 +1,58 @@
## A sample job config for InteractiveTools using local runner. ##

runners:
local:
load: galaxy.jobs.runners.local:LocalJobRunner
workers: 4

# Uncomment if dynamic handlers are defined in "gravity:handlers" section in galaxy.yml
#
#handling:
# assign:
# - db-skip-locked

execution:
default: docker_dispatch
environments:
local:
runner: local

docker_local:
runner: local
docker_enabled: true
#docker_volumes: $defaults,/mnt/galaxyData/libraries:ro,/mnt/galaxyData/indices:ro
#docker_volumes_from: parent_container_name
#docker_memory: 24G
#docker_sudo: false
#docker_sudo_cmd: /usr/bin/sudo -extra_param
#docker_net: bridge
#docker_auto_rm: true
#docker_set_user: $UID
docker_set_user:

# InteractiveTools do need real hostnames or URLs to work - simply specifying IPs will not work.
# If you develop interactive tools on your 'localhost' and don't have a proper domain name
# you need to tell all Docker containers a hostname where Galaxy is running.
# This can be done via the add-host parameter during the `docker run` command.
# 'localhost' here is an arbitrary hostname that matches the IP address of your
# Galaxy host. Make sure this hostname ('localhost') is also set in your galaxy.yml file, e.g.
# `galaxy_infrastructure_url: http://localhost:8080`.
#docker_run_extra_arguments: add-host localhost:host-gateway

#docker_cmd: /usr/local/custom_docker/docker
#docker_host:
#docker_container_id_override: busybox:ubuntu-14.04
#docker_default_container_id: busybox:ubuntu-14.04
#require_container: true
#container_monitor: true
#container_monitor_result: file
#container_monitor_command: python /path/to/galaxy/lib/galaxy_ext/container_monitor/monitor.py
#container_monitor_get_ip_method: null
#container_resolvers_config_file: null
#container_resolvers:

docker_dispatch:
runner: dynamic
type: docker_dispatch
docker_destination_id: docker_local
default_destination_id: local
38 changes: 38 additions & 0 deletions config/job_conf.yml.interactivetools.podman
Original file line number Diff line number Diff line change
@@ -0,0 +1,38 @@
## A sample job config for InteractiveTools using local runner, configured for Podman ##

runners:
local:
load: galaxy.jobs.runners.local:LocalJobRunner
workers: 4

# Uncomment if dynamic handlers are defined in "gravity:handlers" section in galaxy.yml
#
#handling:
# assign:
# - db-skip-locked

execution:
default: docker_dispatch
environments:
local:
runner: local

docker_local:
runner: local
docker_enabled: true

docker_set_user:

# For containers running as root (on the inside)
docker_run_extra_arguments: --security-opt label=disable
# Should work for containers with non-root user (on the inside)
#docker_run_extra_arguments: --userns=keep-id --security-opt label=disable

# Change to home directory of the galaxy user, not the directory of the galaxy installation
docker_cmd: HOME="/home/galaxy"; podman

docker_dispatch:
runner: dynamic
type: docker_dispatch
docker_destination_id: docker_local
default_destination_id: local
12 changes: 0 additions & 12 deletions doc/source/admin/galaxy_options.rst
Original file line number Diff line number Diff line change
Expand Up @@ -2023,18 +2023,6 @@
:Type: str


~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
``interactivetools_shorten_url``
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

:Description:
Shorten the uuid portion of the subdomain or path for interactive
tools. Especially useful for avoiding the need for wildcard
certificates by keeping subdomain under 63 chars
:Default: ``false``
:Type: bool


~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
``retry_interactivetool_metadata_internally``
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Expand Down
12 changes: 2 additions & 10 deletions doc/source/admin/scaling.md
Original file line number Diff line number Diff line change
Expand Up @@ -376,16 +376,8 @@ gravity:
```

In this example 4 processes will be started in total:
3 processes will act as job handlers and workflow schedulers, and one process will be dedicated to handling jobs for the `special` tag only. With the `job_conf.xml` configuration above these would be jobs created by the `test1` tool.
You can omit the `pools` argument, this will then default to:

```yaml
sveinugu marked this conversation as resolved.
Show resolved Hide resolved
...
pools:
- job-handlers
- workflow-schedulers
...
```
3 processes will act as job handlers and workflow schedulers, and one process will be dedicated to handling jobs for the
`special` tag only. With the `job_conf.xml` configuration above these would be jobs created by the `test1` tool.

If you omit the `processes` argument this will default to a single process.
You can further customize the handler names using the `name_template` section,
Expand Down