Installation on Main

Instructions for requestors:

Anyone can request tool installations or updates on usegalaxy.org or test.galaxyproject.org. In the commands below fill the {server_name} as appropriate (usegalaxy.org, test.galaxyproject.org)

1. Fork and clone usegalaxy-tools
2. Create/activate a virtualenv and pip install -r requirements.txt
3. You are either installing a new repo or updating existing repo
   • NEW REPO
     1. If this is a new a section without an existing yml file create a new one like this:
        1. Determine the desired section label
        2. Normalize the section label to an ID/filename with this process
        3. Create {server_name}/<section_id>.yml setting tool_panel_section_label from the section label obtained in previous step (see existing yml files for exact syntax)
        4. Continue with the steps below
     2. Add the entry for the new tool to the section yml example
     3. Run make TOOLSET={server_name} OWNER={repo_owner} update-trusted for every owner in your yml file, then run $ make TOOLSET={server_name} fix, and then $ git add <file> only the updates that you care about.
   • UPDATE REPO
     1. Find the yml and yml.lock files with the repository entries. Add the changeset hash of repo's desired installable revision to the yml.lock file example
     • Alternatively, run make TOOLSET={server_name} OWNER={repo_owner} update-trusted for every owner in your yml file, then run $ make TOOLSET={server_name} fix, and then $ git add <file> only the updates that you care about.
4. Run make TOOLSET={server_name} lint
5. Commit {server_name}/<repo>.yaml{.lock}
6. Create a PR against the master branch of usegalaxy-tools
   • Use PR labels as appropriate
   • To aid PR mergers, you can include information on tools in the repo's use of $GALAXY_SLOTS, or even PR any needed update(s) to Main's job_conf.xml as explained in the "Determine tool requirements" section once the test installation (via Travis) succeeds (see details below)
7. Once the PR is merged and the tool appears on usegalaxy.org or test.galaxyproject.org, test to ensure the tool works

Instructions for tool installers

Members of the @galaxyproject/tool-installers group can deploy and merge tool installation/update PRs.

1. Review PRs to usegalaxy-tools and verify that tool(s) to install are acceptable
2. Wait for the test installation (via Jenkins) to complete
3. Review the test installation output to ensure the tool appears to have installed correctly
   1. You can re-trigger the installation test by commenting on PR with @galaxybot test this
4. If any tools to install or update use $GALAXY_SLOTS:
   1. Check that the tool is assigned to the multi partition in Main's job_conf.yml (or Test's job_conf.yml)
   2. If any updates are needed, commit them to Main's job_conf.yml (or Test's job_conf.yml) as explained in the "Determine tool requirements" section and run make test-config|main-config on galaxy04 (if you have access to do so) or PR the update(s) and request that someone with access merge and run the config playbook (if you do not have access)
5. Trigger the deployment commenting on PR using phrase @galaxybot deploy this
6. Check the deployment on Jenkins to ensure that the installation completes successfully
7. Ensure the tool is installed (CVMFS Stratum 1 snapshots occur at :00, :15, :30, and :45; larger installs can take a while to transfer)
8. Test to ensure the tool works and (using metrics) is assigned the correct number of slots
9. Merge the PR

DEPRECATED Manual Installation

You should not use this method except as a last resort.

Before you begin see Getting Set Up At TACC

Run the installer

Make sure nobody is running a docker container before restarting the service (e.g. with docker ps)...

$ ssh g2main@cvmfs0-tacc0.galaxyproject.org
$ cvmfs_server transaction main.galaxyproject.org
$ docker run --rm -v /cvmfs:/cvmfs -v /galaxy-repl:/galaxy-repl:ro -p 127.0.0.1:8818:8888 galaxy/main-installer

Install Tools

Go to https://main-installer.galaxyproject.org/ and install tools as normal.

Custodial tasks

Conda can install things with less-than-world-readable permissions, which will cause the Galaxy user to be unable to read them, since the CVMFS client mounts everything as the cvmfs user. Additionally, it leaves behind tarballs (which Galaxy will complain about) and often fails to make conda/activate/deactivate symlinks in new environments.

Fix this with:

^C (to stop Galaxy)
$ /cvmfs/main.galaxyproject.org/bin/post_install.sh

Finalize CVMFS transaction

Get the latest tag and append .N to the tag name, e.g. if latest tag is galaxy_16.04.04863c8, the new tag should be galaxy_16.04.04863c8.1, followed by galaxy_16.04.04863c8.2, etc.

$ cvmfs_server tag main.galaxyproject.org
$ cvmfs_server publish -a galaxy_16.04.04863c8.1 -m 'Installed/Updated Foo Tool' main.galaxyproject.org
$ logout

Determine tool requirements

If the tool wrapper uses $GALAXY_SLOTS, edit env/main/templates/galaxy/config/job_conf.xml.j2 and add an entry for the tool to run on the multi partition, e.g. (copy from from other multi tools):

    <tools>
        ...
        <!-- multi/stampede jobs -->
        ...
        <tool id="SHORT_TOOL_ID" destination="dynamic_local_stampede_select_dynamic_walltime" handler="multi_handlers"  resources="local_or_stampede"/>
    </tools>

Update CVMFS stratum 1 servers, local caches, and restart Galaxy

See the Getting Set Up At TACC page for the ansible-env function.

If a job_conf.xml change was not made, run:

$ ansible-env main snapshot
$ ansible-env main backup

If a job_conf.xml change was made, run:

$ ansible-env main snapshot --tags=no-restart
$ ansible-env main config

Push backups to playbook

Inspect the mutable configs and push if everything looks good.

$ ansible-env main update
$ git diff     # verify that none of the mutable configs pulled to the playbook are mangled
$ git add -- .
$ git commit -m 'Update Main mutable configs'
$ git push