From 2834fe64e7ef765c251c472ae88234aded7fc3ef Mon Sep 17 00:00:00 2001 From: John Chilton Date: Mon, 4 Dec 2017 09:14:56 -0500 Subject: [PATCH] Migrate documentation for setting up a production instance to Galaxy. This is a bugfix for the future bug of missing documentation for 17.09 and before. 18.01 will require a significant rewrite of these pages for the uWSGI and configuration changes to Galaxy and the release_17.09 branch of these docs is now being hosted https://docs.galaxyproject.org/en/release_17.09/, so backporting this configuration documentation will allow us to provide permenant links to this 17.09 varaint of all this documentation (with information about paste and galaxy.ini for instance). This content currently includes: - the production configuration page - the cluster configuration page - the job configuration page - the scaling information page - the ftp configuration page - the top-level nginx page (for external auth content still linking out to the hub) - the top-level apache page (for external auth content still linking out to the hub) - the job metrics configuration page Includes small tweaks to sphinx documentation generation. - Add a quick build mode that skips source generation / Python docs (export GALAXY_DOCS_SKIP_SOURCE=1 to build this way). This speeds up review of core Sphinx docs. - Use AutoStructify for better Markdown integration. Updates to the documentation above and beyond what is on the hub to get it to be closer to reflect the latest and greatest in 17.09 including: - Update the job configuration page with information about XML macros. - Mention and link to newer style resubmission condition expressions (not just an enumerated string anymore). - Updated list of dynamic job runner configuration parameters. - Mention the ``raw`` attribute for ``env`` directives in the job configuration page. - Mention the ``total_walltime`` limit added in #3217. - Drop mentions of older URL-based job conf configuration stuff. - Mention newer Local and Paramiko based shell runners in CLI section. - Mention newer slurm job styles in CLI section. - Improved Pulsar links. - Fix dataset security link in Apache display sites config. - Update job metrics page to link to a new Github issue instead of old Trello issues. --- doc/source/admin/cluster.md | 362 ++++++++++++++++ doc/source/admin/index.rst | 4 + doc/source/admin/jobs.md | 403 ++++++++++++++++++ doc/source/admin/production.md | 166 ++++++++ doc/source/admin/scaling.md | 314 ++++++++++++++ doc/source/admin/special_topics/apache.md | 227 ++++++++++ doc/source/admin/special_topics/ftp.md | 249 +++++++++++ doc/source/admin/special_topics/index.rst | 4 + .../admin/special_topics/job_metrics.md | 39 ++ doc/source/admin/special_topics/nginx.md | 307 +++++++++++++ doc/source/conf.py | 21 +- 11 files changed, 2093 insertions(+), 3 deletions(-) create mode 100644 doc/source/admin/cluster.md create mode 100644 doc/source/admin/jobs.md create mode 100644 doc/source/admin/production.md create mode 100644 doc/source/admin/scaling.md create mode 100644 doc/source/admin/special_topics/apache.md create mode 100644 doc/source/admin/special_topics/ftp.md create mode 100644 doc/source/admin/special_topics/job_metrics.md create mode 100644 doc/source/admin/special_topics/nginx.md diff --git a/doc/source/admin/cluster.md b/doc/source/admin/cluster.md new file mode 100644 index 000000000000..214fd00fc573 --- /dev/null +++ b/doc/source/admin/cluster.md @@ -0,0 +1,362 @@ +# Connecting to a Cluster + +Galaxy is designed to run jobs on your local system by default, but it can be configured to run jobs on a cluster. The front-end Galaxy application runs on a single server as usual, but tools are run on cluster nodes instead. + +A [general reference for the job configuration file](jobs.html) is also available. + +## Distributed Resources Managers + +Galaxy is known to work with: + +* [TORQUE Resource Manager](http://www.adaptivecomputing.com/products/open-source/torque/) +* [PBS Professional](http://www.pbsworks.com/Product.aspx?id=1) +* [Open Grid Engine](http://gridscheduler.sourceforge.net/) +* [Univa Grid Engine](http://www.univa.com/products/grid-engine.php) (previously known as Sun Grid Engine and Oracle Grid Engine) +* [Platform LSF](http://www-03.ibm.com/systems/platformcomputing/products/lsf/) +* [HTCondor](http://research.cs.wisc.edu/htcondor/) +* [Slurm](http://slurm.schedmd.com/) +* [Galaxy Pulsar](/src/admin/config/performance/cluster/index.md#pulsar) (formerly LWR) + +It should also work with [any other DRM](http://www.drmaa.org/implementations.php) which implements a [DRMAA](http://www.drmaa.org) interface. If you successfully run Galaxy with a DRM not listed here, please let us know via an email to the [galaxy-dev mailing list](dhttps://galaxyproject.org/mailing-lists/). + +If you do not already have a DRM, [Pulsar](http://pulsar.readthedocs.io/en/latest/) is available which does not require an existing cluster or a shared filesystem and can also run jobs on Windows hosts. + +Installing and configuring your cluster hardware and management software is outside the scope of this document (and specific to each site). That said, a few pitfalls commonly encountered when trying to get the user Galaxy runs as (referred to in this documentation as `galaxy_user`) able to run jobs on the DRM are addressed here: + +* The host on which the Galaxy server processes run (referred to in this documentation as `galaxy_server`) should be configured in the DRM as a "submit host". +* `galaxy_user` must have a real shell configured in your name service (`/etc/passwd`, LDAP, etc.). System accounts may be configured with a disabled shell like `/bin/false` (Debian/Ubuntu) or `/bin/nologin` Fedora/RedHat. + * If Galaxy is configured to submit jobs as real user (see below) then the above must be true for all users of galaxy. +* The Galaxy server and the worker nodes are running the same version of Python (worker nodes will run Python scripts calling the Galaxy code and its dependencies to set job output file metadata). + +To continue, you should have a working DRM that `galaxy_user` can successfully submit jobs to. + +## Preliminary Setup + +Galaxy (with the exception of the [Pulsar](http://pulsar.readthedocs.io/en/latest/) runner) currently requires a shared filesystem between the application server and the cluster nodes. There is some legacy code in the PBS runner that does file staging, but its operational status is unknown. The path to Galaxy must be exactly the same on both the nodes and the application server (although it is possible to use symlinks to partially subvert the absolute path requirement). This is because absolute paths are used to refer to datasets and tools when running the command on the cluster node. The shared filesystem and absolute pathnames are limitations that will eventually be removed as development time permits. + +For example, if Galaxy is installed like so: + +```console +galaxy_user@galaxy_server% git clone https://github.com/galaxyproject/galaxy.git /clusterfs/galaxy/galaxy-app +``` + + +Then that directory should be accessible from all cluster nodes: + +```console +galaxy_user@galaxy_server% qsub -I +qsub: waiting for job 1234.torque.server to start +qsub: job 1234.torque.server ready + +galaxy_user@node1% cd /clusterfs/galaxy/galaxy-app +galaxy_user@node1% +``` + + +If your cluster nodes have Internet access (NAT is okay) and you want to run the data source tools (upload, ucsc, etc.) on the cluster (doing so is highly recommended), set `new_file_path` in `galaxy.ini` to a directory somewhere in your shared filesystem: + +```ini +new_file_path = /clusterfs/galaxy/tmp +``` + + +Additionally some of the runners including DRMAA may use the ``cluster_files_directory`` for sharing files with the cluster, which defaults to ``database/pbs``. You may need to create this folder. + +```ini +cluster_files_directory = database/pbs +``` + +You may also find that attribute caching in your filesystem causes problems with job completion since it interferes with Galaxy detecting the presence and correct sizes of output files. In NFS caching can be disabled with the `-noac` mount option on Linux (on the Galaxy server), but this may have a significant impact on performance since all attributes will have to be read from the file server upon every file access. You should try the `retry_output_collection` option in `galaxy.ini` first to see if this solves the problem. + +## Runner Configuration + +**This documentation covers configuration of the various runner plugins, not how to distribute jobs to the various plugins.** Consult the [job configuration file documentation](jobs.html) for full details on the correct syntax, and for instructions on how to configure tools to actually use the runners explained below. + +### Local + +Runs jobs locally on the Galaxy application server (no DRM). + +#### Workers + +It is possible to configure the number of concurrent local jobs that can be run by using the `workers` attribute on the plugin. + +```xml + + + +``` + + +#### Slots + +For each destination using the local runner, it is possible to specify the number of CPU slots to assign (default is 1). + +```xml + + + + 2 + + +``` + + +The value of *local_slots* is used to define [GALAXY_SLOTS](https://galaxyproject.org/admin/config/galaxy_slots/). + +### DRMAA + +
DRMAA
Runs jobs via any DRM which supports the [Distributed Resource Management Application API](http://www.drmaa.org). Most commonly used to interface with [PBS Professional](http://www.pbsworks.com/Product.aspx?id=1), [Sun Grid Engine](http://gridscheduler.sourceforge.net/), [Univa Grid Engine](http://www.univa.com/products/grid-engine.php), [Platform LSF](http://www-03.ibm.com/systems/technicalcomputing/platformcomputing/products/lsf/index.html), and [SLURM](http://slurm.schedmd.com/). + +#### Dependencies + +Galaxy interfaces with DRMAA via [drmaa-python](https://github.com/pygridtools/drmaa-python). The drmaa-python module is provided with Galaxy, but you must tell it where your DRM's DRMAA library is located, via the `$DRMAA_LIBRARY_PATH` environment variable, for example: + +```console +galaxy_server% export DRMAA_LIBRARY_PATH=/galaxy/lsf/7.0/linux2.6-glibc2.3-x86_64/lib/libdrmaa.so +galaxy_server% export DRMAA_LIBRARY_PATH=/galaxy/sge/lib/lx24-amd64/libdrmaa.so +``` + + +#### DRM Notes + +**TORQUE**: The DRMAA runner can also be used (instead of the [PBS](cluster.html#pbs) runner) to submit jobs to TORQUE, however, problems have been reported when using the `libdrmaa.so` provided with TORQUE. Using this library will result in a segmentation fault when the drmaa runner attempts to write the job template, and any native job runner options will not be passed to the DRM. Instead, you should compile the [pbs-drmaa](http://apps.man.poznan.pl/trac/pbs-drmaa/wiki) library and use this as the value for `$DRMAA_LIBRARY_PATH`. + +**Slurm**: You will need to install [slurm-drmaa](https://github.com/natefoo/slurm-drmaa/). In production on [usegalaxy.org](https://usegalaxy.org) we observed pthread deadlocks in slurm-drmaa that would cause Galaxy job handlers to eventually stop processing jobs until the handler was restarted. Compiling slurm-drmaa using the compiler flags `-g -O0` (keep debugging symbols, disable optimization) caused the deadlock to disappear. + +### Parameters and Configuration + +Most [options defined in the DRMAA interface](http://www.ogf.org/documents/GFD.143.pdf) are supported. Exceptions include `remoteCommand`, `jobName`, `outputPath`, and `errorPath` since these attributes are set by Galaxy. To pass parameters to your underlying DRM, use the `nativeSpecification` parameter. The format of this parameter is dependent upon the underlying DRM. However, for Grid Engine, it is the list of command line parameters that would be passed to `qsub(1)`. + +```xml + + + + + + + -P bignodes -R y -pe threads 8 + + +``` + + +### PBS + +Runs jobs via the [TORQUE Resource Manager](http://www.adaptivecomputing.com/products/open-source/torque/). For PBS Pro, use [DRMAA](cluster.html#drmaa). + +#### Dependencies + +Galaxy uses the [pbs_python](https://oss.trac.surfsara.nl/pbs_python/) module to interface with TORQUE. pbs_python must be compiled against your TORQUE installation, so it cannot be provided with Galaxy. However, we provide all the necessary automation to compile it - ([more about Galaxy's Framework dependencies](framework_dependencies.html)): + +```console +galaxy_user@galaxy_server% cd /clusterfs/galaxy/galaxy-app +galaxy_user@galaxy_server% LIBTORQUE_DIR=/path/to/libtorque python scripts/scramble.py -e pbs_python +``` + + +#### Newer versions of TORQUE (>4.2) + +Galaxy is compatible with newer versions of TORQUE now that pbs_python has been updated to add support for TORQUE >= v4.2. Scrambling the new version of pbs_python is accomplished simply by modifying eggs.ini to use the new version of pbs_python: + +```ini +;pbs_python = 4.3.5 +pbs_python = 4.4.0 +``` + + +Then scramble as normal: +```console +galaxy_user@galaxy_server% cd /clusterfs/galaxy/galaxy-app +galaxy_user@galaxy_server% LIBTORQUE_DIR=/path/to/libtorque python scripts/scramble.py -e pbs_python +``` + + +As of May 2014 there are still some outstanding bugs in pbs_python. Notably, error code translation is out of alignment. For example if you get error #15025 "Bad UID for Job Execution" pbs_python will report this error incorrectly as "Queue already exists". You may consult the [TORQUE source code](https://github.com/adaptivecomputing/torque/blob/4.2.7/src/include/pbs_error_db.h) for the proper error message that corresponds with a given error number. + + +#### Parameters and Configuration + +Most options available to `qsub(1b)` and `pbs_submit(3b)` are supported. Exceptions include `-o/Output_Path`, `-e/Error_Path`, and `-N/Job_Name` since these PBS job attributes are set by Galaxy. Parameters can be specified by either their flag (as used with `qsub`) or long name (as used with `pbs_submit`). + +```xml + + + + + + + @other.cluster + + + walltime=72:00:00,nodes=1:ppn=8 + 128 + + +``` + + +The value of *ppn=* is used by PBS to define the environment variable $PBS_NCPUS which in turn is used by galaxy for [GALAXY_SLOTS](https://galaxyproject.org/admin/config/galaxy_slots/). + +### Condor + +Runs jobs via the [HTCondor](http://research.cs.wisc.edu/htcondor/) DRM. There are no configurable parameters. Galaxy's interface is via calls to HTCondor's command line tools, rather than via an API. + +```xml + + + + + + +``` + +Galaxy will submit jobs to HTCondor as the "galaxy" user (or whatever user the Galaxy server runs as). In order for vanilla job execution to work as expected, your cluster should be configured with a common UID_DOMAIN to allow Galaxy's jobs to run as "galaxy" everywhere (instead of "nobody"). + +If you need to add additional parameters to your condor submission, you can do so by supplying ``s: + +```xml + + + (machine == some.specific.host) + 4 + + +``` + +### Pulsar + +Runs jobs via Galaxy [Pulsar](/src/admin/config/pulsar/index.md). Pulsar does not require an existing cluster or a shared filesystem and can also run jobs on Windows hosts. It also has the ability to interface with all of the DRMs supported by Galaxy. Pulsar provides a much looser coupling between Galaxy job execution and the Galaxy server host than is possible with Galaxy's native job execution code. + +[Full documentation on setup and configuration](http://pulsar.readthedocs.org/) is available. + +### CLI + +Runs jobs via a command-line/shell interface. The CLI runner itself takes plugins of two types: + +* Shell: For interacting with different shell types. Plugins for rsh, ssh, and gsi-ssh are provided. +* Job: For interacting with a DRM via the DRM's command-line interface. A plugin for Torque is provided + +If you are interested in developing additional plugins, see `galaxy-app/lib/galaxy/jobs/runners/cli_*` for examples. + +#### Parameters and Configuration + +The cli runner requires, at a minimum, two parameters: + +```eval_rst +``shell_plugin`` + This required parameter should be [a cli_shell class](https://github.com/galaxyproject/galaxy/tree/dev/lib/galaxy/jobs/runners/util/cli/shell) currently one of: ``LocalShell``, ``RemoteShell``, ``SecureShell``, ``ParamikoShell``, or ``GlobusSecureShell`` describing which shell plugin to use. + +``job_plugin`` + This required parameter should be [a cli_job class](https://github.com/galaxyproject/galaxy/tree/dev/lib/galaxy/jobs/runners/util/cli/job) currently one of ``Torque``, ``SlurmTorque``, or ``Slurm``. +``` + +All other parameters are specific to the chosen plugins. Parameters to pass to the shell plugin begin with the id `shell_` and parameters to pass to the job plugin begin with the id `job_`. + +#### Shell Plugins + +The `RemoteShell` plugin uses `rsh(1)` to connect to a remote system and execute shell commands. + +```eval_rst +``shell_username`` + Optional user to log in to the remote system as. If unset uses ``rsh``'s default behavior (attempt to log in with the current user's username). + +``shell_hostname`` + Remote system hostname to log in to. + +``shell_rsh`` + ``rsh``-like command to excute (e.g. ``/opt/example/bin/remsh``) - just defaults to ``rst``. +``` + +The `RemoteShell` parameters translate to a command line of `% [-l ] ""`, where the inclusion of `-l` is dependent on whether `shell_username` is set. Alternate values for `shell_rsh` must be compatible with this syntax. + +The `SecureShell` plugin works like the `RemoteShell` plugin and takes the same parameters, with the following differences: + +* The `shell_rsh` default value is `ssh` +* The command line that will be executed is `% -oStrictHostKeyChecking=yes -oConnectTimeout=60 [-l ] ""` + +The `GlobusSecureShell` plugin works like the `SecureShell` plugin and takes the same parameters, with the following difference: + +* The `shell_rsh` default value is `gsi-ssh` + +The ``ParamikoShell`` option was added in 17.09 with this pull request https://github.com/galaxyproject/galaxy/pull/4358 from Marius van den Beek. + +#### Job Plugins + +The `Torque` plugin uses `qsub(1)` and `qstat(1)` to interface with a Torque server on the command line. + +```eval_rst +``job_`` + ```` refers to a ``qsub(1B)`` or ``pbs_submit(3B)`` argument/attribute + (e.g. ``walltime=24:00:00,ncpus=4``). + +Torque attributes can be defined in either their short (e.g. [qsub(1B)](http://cf.ccmr.cornell.edu/cgi-bin/w3mman2html.cgi?qsub(1B)) argument as used on the command line or in a script as `#PBS -`) or long (e.g. [pbs_submit(3B)](http://cf.ccmr.cornell.edu/cgi-bin/w3mman2html.cgi?pbs_submit(3B)) attribute as used in the C library) oforms. Some additional examples follow: + +* `queue`: set the PBS destination (in this example, a queue), equivalent to `queue` +* `128`: set the job priority, equivalent to `128` + +```xml + + + + + + SecureShell + Torque + cluster.example.org + + + walltime=72:00:00,nodes=1:ppn=8 + 128 + + +``` + + +Most options available to `qsub(1b)` and `pbs_submit(3b)` are supported. Exceptions include `-o/Output_Path`, `-e/Error_Path`, and `-N/Job_Name` since these PBS job attributes are set by Galaxy. + +## Submitting Jobs as the Real User + +Galaxy runs as a process on your server as whatever user starts the server - usually an account created for the purpose of running Galaxy. Jobs will be submitted to your cluster(s) as this user. In environments where users in Galaxy are guaranteed to be users on the underlying system (i.e. Galaxy is configured to use external authentication), it may be desirable to submit jobs to the cluster as the user logged in to Galaxy rather than Galaxy's system user. + +### Caveats + +Since this is a complex problem, the current solution does have some caveats: + +* All of the datasets stored in Galaxy will have to be readable on the underlying filesystem by all Galaxy users. Said users need not have direct access to any systems which mount these filesystems, only the ability to run jobs on clusters that mount them. But I expect that in most environments, users will have the ability to submit jobs to these clusters or log in to these clusters outside of Galaxy, so this will be a security concern to evaluate for most environments. + * *Technical details* - Since Galaxy maintains dataset sharing internally and all files are owned by the Galaxy user, when running jobs only under a single user, permissions can be set such that only the Galaxy user can read all datasets. Since the dataset may be shared by multiple users, it is not suitable to simply change ownership of inputs before a job runs (what if another user tried to use the same dataset as an input during this time?). This could possibly be solved if Galaxy had tight control over extended ACLs on the file, but since many different ACL schemes exist, Galaxy would need a module for each scheme to be supported. +* The real user system works by changing ownership of the job's working directory to the system prior to running the job, and back to the Galaxy user once the job has completed. It does this by executing a site-customizable script via [sudo](https://www.sudo.ws/). + * Two possibilities to determine the system user that corresponds to a galaxy user are implemented: i) the user whos name matches the Galaxy user's email address (with the @domain stripped off) and ii) the user whos name is equal to the galaxy user name. Until release 17.05 only the former option is available. The latter option is suitable for Galaxy installations that user external authentification (e.g. LDAP) against a source that is also the source of the system users. + * The script accepts a path and does nothing to ensure that this path is a Galaxy working directory per default (and not at all up to release 17.05). So anyone who has access to the Galaxy user could use this script and sudo to change the ownership of any file or directory. Furthermore, anyone with write access to the script could introduce arbitrary (harmful) code -- so it might be a good idea to give write access only to trustworthy users, e.g., root. + +### Configuration + +You'll need to ensure that all datasets are stored on the filesystem such that they are readable by all users that will use Galaxy: either made readable by a group, or world-readable. If using a group, set your `umask(1)` to `027` or for world-readable, use `022` Setting the umask assumes your underlying filesystem uses POSIX permissions, so if this is not the case, your environment changes may be different. + +The directory specified in `new_file_path` in the Galaxy config should be world-writable, cluster-accessible (via the same absolute path) and have its sticky bit (+t) set. This directory should also be cleaned regularly using a script or program as is appropriate for your site, since temporary files created here may not always be cleaned up under certain conditions. + +The `outputs_to_working_directory` option in the Galaxy config **must** be set to `True`. This ensures that a tool/job's outputs are written to the temporary working directory, which (when using the real user system) is owned by the real user who submitted the job. If left set to the default (`False`), the tool will attempt to write directly to the directory specified in `file_path` (by default, `galaxy-app/database/files/`), which must be owned by the Galaxy user (and thus will not be writable by the real user). + +For releases later than 17.05 you can configure the method how the system user is determined in `config/galaxy.ini` via the variable `real_system_username`. For determining the system user from the email adress stored in Galaxy set it to `user_email`, otherwise for determining the system user from the Galaxy user name set it to `username`. + +Once these are set, you must set the `drmaa_external_*` and `external_chown_script` settings in the Galaxy config and configure `sudo(8)` to allow them to be run. A sudo config using the three scripts set in the sample `galaxy.ini` would be: + +``` +galaxy ALL = (root) NOPASSWD: SETENV: /opt/galaxy/scripts/drmaa_external_runner.py +galaxy ALL = (root) NOPASSWD: SETENV: /opt/galaxy/scripts/drmaa_external_killer.py +galaxy ALL = (root) NOPASSWD: SETENV: /opt/galaxy/scripts/external_chown_script.py +``` + +If your sudo config contains `Defaults requiretty`, this option must be disabled. + +For Galaxy releases > 17.05 the sudo call has been moved to `galaxy.ini` and is thereby configurable by the Galaxy admin. This can be of interest because sudo removes `PATH`, `LD_LIBRARY_PATH`, etc. variables per default in some installations. In such cases the sudo calls in the three variables in galaxy.ini can be adapted, e.g., `sudo -E PATH=... LD_LIBRARY_PATH=... /PATH/TO/GALAXY/scripts/drmaa_external_runner.py`. In order to allow setting the variables this way adaptions to the sudo configuration might be necessary. +Also for Galaxy releases > 17.05: In order to allow `external_chown_script.py` to chown only path below certain entry points the variable `ALLOWED_PATHS` in the python script can be adapted. It is sufficient to include the directorries `job_working_directory` and `new_file_path` as configured in `galaxy.ini`. + +It is also a good idea to make sure that only trusted users, e.g. root, have write access to all three scripts. + +Some maintenance and support of this code will be provided via the usual [Support](https://galaxyproject.org/support/) channels, but improvements and fixes would be greatly welcomed, as this is a complex feature which is not used by the Galaxy Development Team. + +## Contributors + +* **Oleksandr Moskalenko**, debugged a number of problems related to running jobs as the real user and using DRMAA with TORQUE. +* **Jaime Frey**, developer of the HTCondor job runner plugin. +* **Ilya Chorny**, developer of the original "real user" job running code. diff --git a/doc/source/admin/index.rst b/doc/source/admin/index.rst index 552103c851ba..33902198a432 100644 --- a/doc/source/admin/index.rst +++ b/doc/source/admin/index.rst @@ -6,6 +6,10 @@ This documentation is in the midst of being ported and unified based on resource .. toctree:: :maxdepth: 2 + production + cluster + scaling + jobs tool_panel authentication dependency_resolvers diff --git a/doc/source/admin/jobs.md b/doc/source/admin/jobs.md new file mode 100644 index 000000000000..bb6a5455da1a --- /dev/null +++ b/doc/source/admin/jobs.md @@ -0,0 +1,403 @@ +# Galaxy Job Configuration + +By default, jobs in Galaxy are run locally on the server on which the Galaxy application was started. Many options are available for running Galaxy jobs on other systems, including clusters and other remote resources. + +This document is a reference for the job configuration file. [Detailed documentation](cluster) is provided for configuring Galaxy to work with a variety of Distributed Resource Managers (DRMs) such as TORQUE, Grid Engine, LSF, and HTCondor. Additionally, a wide range of infrastructure decisions and configuration changes should be made when running Galaxy as a production service, as one is likely doing if using a cluster. It is highly recommended that the [production server documentation](production.html) and [cluster configuration documentation](cluster.html) be read before making changes to the job configuration. + +**The most up-to-date details of advanced job configuration features can be found in the [sample job_conf.xml](https://github.com/galaxyproject/galaxy/blob/dev/config/job_conf.xml.sample_advanced) found in the Galaxy distribution.** + +### Galaxy Configuration + +Configuration of where to run jobs is performed in the `job_conf.xml` file in `$GALAXY_ROOT/config/`. The path to the config file can be overridden by setting the value of `job_config_file` in `config/galaxy.ini`. Sample configurations are provided at `galaxy-dist/config/job_conf.xml.sample_basic` and `galaxy-dist/config/job_conf.xml.sample_advanced`. The job configuration file is not required - if it does not exist, a default configuration that runs jobs on the local system (with a maximum of 4 concurrent jobs) will be used. `job_conf.xml.sample_basic` provides a configuration identical to the default configuration if no `job_conf.xml` exists. + +## job_conf.xml Syntax + +The root element is ``. + +## Job Runner Plugins + +The `` collection defines job runner plugins that should be loaded when Galaxy starts. + +This configuration element may define a ``workers`` parameters which is the default number of worker threads to spawn for doing runner plugin "work", e.g. doing job preparation, post-processing, and cleanup. The default number of such workers is ``4``. + +The collection contains `` elements. Each ``plugin`` element may define the following parameters. + +```eval_rst +id + ``id`` of the runner plugin referenced in ``destination`` configuration elements. + +type + This must be ``runner`` currently. + +load + Python module containing the plugin, and the class to instantiate. If no class name is provided, the module must list class names to load in a module-level ``__all__`` list. + For example ``galaxy.jobs.runners.local:LocalJobRunner``. + +workers + Number of worker threads to start for this plugin only (defaults to the value specified + on ``plugins`` configuration). +``` + +## Job Handlers + +The `` configuration elements defines which Galaxy server processes (when [running multiple server processes](https://galaxyproject.org/admin/config/performance/scaling/)) should be used for running jobs, and how to group those processes. + +The handlers configuration may define a ``default`` attribute. This is the the handler(s) that should be used if no explicit handler is defined for a job and is required if >1 handlers defined. + +The collection contains `` elements. + +```eval_rst +id + A server name (e.g. a ``[server:]`` in config/galaxy.ini such as ``[server:main]``) that should be used to run jobs. + required + +tags + A comma-separated set of strings that optional define tags to which this handler belongs. +``` + +## Job Destinations + +The `` collection defines the parameters that should be used to run a job that is sent to the specified destination. This configuration element should define a ``default`` +attribute that should be the ``id`` of the ``destination`` to used if no explicit destination is defined for a job. + +The collection contains ``s, which are can be collections or single elements. + +```eval_rst +id + Identifier to be referenced in ```` configuration elements in the ``tools`` section. + +runner + Job runner ``plugin`` to be used to run jobs sent to this destination. + +tags + Tags to which this destination belongs (for example `tags="longwalltime,bigcluster"`). +``` + +``destination`` elements may contain zero or more ````s, which are passed to the destination's defined runner plugin and interpreted in a way native to that plugin. For details on the parameter specification, see the documentation on [Cluster configuration](cluster.html). + +## Environment Modifications + +As of the June 2014 release, destinations may contain additional `env` elements to configure the environment for jobs on that resource. These each map to shell commands that will be injected to Galaxy's job script and executed on the destination resource. + +```eval_rst +id + Environment variable to set (in this case text of element is value this is set to + (e.g. ``id="_JAVA_OPTIONS"``). + +file + Optional path to script File will be sourced to configure environment + (e.g. ``file="/mnt/java_cluster/environment_setup.sh"``). + +exec + Optional shell command to execute to configure environment + (e.g. ``module load javastuff/2.10``) + +raw + Disable auto-quoting of values when setting up environment variables. +``` + +Destinations may also specify other destinations (which may be dynamic destinations) that jobs should be resubmitted to if they fail to complete at the first destination for certain reasons. This is done with the `` tag contained within a ``. + +```eval_rst +condition + Failure expression on which to resubmit jobs - this Python expression may contain + the boolean variables ``memory_limit_reached``, ``walltime_reached``, + ``unknown_error``, or ``any_failure`` and the numeric variables ``seconds_running`` + and ``attempt``. See the test case configuration https://github.com/galaxyproject/galaxy/blob/dev/test/integration/resubmission_job_conf.xml for examples of various expressions. + +handler + Job handler(s) that should be used to run jobs for this tool after resubmission. + +destination + Job destination(s) that should be used to run jobs for this tool after resubmission. +``` + +**Note:** Currently, failure conditions for memory limits and walltime are only implemented for the [Slurm](cluster.html) job runner plugin. Contributions for other implementations would be greatly appreciated! An example job configuration and an always-fail job runner plugin for development [can be found in this gist](https://gist.github.com/natefoo/361414fbca3c0ea63aa5). + +### Macros + +The job configuration XML file may contain any number of macro definitions using the same +XML macro syntax used by [Galaxy tools](http://planemo.readthedocs.io/en/latest/writing_advanced.html#macros-reusable-elements). + +See [Pull Request #362](https://github.com/galaxyproject/galaxy/pull/362) for implementation details and the [advanced sample job_conf.xml](https://github.com/galaxyproject/galaxy/blob/dev/config/job_conf.xml.sample_advanced) for examples. + +## Mapping Tools To Destinations + +### Static Destination Mapping + +The `` collection provides a mapping from tools to a destination (or collection of destinations identified by tag) and handler (or collection of handlers). Any tools not matching an entry in the collection will use the default handler and default destination as explained above. + +The `` collection has no attributes. + +The collection contains ``s, which are can be collections or single elements. + +```eval_rst +id + ``id`` attribute of a Galaxy tool. Valid forms include the short ``id``` as found in the Tool's XML configuration, a full Tool Shed GUID, or a Tool Shed GUID without the version component (for example ``id="toolshed.example.org/repos/nate/filter_tool_repo/filter_tool/1.0.0"`` or ``id="toolshed.example.org/repos/nate/filter_tool_repo/filter_tool"`` or ``id="filter_tool"``). + +handler + Job handler(s) that should be used to run jobs for this tool. + (e.g. ``handler="handler0"`` or ``handler="ngs"``). This is optional and if unspecified + will default to the handler specified as the default handler in the job configuration or + the only job handler if only one is specified. + +destination + Job destination(s) that should be used to run jobs for this tool (e.g. ``destination="galaxy_cluster"`` or ``destination="long_walltime"``). The is optional + and defaults the default destination. +``` + +Tool collections contain zero or more ``s, which map to parameters set at job-creation, to allow for assignment of handlers and destinations based on the manner in which the job was created. Currently, only one parameter is defined - namely ``source``. + +The *content* of the `` tag is the component that created the job. Currently, only Galaxy's visualization component sets this job parameter, and its value is `trackster`. + +```xml +trackster +``` + +### Dynamic Destination Mapping + +Galaxy has very sophisticated job configuration options that allow different tools to be submitted to queuing systems with various parameters and in most cases this is sufficient. However, sometimes it is necessary to have job execution parameters be determined at runtime based on factors such as the job inputs, user submitting the job, cluster status, etc... In these cases the dynamic job destination mechanism allows the deployer to describe how the job should be executed using python functions. There are various flavors of dynamic destinations to handle these scenarios. + +The two most generic and useful dynamic destination types are `python` and `dtd`. The `python` type allows arbitrary Python functions to define destinations for jobs, while the DTD method (introduced in Galaxy 16.07) defines rules for routing in a YAML file. + +#### Dynamic Destination Mapping (DTD method) + +DTD is a special dynamic job destination type that builds up rules given a YAML-based DSL - see `config/tool_destinations.yml.sample` (on [Github](https://github.com/galaxyproject/galaxy/blob/dev/config/tool_destinations.yml.sample)) for a syntax description, examples, and a description of how to validate and debug this file. + +To define and use rules, copy this sample file to `config/tool_destinations.yml` and add your rules. Anything routed with a `dynamic` runner of type `dtd` will then use this file (such as the destination defined with the following XML block in `job_conf.xml`). + +```xml + + dtd + +``` + + +#### Dynamic Destination Mapping (Python method) + +The simplest way to get started with dynamic job destinations is to first create a dynamic job destination in `job_conf.xml`'s `` section: + +```xml + + python + ncbi_blastn_wrapper + +``` + + +Note that any parameters defined on dynamic destinations are only available to the function. If your function dispatches to a static destination, parameters are not propagated automatically. + +Next for any tool one wants to dynamically assign job destinations for, this `blast` dynamic destination must be specified in the `job_conf.xml`'s `` section: + +```xml + +``` + + +Finally, you will need to define a function that describes how `ncbi_blastn_wrapper` should be executed. To do this, one must create a python source file in `lib/galaxy/jobs/rules`, for instance `destinations.py` (though the name of this file is largely unimportant, one can distribute any number of functions across any number of files and they will be automatically detected by Galaxy). + +So open `lib/galaxy/jobs/rules/destinations.py` and define a `ncbi_blastn_wrapper` function. A couple possible examples may be: + +```python +from galaxy.jobs import JobDestination +import os + +def ncbi_blastn_wrapper(job): + # Allocate extra time + inp_data = dict( [ ( da.name, da.dataset ) for da in job.input_datasets ] ) + inp_data.update( [ ( da.name, da.dataset ) for da in job.input_library_datasets ] ) + query_file = inp_data[ "query" ].file_name + query_size = os.path.getsize( query_file ) + if query_size > 1024 * 1024: + walltime_str = "walltime=24:00:00/" + else: + walltime_str = "walltime=12:00:00/" + return JobDestination(runner="pbs", params={"Resource_List": walltime_str}) + +``` + + +or + +```python +from galaxy.jobs import JobDestination +import os + +def ncbi_blastn_wrapper(app, user_email): + # Assign admin users' jobs to special admin_project. + admin_users = app.config.get( "admin_users", "" ).split( "," ) + params = {} + if user_email in admin_users: + params["nativeSpecification"] = "-P bigNodes" + return JobDestination(runner="drmaa", params=params) +``` + + +The first example above delegates to the PBS job runner and allocates extra walltime for larger input files (based on tool input parameter named `query`). The second example delegates to the DRMAA job runner and assigns users in the in the admin list to a special project (perhaps configured to have a higher priority or extended walltime). + +The above examples demonstrate that the dynamic job destination framework will pass in the arguments to your function that are needed based on the argument names. The valid argument names at this time are: + +```eval_rst +``app`` + Global Galaxy application object, has attributes such as config (the configuration parameters loaded from ``config/galaxy.ini``) and ``job_config`` (Galaxy representation of the data loaded in from ``job_conf.xml``). + +``user_email`` + E-mail of user submitting this job. + +``user`` + Galaxy model object for user submitting this job. + +``job`` + Galaxy model object for submitted job, see the above example for how input information can be derived from this. + +``job_wrapper`` + An object meant a higher level utility for reasoning about jobs than ``job``. + +``tool`` + Tool object corresponding to this job. + +``tool_id`` + ID of the tool corresponding to this job + +``rule_helper`` + Utility object with methods designed to allow job rules to interface cleanly with the rest of Galaxy and shield them from low-level details of models, metrics, etc.... + +``resource_params`` + A dictionary of parameters specified by the user using ``job_resource_params_conf.xml`` (if configured). + +``workflow_invocation_uuid`` + A randomly generated UUID for the workflow invocation generating this job - this can be + useful for instance in routing all the jobs in the same workflow to one resource. +``` + +Also available though less likely useful are ``job_id``. + +The above examples demonstrated mapping one tool to one function. Multiple tools may be mapped to the same function, by specifying a function the the dynamic destination: + +```xml + + python + blast_dest + +``` + + +```xml + + + +``` + +In this case, you would need to define a function named `blast_dest` in your python rules file and it would be called for all three tools. In cases like this, it may make sense to take in `tool_id` or `tool` as argument to factor the actual tool being used into your decision. + +As a natural extension to this, a dynamic job runner can be used as the default destination. The below examples demonstrate this and other features such as returning mapping failure messages to your users and defaulting back to existing static destinations defined in `job_conf.xml`. + +##### Additional Dynamic Job Destination Examples + +The following example assumes the existence of a job destination with ids `short_pbs` and `long_pbs` and that a default dynamic job runner has been defined as follows in `job_conf.xml`: + +```xml + + + python + default_runner + + ... +``` + + +With these if place, the following `default_runner` rule function will route all tools with id containing `mothur` to the `long_pbs` destination defined `jobs_conf.xml` and all other tools to the `short_pbs` destination: + +```python +def default_runner(tool_id): + if 'mothur' in tool_id >= 0: + return 'long_pbs' + else: + return 'short_pbs' +``` + + +As another example, assume that a few tools should be only accessible to developers and all other users should receive a message indicating they are not authorized to use this tool. This can be accomplished with the following `job_conf.xml` fragment + +```xml + + + python + dev_only + + ... + + + + ... +``` + + +Coupled with placing the following function in a rules file. + +```python +from galaxy.jobs.mapper import JobMappingException +from galaxy.jobs import JobDestination +DEV_EMAILS = ["mary@example.com"] + +def dev_only(user_email): + if user_email in DEV_EMAILS + return JobDestination(runner="drmaa") + else: + raise JobMappingException("This tool is under development and you are not authorized to it.") +``` + + +There is an additional page on [Access Control](https://galaxyproject.org/admin/config/access-control/) for those interested. + +##### Additional Tricks + +If one would like to tweak existing static job destinations in just one or two parameters, the following idiom can be used to fetch static JobDestination objects from Galaxy in these rule methods - `dest = app.job_config.get_destination( id_or_tag )`. + +## Limiting Job Resource Usage + +The `` collection defines the number of concurrent jobs users can run, output size limits, and a Galaxy-specific limit on the maximum amount of time a job can run (rather than relying on a DRM's time-limiting feature). This replaces the former `job_walltime`, `output_size_limit`, `registered_user_job_limit`, `anonymous_user_job_limit` configuration parameters, as well as the (mostly broken) `[galaxy:job_limits]` section. + +*NB: The `job_walltime` and `output_size` limits are only implemented in the `local` and `pbs` job runner plugins. Implementation in other runners is likely to be fairly easy and would simply require a bit of testing - we would gladly accept a pull request implementing these features in the other provided runner plugins.* + +The `` collection has no attributes. + +The collection contains ``s, which have different meanings based on their required `type` attribute: + +```eval_rst +type + Type of limit to define - one of ``registered_user_concurrent_jobs``, ``anonymous_user_concurrent_jobs``, ``destination_user_concurrent_jobs``, ``destination_total_concurrent_jobs``, ``walltime``, and ``output_size``. + +id + Optional destination on which to apply limit (for ``destination_user_concurrent_jobs`` and ``destination_total_concurrent_jobs`` types only) (e.g. ``id="galaxy_cluster"``). + +tag + Optional destinations on which to apply limit (for ``destination_user_concurrent_jobs`` and ``destination_total_concurrent_jobs`` types only). + +If a limit tag is defined, its value must be set. If the limit tag is not defined, the default for each type is unlimited. The syntax for the available `type`s are: + +``registered_user_concurrent_jobs`` + Limit on the number of jobs a user with a registered Galaxy account can have active across all destinations. + +``anonymous_user_concurrent_jobs`` + Limit on the number of jobs an unregistered/anonymous user can have active across all destinations. + +``destination_user_concurrent_jobs`` + The number of jobs a user can have active in the specified destination, or across all destinations identified by the specified tag. + +``destination_total_concurrent_jobs`` + The number of jobs that can be active in the specified destination (or across all destinations identified by the specified tag) by any/all users. + +``walltime`` + Amount of time a job can run (in any destination) before it will be terminated by Galaxy. + +``total_walltime`` + Total walltime that jobs may not exceed during a set period. If total walltime of finished + jobs exceeds this value, any new jobs are paused. This limit should include a `window` + attribute that is the number in days representing the period. + +``output_size`` + Size that any defined tool output can grow to before the job will be terminated. This does not include temporary files created by the job (e.g. ``53687091200`` for (50 GB)). +``` + +The concept of "across all destinations" is used because Galaxy allows users to run jobs across any number of local or remote (cluster) resources. A user may always queue an unlimited number of jobs in Galaxy's internal job queue. The concurrency limits apply to jobs that have been dispatched and are in the `queued` or `running` states. These limits prevent users from monopolizing the resources Galaxy runs on by, for example, preventing a single user from submitting more long-running jobs than Galaxy has cluster slots to run and subsequently blocking all Galaxy jobs from running for any other user. diff --git a/doc/source/admin/production.md b/doc/source/admin/production.md new file mode 100644 index 000000000000..dd73cf5b8b6d --- /dev/null +++ b/doc/source/admin/production.md @@ -0,0 +1,166 @@ +# Production Environments + +The [basic installation instructions](https://getgalaxy.org) are suitable for development by a single user, but when setting up Galaxy for a multi-user production environment, there are some additional steps that should be taken for the best performance. + +### Why bother? + +By default, Galaxy: + +* Uses [SQLite](http://www.sqlite.org/) (a serverless database), so you don't have to run/configure a database server for quick or basic development. However, while SQLite [supports concurrent access](https://sqlite.org/lockingv3.html) it does not support multiple concurrent writes, which can reduce system throughput. +* Uses a built-in HTTP server, written in Python. Much of the work performed by this server can be moved to [nginx](special_topics/nginx.html) or Apache, which will increase performance. +* Runs all tools locally. Moving to a [cluster](cluster.html) will greatly increase capacity. +* Runs in a single process, which is a performance problem in [CPython](http://en.wikipedia.org/wiki/CPython). + +Galaxy ships with this default configuration to ensure the simplest, most error-proof configuration possible when doing basic development. As you'll soon see, the goal is to remove as much work as possible from the Galaxy process, since doing so will greatly speed up the performance of its remaining duties. This is due to the Python Global Interpreter Lock (GIL), which is explained in detail in the [Advanced Configuration](#advanced-configuration) section. + +### Groundwork for scalability + +#### Use a clean environment + +Many of the following instructions are best practices for any production application. + +* Create a **NON-ROOT** user called galaxy. Running as an existing user will cause problems down the line when you want to grant or restrict access to data. +* Start with a fresh checkout of Galaxy, don't try to convert one previously used for development. Download and install it in the galaxy user home directory. +* Galaxy should be a managed system service (like Apache, mail servers, database servers, *etc.*) run by the galaxy user. Init scripts, OS X launchd definitions and Solaris SMF manifests are provided in the `contrib/` directory of the distribution. You can also use the `--daemon` and `--stop-daemon` arguments to `run.sh` to start and stop by hand, but still run detached. When running as a daemon the server's output log will be written to `paster.log` instead of the terminal, unless instructed otherwise with the `--log-file` argument. +* Give Galaxy its own database user and database to prevent Galaxy's schema from conflicting with other tables in your database. Also, restrict Galaxy's database user so it only has access to its own database. +* Make sure Galaxy is using a clean Python interpreter. Conflicts in $PYTHONPATH or the interpreter's `site-packages/` directory could cause problems. Galaxy manages its own dependencies for the framework, so you do not need to worry about these. The easiest way to do this is with a [virtualenv](http://pypi.python.org/pypi/virtualenv): + +``` +nate@weyerbacher% pip install virtualenv +nate@weyerbacher% virtualenv --no-site-packages galaxy_env +nate@weyerbacher% . ./galaxy_env/bin/activate +nate@weyerbacher% cd galaxy-dist +nate@weyerbacher% sh run.sh +``` + +* Galaxy can be housed in a cluster/network filesystem (it's been tested with NFS and GPFS), and you'll want to do this if you'll be running it on a [cluster](cluster.html). + +## Basic configuration + +The steps to install Galaxy mostly follow those of the regular instructions at [Admin/GetGalaxy](https://getgalaxy.org). The difference is that after performing the groundwork above, you should initialize the configuration file (`cp config/galaxy.ini.sample config/galaxy.ini`) and modify it as outlined below before starting the server. If you make any changes to this configuration file while the server is running, you will have to restart the server for the changes to take effect. + +### Disable the developer settings + +Two options are set in the sample `config/galaxy.ini` which should not be enabled on a production server. You should set both to `False`: + +* `debug = False` - Disable middleware that loads the entire response in memory for displaying debugging information in the page. If left enabled, the proxy server may timeout waiting for a response or your Galaxy process may run out of memory if it's serving large files. +* `use_interactive = False` - Disables displaying and live debugging of tracebacks via the web. Leaving it enabled will expose your configuration (database password, id_secret, etc.). +* Disable `filter-with = gzip`. Leaving the gzip filter enabled will cause UI failures because of the way templates are streamed once `debug` is set to `False`. You will still be able (and are encouraged) to enable gzip in the proxy server. + +During deployment, you may run into problems with failed jobs. By default, Galaxy removes files related to job execution. You can instruct Galaxy to keep files of failed jobs with: `cleanup_job = onsuccess` + +### Switching to a database server + +The most important recommendation is to switch to an actual database server. By default, Galaxy will use [SQLite](http://www.sqlite.org/), which is a serverless simple file database engine. Since it's serverless, all of the database processing occurs in the Galaxy process itself. This has two downsides: it occupies the aforementioned GIL (meaning that the process is not free to do other tasks), and it is not nearly as efficient as a dedicated database server. There are other drawbacks, too. When load increases with multiple users, the risk of transactional locks also increases. Locks will cause (among other things) timeouts and job errors. If you start with SQLite and then later realize a need for a database server, you'll need to migrate your database or start over. Galaxy does not provide an internal method to migrate data from SQLite, and although free conversion tools are available on the web, this process is non-trivial. + +
PostgreSQL
+ +For this reason, Galaxy also supports [PostgreSQL](http://www.postgresql.org/) and [MySQL](http://dev.mysql.com/). *PostgreSQL is much preferred since we've found it works better with our DB abstraction layer, [SQLAlchemy](http://www.sqlalchemy.org).* + +To use an external database, you'll need to set one up. That process is outside the scope of this document, but is usually simple. For example, on Debian and Redhat-based Linuxes, one may already be installed. If not, it should be an `apt-get install` or `yum install` away. On Mac OS X, there are installers available from the [PostgreSQL](http://www.postgresql.org) website. + +Once installed, create a new database user and new database which the new user is the owner of. No further setup is required, since Galaxy manages its own schema. If you are using a UNIX socket to connect the application to the database (this is the standard case if Galaxy and the database are on the same system), you'll want to name the database user the same as the system user under which you run the Galaxy process. + +To configure Galaxy, set `database_connection` in Galaxy's config file, `config/galaxy.ini`. The syntax for a database URL is explained in the [SQLAlchemy documentation](http://docs.sqlalchemy.org/en/latest/core/engines.html). + +Here follow two example database URLs with username and password: + +``` +postgresql://username:password@localhost/mydatabase +mysql://username:password@localhost/mydatabase +``` + + +It's worth noting that some platforms (for example, Debian/Ubuntu) store database sockets in a directory other than the database engine's default. If you're connecting to a database server on the same host as the Galaxy server and the socket is in a non-standard location, you'll need to use these custom arguments (these are the defaults for Debian/Ubuntu, change as necessary for your installation): + +``` +postgresql:///mydatabase?host=/var/run/postgresql +mysql:///mydatabase?unix_socket=/var/run/mysqld/mysqld.sock +``` + + +For more hints on available options for the database URL, see the [SQLAlchemy documentation](http://docs.sqlalchemy.org/en/latest/core/engines.html#database-urls). + +If you are using [MySQL](http://dev.mysql.com/) and encounter the "MySQL server has gone away" error, please note the `database_engine_option_pool_recycle` option in `config/galaxy.ini`. If this does not solve your problem, see [this post](http://gmod.827538.n3.nabble.com/template/NamlServlet.jtp?macro=print_post&node=2354941) on the Galaxy Development [mailing list](/src/mailing-lists/index.md). + +If you are using [MySQL](http://dev.mysql.com/) please make sure the database output is in UTF-8, otherwise you may encounter python TypeErrors. + +If you are using [MySQL](http://dev.mysql.com/) with [MyISAM](http://dev.mysql.com/doc/refman/5.1/en/myisam-storage-engine.html) table engine when Galaxy is in multiprocess configuration, workflow steps will [get executed out of order](http://dev.list.galaxyproject.org/Job-execution-order-mixed-up-tt4662488.html) and fail. Use [InnoDB](http://dev.mysql.com/doc/refman/5.1/en/innodb-storage-engine.html) engine instead or switch to [PostgreSQL](http://www.postgresql.org/). + +### Using a proxy server + +Galaxy contains a standalone web server and can serve all of its content directly to clients. However, some tasks (such as serving static content) can be offloaded to a dedicated server that handles these tasks more efficiently. A proxy server also allows you to authenticate users externally using any method supported by the proxy (for example, Kerberos or LDAP), instruct browsers to cache content, and compress outbound data. Also, Galaxy's built-in web server does not support byte-range requests (required for many external display applications), but this functionality can be offloaded to a proxy server. In addition to freeing the GIL, compression and caching will reduce page load times. + +Downloading and uploading data can also be moved to the proxy server. This is explained in the [Make the proxy handle uploads and downloads](#make-the-proxy-handle-uploads-and-downloads) section below. + +Virtually any server that proxies HTTP should work, although we provide configuration examples for: + +* [Apache](special_topics/apache.html), and +* [nginx](special_topics/nginx.html), a high performance reverse proxy, used by our public Galaxy sites + +### Using a compute cluster + +Galaxy is a framework that runs command-line tools, and if properly configured, can run these tools on a compute [cluster](cluster.html). Without a cluster, you'll be limited to the number of cores in your server, minus those needed to run Galaxy itself. Galaxy currently supports TORQUE PBS, PBS Pro, Platform LSF, and Sun Grid Engine clusters, and does not require a dedicated or special cluster configuration. Tools can even run on heterogeneous cluster nodes (differing operating systems), as long as any dependencies necessary to run the tool are available on that platform. + +Using a cluster will also net you a fringe benefit: When running tools locally, they are child processes of the Galaxy server. This means that if you restart the server, you lose contact with those jobs, and they must be restarted. However on the cluster, if the Galaxy server restarts, the jobs will continue to run and finish. Once the Galaxy job manager starts up, it'll resume tracking and finishing jobs as if nothing had happened. + +Configuration is not difficult once your cluster is set up. Details can be found on the [cluster](cluster.html) page. + +### Cleaning up datasets + +When datasets are deleted from a history or library, it is simply marked as deleted and not actually removed, since it can later be undeleted. To free disk space, a set of scripts can be run (e.g. from `cron`) to remove the data files as specified by local policy. See the [Purge histories and datasets](https://galaxyproject.org/admin/config/performance/purge-histories-and-datasets/) page for instructions. + +### Rotate log files + +To use logrotate to rotate Galaxy log files, add a new file named "galaxy" to /etc/logrotate.d/ directory with something like: + +``` +PATH_TO_GALAXY_LOG_FILES { + weekly + rotate 8 + copytruncate + compress + missingok + notifempty +} +``` + + +### Local Data + +To get started with setting up local data, please see [Data Integration](https://wiki.galaxyproject.org/Admin/DataIntegration) +* All local *reference genomes* must be included in the `builds.txt` file. +* Some tools (for example, Extract Genomic DNA) require that you cache (potentially huge) local .2bit data. +* Other tools (for example, Bowtie2) require that you cache both .fasta data and tool-specific indexes. +* The `galaxy_dist/tool-data/` directory contains a set of sample location (`.loc`) files that describe the metadata and path to local data and indexes. +* Installed tool packages from the [Tool Shed](/src/toolshed/index.md) may also include location files. +* Comments in location files explain the expected format. +* Wikis linked from [Data Integration](https://wiki.galaxyproject.org/Admin/DataIntegration) explain how to obtain, create, or rysnc many common data and indexes. See an individual Tool Shed repository's documentation for more details. + +### Enable upload via FTP + +File sizes have grown very large thanks to rapidly advancing sequencer technology, and it is not always practical to upload these files through the browser. Thankfully, a simple solution is to allow Galaxy users to upload them via FTP and import those files in to their histories. Configuration for FTP is explained on the [File Upload via FTP](https://galaxyproject.org/admin/config/upload-via-ftp/) page. + +## Advanced configuration + +### Load balancing and web application scaling + +As already mentioned, unloading work from the Galaxy process is important due to the Python [Global Interpreter Lock](http://docs.python.org/c-api/init.html#thread-state-and-the-global-interpreter-lock) (GIL). The GIL is how Python ensures thread safety, and it accomplishes this by only allowing one thread to control execution at a time. This means that regardless of the number of cores in your server, Galaxy can only use one. However, there's a solution: Run multiple Galaxy processes and use the proxy server to balance across all of these processes. In practice, Galaxy is split into job handler and web server processes. Job handlers do not service any user requests directly via the web. Instead, they watch the database for new jobs, and upon finding them, handle the preparation, monitoring, running, and completion of them. Likewise, the web server processes are free to deal only with serving content and files to web clients. + +Full details on how to configure scaling and load balancing can be found in the [scaling](https://galaxyproject.org/admin/config/performance/scaling/) documentation. + +### Unloading even more work + +For those readers who've already been running Galaxy on a cluster, a bit of information was recently added to the [cluster](cluster.html) documentation regarding running the data source tools on the cluster (contrary to the default configuration). Running all tools on the cluster is strongly encouraged, so if you have not done this, please check out the new information. + +### Tune the database + +[PostgreSQL](http://www.postgresql.org/) can store results more efficiently than Galaxy, and as a result, reduce Galaxy's memory footprint. When a query is made, the result will remain on the Postgres server and Galaxy can retrieve only the rows it needs. To enable this, set `database_engine_option_server_side_cursors = True` in the Galaxy config. + +If your server logs errors about the database connection pool size, you may need to increase the default minimum and maximum number of pool connections, 5 and 10. These config file options are `database_engine_option_pool_size` and `database_engine_option_max_overflow`. + +Finally, if you are using Galaxy <= release_2014.06.02, we recommend that you instruct Galaxy to use one database connection per thread, to avoid connection overhead and overuse. This can be enabled with `database_engine_option_strategy = threadlocal`. + +### Make the proxy handle uploads and downloads + +By default, Galaxy receives file uploads as a stream from the proxy server and then writes this file to disk. Likewise, it sends files as a stream to the proxy server. This occupies the GIL in that Galaxy process and will decrease responsiveness for other operations in that process. To solve this problem, you can configure your proxy server to serve downloads directly, involving Galaxy only for the task of authorizing that the user has permission to read the dataset. If using nginx as the proxy, you can configure it to receive uploaded files and write them to disk itself, only notifying Galaxy of the upload once it's completed. All the details on how to configure these can be found on the [Apache](special_topics/apache.html) and [nginx](special_topics/nginx.html) proxy instruction pages. diff --git a/doc/source/admin/scaling.md b/doc/source/admin/scaling.md new file mode 100644 index 000000000000..97b2cda4e28d --- /dev/null +++ b/doc/source/admin/scaling.md @@ -0,0 +1,314 @@ +# Scaling and Load Balancing + +The Galaxy framework is written in Python and makes extensive use of threads. However, one of the drawbacks of Python is the [Global Interpreter Lock](http://docs.python.org/c-api/init.html#thread-state-and-the-global-interpreter-lock), which prevents more than one thread from being on CPU at a time. Because of this, having a multi-core system will not improve the Galaxy framework's performance out of the box since Galaxy can use (at most) one core at a time. However, Galaxy can easily run in multiple separate processes, which solves this problem. For a more thorough explanation of this problem and why you will almost surely want to switch to the load balanced configuration if running for more than a small handful of users, see the [production configuration](production.html) page. + +Just to be clear: increasing the values of `threadpool_workers` in `galaxy.ini` or the number of plugin workers in `job_conf.xml` will not make you Galaxy server much more responsive. The key to scaling Galaxy is the ability to run *multiple* Galaxy servers which co-operatively work on the same database. + +A simple configuration: +* 1 "job handler" process - responsible for starting and monitoring jobs, submitting jobs to a cluster (if configured), and for setting metadata (externally or internally). +* 1 "web server" process - responsible for servicing web pages to users. + +An advanced configuration: +* Multiple "job handler" processes. +* Multiple "web server" processes, proxied through a load-balancing capable web server (e.g. nginx or apache). + +### Web Server(s) + +There are a few different ways you can run multiple web server processes: + +**Standalone Paste-based processes:** +* Pros: + * Simplest setup, especially if only using a single web server process + * No additional dependencies + * Proxy not required if only using a single web server process +* Cons: + * Not as resilient to failure + * Load balancing typically round-robin regardless of individual process load + * No dynamic scaling + +**uWSGI:** +* Pros: + * Higher performance server than Paste + * Better scalability and fault tolerance + * Easier process management and Galaxy server restartability +* Cons: + * Requires uWSGI + +Using uWSGI for production servers is recommended by the Galaxy team. + +#### Standalone Paste-based processes + +In `galaxy.ini`, define one or more `[server:...]` sections: + +```ini +[server:web0] +use = egg:Paste#http +port = 8080 +host = 127.0.0.1 +use_threadpool = true +threadpool_workers = 7 + +[server:web1] +use = egg:Paste#http +port = 8081 +host = 127.0.0.1 +use_threadpool = true +threadpool_workers = 7 +``` + + +Two are shown, you should create as many as are suitable for your usage and hardware. On our eight-core server, I run six web server processes. You may find you only need one, which is a slightly simpler configuration. + +#### uWSGI + +In `galaxy.ini`, define a `[uwsgi]` section: + +```ini +[uwsgi] +processes = 8 +stats = 127.0.0.1:9191 +socket = 127.0.0.1:4001 +pythonpath = lib +threads = 4 +logto = /path/to/uwsgi.log +master = True +``` + + +Port numbers for `stats` and `socket` can be adjusted as desired. Moreover, in the `[app:main]` section, you must set: + +```ini +static_enabled = False +track_jobs_in_database = True +``` + + +You will also need to have uWSGI installed. There are a variety of ways to do this. It can be installed system-wide by installing from your system's package manager (on Debian and Ubuntu systems, the `uwsgi` and `uwsgi-plugin-python` provide the necessary components), or with the `easy_install` or `pip` commands (which will install it to the system's Python `site-packages` directory). Alternatively, if you are already running Galaxy from a Python virtualenv, you can use `pip install uwsgi` with that virtualenv's copy of `pip` to install to that virtualenv as your unprivileged Galaxy user. + +Also, make sure you have installed PasteDeploy, you can follow the same ways from above. + +The web processes can then be started under uWSGI using: + +```console +% cd /path/to/galaxy-dist +% PYTHONPATH=eggs/PasteDeploy-1.5.0-py2.7.egg uwsgi --ini-paste config/galaxy.ini +``` + + +The `--daemonize` option can be used to start in the background. uWSGI has an astounding number of options, see [its documentation](http://uwsgi.readthedocs.org/) for help. + +Once started, a proxy server (typically Apache or nginx) must be configured to proxy requests to uWSGI (using uWSGI's native protocol). Configuration details for these can be found below. + +### Job Handler(s) + +In `galaxy.ini`, define one or more additional `[server:...]` sections: + +```ini +[server:handler0] +use = egg:Paste#http +port = 8090 +host = 127.0.0.1 +use_threadpool = true +threadpool_workers = 5 + +[server:handler1] +use = egg:Paste#http +port = 8091 +host = 127.0.0.1 +use_threadpool = true +threadpool_workers = 5 +``` + + +Using web processes as handlers is possible, but it is not recommended since handler operations can impact web UI performance. + +### Remaining configuration options + +If you do not have a `job_conf.xml` file, you will need to create one. There are samples for a basic configuration and an advanced configuration provided in the distribution. Please note that creating `job_conf.xml` overrides any legacy job running settings in `galaxy.ini`. See the [jobs configuration documentation](jobs.html) for more detail on job configuration. + +In `job_conf.xml`, create `` tags with `id` attributes that match the handler server names you defined in `galaxy.ini`. For example, using the configuration above, the `` section of `job_conf.xml` would look like: + +```xml + + + + +``` + + +Any tool not set to an explicit job destination will then be serviced by one of the handlers with the `handlers` tag. It is possible to dedicate handlers to specific destinations or tools. For details on how to do this, please see the [job configuration documentation](jobs.html). + +## Starting and Stopping + +Since you need to run multiple processes, the typical `run.sh` method for starting and stopping Galaxy won't work. The current recommended way to manage these multiple processes is with [Supervisord](http://supervisord.org/). You can use a supervisord config file like the following or be inspired by [this example](https://github.com/galaxyproject/galaxy/blob/dev/contrib/galaxy_supervisor.conf). Be sure to `supervisord restart` or `supervisord reread && supervisord update` whenever you make configuration changes. + +```ini +[program:galaxy_uwsgi] +command = /usr/bin/uwsgi --plugin python --ini-paste /path/to/galaxy/config/galaxy.ini +directory = /path/to/galaxy +umask = 022 +autostart = true +autorestart = true +startsecs = 10 +user = gxprod +environment = PATH=/path/to/galaxy/venv:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin,PYTHON_EGG_CACHE=/path/to/galaxy/.python-eggs,PYTHONPATH=/path/to/galaxy/eggs/PasteDeploy-1.5.0-py2.7.egg +numprocs = 1 +stopsignal = INT +``` + + +This configuration defines a "program" named "galaxy_uwsgi" which represents our galaxy uWSGI frontend. You'll notice that we've set a command, a directory, a umask, all of which you should be familiar with. Additionally we've specified that the process should **autostart** on boot, and **autorestart** if it ever crashes. We specify **startsecs** to say "the process must stay up for this long before we consider it OK. If the process crashes sooner than that (e.g. bad changes you've made to your local installation) supervisord will try again a couple of times to restart the process before giving up and marking it as failed. This is one of the many ways supervisord is much friendly for managing these sorts of tasks. + +Next, we set up our job handlers: + +```ini +[program:handler] +command = /path/to/galaxy/venv/bin/python ./scripts/paster.py serve config/galaxy.ini --server-name=handler%(process_num)s --pid-file=/path/to/galaxy/handler%(process_num)s.pid --log-file=/path/to/galaxy/handler%(process_num)s.log +directory = /path/to/galaxy +process_name = handler%(process_num)s +numprocs = 2 +umask = 022 +autostart = true +autorestart = true +startsecs = 15 +user = gxprod +environment = PYTHON_EGG_CACHE=/path/to/galaxy/.python-eggs,SGE_ROOT=/var/lib/gridengine +``` + + +Nearly all of this is the same as above, however, you'll notice that we use `$(process_num)s`. That's a variable substitution in the command and process_name fields. We've set **numproces=2** which says to launch two handler processes. Supervisord will launch loop over `0..numprocs` and launch a `handler0` and `handler1` process automatically for us, templating out the command string so each handler receives a different log file and name. + +Lastly, we collect the two tasks above into a single group: + +```ini +[group:galaxy] +programs = handler, galaxy_uwsgi +``` + + +This will let us manage these tasks more globally with the `supervisorctl` command line tool: + +```console +# supervisorctl status +galaxy:handler0 RUNNING pid 7275, uptime 16:32:17 +galaxy:handler1 RUNNING pid 7276, uptime 16:32:17 +galaxy:uwsgi RUNNING pid 7299, uptime 16:32:16 +``` + + +This command shows us the status of our jobs, and we can easily restart all of the processes at once by naming the group. Familiar commands like start and stop are also available. + +```console +# supervisorctl restart galaxy: +galaxy:handler0: stopped +galaxy:handler1: stopped +galaxy:uwsgi: stopped +galaxy:uwsgi: started +galaxy:handler0: started +galaxy:handler1: started +``` + +### Transparent restart - Zerg Mode + +The standard uWSGI operation mode allows you to restart the Galaxy application while blocking client connections. Zerg Mode does away with the waiting by running a special Zerg Pool process, and connecting Zergling workers (aka Galaxy application processes) to the pool. As long as at least one is connected, requests can be served. + +See the [GCC2017 Admin Training session](https://github.com/galaxyproject/dagobah-training/blob/2017-montpellier/sessions/10-uwsgi/ex2-zerg-mode.md) on how to set this up. + +## Proxy Server + +If using only one web process, you can proxy as per the normal instructions for a [production configuration page](production.html). Otherwise, you'll need to set up load balancing. + +If you have specified a separate job runner and you want to use the "Manage jobs" interface as administrator you also have to define a proxy for the job runner as shown [below](#manage-jobs). + +### Apache + +Be sure to consult the [Apache proxy documentation](special_topics/apache.html) for additional features such as proxying static content and accelerated downloads. + +#### Standalone Paste-based processes + +To balance on Apache, you'll need to enable `mod_proxy_balancer` in addition to `mod_proxy`, which is available in Apache 2.2 (but not older versions such as 1.3 or 2.0). Add the following to your Apache configuration to set up balancing for the two example web servers defined above: + +```apache + + BalancerMember http://localhost:8080 + BalancerMember http://localhost:8081 + +``` + + +And replace the following line from the [regular proxy configuration](special_topics/apache.html): + +```apache +RewriteRule ^(.*) http://localhost:8080$1 [P] +``` + + +With: + +```apache +RewriteRule ^(.*) balancer://galaxy$1 [P] +``` + + +#### uWSGI + +mod_uwsgi is available in apache2.4 and later. This means you *must* be on Ubuntu 14.04 or later. There are ways to do this on older systems, which is outside the scope of this documentation. You'll need to enable `mod_uwsgi`, and then add the following to your Apache configuration: + +```apache + +Sethandler uwsgi-handler +uWSGISocket 127.0.0.1:4001 +uWSGImaxVars 512 + +``` + + +### nginx + +Be sure to consult the [nginx proxy documentation](special_topics/nginx.html) for additional features such as proxying static content and accelerated downloads. + +#### Standalone Paste-based processes + +To proxy with nginx, you'll simply need to add all of the web applications to the `upstream` section, [which already exists](special_topics/nginx.html). The relevant parts of the configuration would look like this: + +```nginx +http { + upstream galaxy_app { + server localhost:8080; + server localhost:8081; + } + server { + location / { + proxy_pass http://galaxy_app; + proxy_set_header X-Forwarded-Host $host; + proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; + } + } +} +``` + + +#### uWSGI + +uWSGI support is built in to nginx, so no extra modules or recompiling should be required. To proxy to Galaxy, use the following configuration: + +```nginx +uwsgi_read_timeout 180; + +location / { + uwsgi_pass 127.0.0.1:4001; + uwsgi_param UWSGI_SCHEME $scheme; + include uwsgi_params; +} +``` + + +`uwsgi_read_timeout` can be adjusted as appropriate for your site. This is the amount of time connections will block between while nginx waits for a response from uWSGI and is useful for holding client (browser) connections while uWSGI is restarting Galaxy subprocesses. + +## Notes on legacy configurations + +Previously it was necessary to create two separate Galaxy config files to use multiple processes. This is no longer necessary, and if you have multiple config files in your existing installation, it is suggested that you merge them in to a single file. + +Galaxy previously used a single "job manager" process to assign jobs to handlers. This is no longer necessary as handlers are selected by the web processes at the time of job creation. + +The `track_jobs_in_database` option in `galaxy.ini` can still be set but should be unnecessary. If there are more than one `[server:...]` sections in the file, database job tracking will be enabled automatically. diff --git a/doc/source/admin/special_topics/apache.md b/doc/source/admin/special_topics/apache.md new file mode 100644 index 000000000000..e88b1fc50480 --- /dev/null +++ b/doc/source/admin/special_topics/apache.md @@ -0,0 +1,227 @@ +# Proxying Galaxy with Apache + +For various reasons (performance, authentication, etc.) in a production environment, it's recommended to run Galaxy behind a web server proxy. Although any proxy could work, Apache is a common choice. Alternatively, we use [nginx](http://nginx.net/) for our public sites and open source infrastructure, and [details are available](./nginx.html) for it, too. + +Currently the only recommended way to run Galaxy with Apache is using `mod_rewrite` and `mod_proxy`. Either `mod_proxy_uwsgi` or `mod_proxy_http` may be used from there. + +To support proxying, the `mod_proxy`, `mod_http_proxy` and `mod_rewrite` modules must be enabled in the Apache config. The main proxy directives, `ProxyRequests` and `ProxyVia` do **not** need to be enabled. + +## Prerequisites + +Make sure that inbound (and outbound) traffic to the TCP protocol HTTP on port 80 (and HTTPS on port 443 if using SSL) is permitted by your server's firewall/security. + +```eval_rst +.. warning:: Please note that Galaxy should *never* be located on disk inside Apache's `DocumentRoot`. By default, this would expose all of Galaxy (including datasets) to anyone on the web. +``` + +## Basic configuration + +### Allow Encoded Slashes in URLs + +Some Galaxy URLs contain encoded slashes (%2F) in the path and Apache will not serve these URLs by default. To configure Apache to serve URLs with encoded slashes in the path, add the following line to your Apache configuration file: + +```apache +AllowEncodedSlashes NoDecode +``` + +**Note**: The `NoDecode` setting was added in httpd 2.2.18, and CentOS 6 (as RHEL 6 does) only has 2.2.15. The [CentOS SCLo SIG Repo](https://wiki.centos.org/SpecialInterestGroup/SCLo/CollectionsList) has an httpd24 package. + +### Serving Galaxy at the web server root (/) + +For a default Galaxy configuration running on [http://localhost:8080/](http://localhost:8080/), the following lines in the Apache configuration will proxy requests to the Galaxy application: + +```apache +# Rewrite +RewriteEngine on +RewriteRule ^(.*) http://localhost:8080$1 [P] +``` + +Or, if using `mod_proxy` with HTTP transport + +```apache +ProxyPass / http://127.0.0.1:8080/ +``` + +Or, if using `mod_proxy` with uWSGI transport + +```apache +ProxyPass / uwsgi://127.0.0.1:4001/ +``` + +Thus, all requests on your server are now redirected to Galaxy. Because this example uses the "root" of your web server, you may want to use a [VirtualHost](http://httpd.apache.org/docs/2.2/vhosts/) to be able to run other sites from this same server. + +If your Apache server is set up to use `mod_security`, you may need to modify the value of the `SecRequestBodyLimit`. The default value on some systems will limit uploads to only a few kilobytes. + +Since Apache is more efficient at serving static content, it is best to serve it directly, reducing the load on the Galaxy process and allowing for more effective compression (if enabled), caching, and pipelining. To do so, your configuration will now include the following, where `$GALAXY_ROOT` should be replaced with the filesystem path to your Galaxy installation + +```apache +RewriteEngine on +RewriteRule ^/static/style/(.*) $GALAXY_ROOT/static/june_2007_style/blue/$1 [L] +RewriteRule ^/static/scripts/(.*) $GALAXY_ROOT/static/scripts/packed/$1 [L] +RewriteRule ^/static/(.*) $GALAXY_ROOT/static/$1 [L] +RewriteRule ^/favicon.ico $GALAXY_ROOT/static/favicon.ico [L] +RewriteRule ^/robots.txt $GALAXY_ROOT/static/robots.txt [L] +``` + +You will need to ensure that filesystem permissions are set such that the user running your Apache server has access to the Galaxy static/ directory. + +### Serving Galaxy at a sub directory (such as /galaxy) + +It may be necessary to house Galaxy at an address other than the web server root (`http://www.example.org/galaxy`), instead of `http://www.example.org`). To do this, you need to make the following changes: + +Two changes are necessary: + +1. In the apache config, prefix all of the location directives with your prefix, like so: + +```apache +RewriteEngine on +RewriteRule ^/galaxy/static/style/(.*) $GALAXY_ROOT/static/june_2007_style/blue/$1 [L] +RewriteRule ^/galaxy/static/scripts/(.*) $GALAXY_ROOT/static/scripts/packed/$1 [L] +RewriteRule ^/galaxy/static/(.*) $GALAXY_ROOT/static/$1 [L] +RewriteRule ^/galaxy/favicon.ico $GALAXY_ROOT/static/favicon.ico [L] +RewriteRule ^/galaxy/robots.txt $GALAXY_ROOT/static/robots.txt [L] +``` + +Note the first rewrite rule deals with the missing trailing slash problem. If left out, [http://www.example.org/galaxy](http://www.example.org/galaxy) will result in a 404 error. + +If you are using `mod_rewrite` for serving: + +``` +RewriteRule ^/galaxy$ /galaxy/ [R] +RewriteRule ^/galaxy(.*) http://localhost:8080$1 [P] +``` + +Or for `proxy_http`/`proxy_uwsgi`: + +``` +ProxyPass /galaxy http://127.0.0.1:8080/galaxy +# or +ProxyPass /galaxy uwsgi://127.0.0.1:4001/ +``` + +2. The Galaxy application needs to be aware that it is running with a prefix (for generating URLs in dynamic pages). This is accomplished by configuring a Paste proxy-prefix filter in the `[app:main]` section of `config/galaxy.ini` and restarting Galaxy: + + +```ini +[filter:proxy-prefix] +use = egg:PasteDeploy#prefix +prefix = /galaxy + +[app:main] +filter-with = proxy-prefix +cookie_path = /galaxy +``` + +`cookie_prefix` should be set to prevent Galaxy's session cookies from clobbering each other if running more than one instance of Galaxy in different subdirectories on the same hostname. + +### SSL + +If you place Galaxy behind a proxy address that uses SSL (i.e., `https://` URLs), edit your galaxy location block (e.g. `location /` when served at the root, or something else like `location /galaxy` when served under a prefix) + +```apache + + ... + RequestHeader set X-URL-SCHEME https + ... + +``` + +Setting `X-URL-SCHEME` makes Galaxy aware of what type of URL it should generate for external sites like Biomart. This should be added to the existing `` block if you already have one, and adjusted accordingly if you're serving Galaxy from a subdirectory. + +## Advanced Configuration Topics + +### Compression and caching + +All of Galaxy's static content can be cached on the client side, and everything (including dynamic content) can be compressed on the fly. This will decrease download and page load times for your clients, as well as decrease server load and bandwidth usage. To enable, you'll need to load `mod_deflate` and `mod_expires` in your Apache configuration, and then set: + +```apache + + ... + # Compress all uncompressed content. + SetOutputFilter DEFLATE + SetEnvIfNoCase Request_URI \.(?:gif|jpe?g|png)$ no-gzip dont-vary + SetEnvIfNoCase Request_URI \.(?:t?gz|zip|bz2)$ no-gzip dont-vary + SetEnvIfNoCase Request_URI /history/export_archive no-gzip dont-vary + + + # Allow browsers to cache everything from /static for 6 hours + ExpiresActive On + ExpiresDefault "access plus 6 hours" + ... + +``` + +The contents above should be added to the existing `` block if you already have one, and adjusted accordingly if you're serving Galaxy from a subdirectory. + +### Sending files using Apache + +Galaxy sends files (e.g. dataset downloads) by opening the file and streaming it in chunks through the proxy server. However, this ties up the Galaxy process, which can impact the performance of other operations (see [production configuration](../production.html) for a more in-depth explanation). + +Apache can assume this task instead and as an added benefit, speed up downloads. This is accomplished through the use of `mod_xsendfile`, a 3rd-party Apache module. Dataset security is maintained in this configuration because Apache will still check with Galaxy to ensure that the requesting user has permission to access the dataset before sending it. + +To enable it, you must first install `mod_xsendfile`, this is usually available via your OS's repositories. Once done, add the appropriate `LoadModule` directive to your Apache configuration to load the xsendfile module and the `XSendFile` directives to your proxy configuration: + +```apache + + XSendFile on + XSendFilePath / + +``` + +Finally edit your `$GALAXY_ROOT/config/galaxy.ini` and make the following change before restarting Galaxy: + +```ini +[app:main] +apache_xsendfile = True +``` + +For this to work, the user under which your nginx server runs will need read access to Galaxy's `$GALAXY_ROOT/database/files/` directory and its contents. + +### External user authentication + +- [Apache for External Authentication](https://galaxyproject.org/admin/config/apache-external-user-auth/) +- [Built-in Galaxy External Authentication](../authentication.md) + +### Display Sites + +Display sites such as UCSC work not by sending data directly from Galaxy to UCSC via the client's browser, but by sending UCSC a URL to the data in Galaxy that the UCSC server will retrieve data from. Since enabling authentication will place **all** of Galaxy behind authentication, such display sites will no longer be able to access data via that URL. If `display_servers` is set to a non-empty value in `$GALAXY_ROOT/config/galaxy.ini`, this tells Galaxy it should allow the named servers access to data in Galaxy. However, you still need to configure Apache to allow access to the datasets. An example config is provided here that allows the UCSC Main/Test backends: + +```apache + + Satisfy Any + Order deny,allow + Deny from all + Allow from hgw1.cse.ucsc.edu + Allow from hgw2.cse.ucsc.edu + Allow from hgw3.cse.ucsc.edu + Allow from hgw4.cse.ucsc.edu + Allow from hgw5.cse.ucsc.edu + Allow from hgw6.cse.ucsc.edu + Allow from hgw7.cse.ucsc.edu + Allow from hgw8.cse.ucsc.edu + +``` + +**PLEASE NOTE that this introduces a security hole** , the impact of which depends on whether you have restricted access to the dataset via Galaxy's [internal dataset permissions](https://galaxyproject.org/learn/security-features/). + +- By default, data in Galaxy is public. Normally with a Galaxy server behind authentication in a proxy server this is of little concern since only clients who've authenticated can access Galaxy. However, if display site exceptions are made as shown above, anyone could use those public sites to bypass authentication and view any **public** dataset on your Galaxy server. If you have not changed from the default and most of your datasets are public, you should consider running your own display sites that are also behind authentication rather than using the public ones. + +- For datasets for which access has been restricted to one or more roles (i.e. it is no longer "public"), access for reading via external browsers is only allowed for a brief period, when someone with access permission clicks the "display at..." link. During this period, anyone who has the dataset ID would then be able to use the browser to view this dataset. Although such a scenario is unlikely, it is technically possible. + + +### Proxying multiple galaxy worker threads + +If you've configured multiple threads for galaxy in the `config/galaxy.ini` file, you will need a `ProxyBalancer` to manage sending requests to each of the threads. You can do that with apache configuration as follows: + +```apache + + BalancerMember http://localhost:8400 + BalancerMember http://localhost:8401 + + + +# Replace the following line from the regular proxy configuration: +# RewriteRule ^(.*) http://localhost:8080$1 [P] +# With: +RewriteRule ^(.*) balancer://galaxy$1 [P] +``` diff --git a/doc/source/admin/special_topics/ftp.md b/doc/source/admin/special_topics/ftp.md new file mode 100644 index 000000000000..cb41e52a3d11 --- /dev/null +++ b/doc/source/admin/special_topics/ftp.md @@ -0,0 +1,249 @@ +# Galaxy FTP Uploads + +To allow users to upload files to Galaxy via FTP, you'll need to configure Galaxy and install an FTP server. After everything is configured users will be able to upload their files through the FTP server and then select them for importing in the upload dialog in Galaxy. + +For help with uploading data via FTP on Galaxy [Main](https://galaxyproject.org/main/), please see this [tutorial](https://galaxyproject.org/ftp-upload/). + +## Install some FTP server + +Although there is no specific required server, we use [ProFTPD](http://proftpd.org/) for our [public site](http://usegalaxy.org/) since it supports all the things we'll need to be able to do, such as authenticating against the Galaxy database. We recommend you to use the same FTP server as the configurations we provide are targeting it. You can also browse the list of alternative FTP servers at http://en.wikipedia.org/wiki/List_of_FTP_server_software + +## Configure Galaxy + +The first step is to choose a directory into which your users will upload files. Preferably this will be on the same filesystem as Galaxy's datasets (by default, `galaxy_dist/database/files/`). The FTP server will create subdirectories inside of this directory which match the user's email address. Likewise, Galaxy will expect to find email-named subdirectories at that path. This directory should be set in the config file (`galaxy.ini`) as `ftp_upload_dir`. + +In the config file, you'll also want to set `ftp_upload_site` to the hostname your users should connect to via FTP. This will be provided in the help text on the Upload File form. + +## Allow your FTP server to read Galaxy's database + +You'll need to grant a user access to read emails and passwords from the Galaxy database. Although the user Galaxy connects with could be used, I prefer to use a least-privilege setup wherein a separate user is created for the FTP server which has permission to `SELECT` from the `galaxy_user` table and nothing else. In postgres this is accomplished with: + +```bash +postgres@dbserver% createuser -SDR galaxyftp +postgres@dbserver% psql galaxydb +Welcome to psql 8.X.Y, the PostgreSQL interactive terminal. + +Type: \copyright for distribution terms + \h for help with SQL commands + \? for help with psql commands + \g or terminate with semicolon to execute query + \q to quit + +galaxydb=# ALTER ROLE galaxyftp PASSWORD 'dbpassword'; +ALTER ROLE +galaxydb=# GRANT SELECT ON galaxy_user TO galaxyftp; +GRANT +``` + + +## Configuring ProFTPD + +By default, Galaxy stores passwords using [PBKDF2](http://en.wikipedia.org/wiki/PBKDF2). It's possible to disable this using the `use_pbkdf2 = False` setting in `galaxy.ini`. Once disabled, any new passwords created will be stored in an older hex-encoded SHA1 format. Because of this, it's possible to have both PBKDF2 and SHA1 passwords in your database (especially if your server has been around since before PBKDF2 support was added). Although this is fine (Galaxy can read passwords in either format), ProFTPD will expect them in one format or the other (although with some amount of hackery it could probably be made to read both). + +Because of this, you'll need to choose one or the other in your Galaxy config (PBKDF2 is more secure and therefore preferred) and configure ProFTPD accordingly. If users cannot log in because their password is stored in the wrong format, they can simply use Galaxy's password change form to set their password, which will rewrite their password using the currently configured algorithm. + +For more hints on the PBKDF2 configuration, see the fantastic blog post [FTP upload to Galaxy using ProFTPd and PBKDF2](http://galacticengineer.blogspot.co.uk/2015/02/ftp-upload-to-galaxy-using-proftpd-and.html) by Peter Briggs (which was used to create the documentation below). + +Although any FTP server should work, our public site uses ProFTPD. You'll need the following extra modules for ProFTPD: + +* mod_sql +* mod_sql_postgres or mod_sql_mysql +* mod_sql_passwd + +We compile by hand using the following configure arguments (OpenSSL is prebuilt and statically linked), you should read the INSTALL file that come with the proftpd source distribution. At least you should consider if you need to use any of these options "install_user= install_group= ./configure --sysconfdir=/etc --localstatedir=/var": + +```console +./configure --prefix=/foo --disable-auth-file --disable-ncurses --disable-ident --disable-shadow --enable-openssl --with-modules=mod_sql:mod_sql_postgres:mod_sql_passwd --with-includes=/usr/postgres/9.1-pgdg/include:`pwd`/../openssl/.openssl/include --with-libraries=/usr/postgres/9.1-pgdg/lib/64:`pwd`/../openssl/.openssl/lib +``` + + +An example configuration follows, assuming `ftp_upload_dir = /home/nate/galaxy_dist/database/ftp` in the Galaxy config file: + +```apache + +# Basics, some site-specific +ServerName "Public Galaxy FTP" +ServerType standalone +DefaultServer on +Port 21 +Umask 077 +SyslogFacility DAEMON +SyslogLevel debug +MaxInstances 30 + +# This User & Group should be set to the actual user and group name which matche the UID & GID you will specify later in the SQLNamedQuery. +User nobody +Group nogroup +DisplayConnect /etc/opt/local/proftpd_welcome.txt + +# Passive port range for the firewall +PassivePorts 30000 40000 + +# Cause every FTP user to be "jailed" (chrooted) into their home directory +DefaultRoot ~ + +# Automatically create home directory if it doesn't exist +CreateHome on dirmode 700 + +# Allow users to overwrite their files +AllowOverwrite on + +# Allow users to resume interrupted uploads +AllowStoreRestart on + +# Bar use of SITE CHMOD + + DenyAll + + +# Bar use of RETR (download) since this is not a public file drop + + DenyAll + + +# Do not authenticate against real (system) users + +AuthPAM off + + +# Common SQL authentication options +SQLEngine on +SQLPasswordEngine on +SQLBackend postgres +SQLConnectInfo galaxydb@dbserver.example.org[:port] +SQLAuthenticate users +``` + + +For PBKDF2 passwords, the following additions to `proftpd.conf` should work: + +```apache +# Configuration that handles PBKDF2 encryption +# Set up mod_sql to authenticate against the Galaxy database +SQLAuthTypes PBKDF2 +SQLPasswordPBKDF2 SHA256 10000 24 +SQLPasswordEncoding base64 + +# For PBKDF2 authentication +# See http://dev.list.galaxyproject.org/ProFTPD-integration-with-Galaxy-td4660295.html +SQLPasswordUserSalt sql:/GetUserSalt + +# Define a custom query for lookup that returns a passwd-like entry. Replace 512s with the UID and GID of the user running the Galaxy server +SQLUserInfo custom:/LookupGalaxyUser +SQLNamedQuery LookupGalaxyUser SELECT "email, (CASE WHEN substring(password from 1 for 6) = 'PBKDF2' THEN substring(password from 38 for 69) ELSE password END) AS password2,512,512,'/home/nate/galaxy_dist/database/ftp/%U','/bin/bash' FROM galaxy_user WHERE email='%U'" + +# Define custom query to fetch the password salt +SQLNamedQuery GetUserSalt SELECT "(CASE WHEN SUBSTRING (password from 1 for 6) = 'PBKDF2' THEN SUBSTRING (password from 21 for 16) END) AS salt FROM galaxy_user WHERE email='%U'" +``` + +For SHA1 passwords, the following additions to `proftpd.conf` should work: + +```apache +# Set up mod_sql/mod_sql_password - Galaxy passwords are stored as hex-encoded SHA1 +SQLAuthTypes SHA1 +SQLPasswordEncoding hex + +# An empty directory in case chroot fails +SQLDefaultHomedir /var/opt/local/proftpd + +# Define a custom query for lookup that returns a passwd-like entry. Replace 512s with the UID and GID of the user running the Galaxy server +SQLUserInfo custom:/LookupGalaxyUser +SQLNamedQuery LookupGalaxyUser SELECT "email,password,512,512,'/home/nate/galaxy_dist/database/ftp/%U','/bin/bash' FROM galaxy_user WHERE email='%U'" +``` + + +## Further security measures + +FTP protocol is **not** encrypted by default, thus any usernames and passwords are sent over clear text to Galaxy. You may wish to implement further security measures by forcing the FTP connection to use SSL/TLS or to allow users to send their files using SFTP (a completely different protocol than FTP - see http://www.proftpd.org/docs/contrib/mod_sftp.html). Here are some extra steps that you can use with ProFTPD. + +```apache + + # You must put this in a virtual host if you want it to listen on its own port. VHost != Apache Vhost. + + # You may wish to open a new port for this so that you don't lock yourself out of SSH. I chose 2222 + Port 2222 + SFTPEngine on + AuthOrder mod_auth_unix.c mod_sql.c # If you don't do this you will get weird disconnects + SFTPHostKey /etc/ssh/ssh_host_rsa_key + # SFTPCiphers aes256-ctr aes192-ctr aes128-ctr # You may wish to lock it to only certain ciphers, + # but this is likely to lock out certain users + RequireValidShell no + MaxLoginAttempts 6 + ServerName "Galaxy SFTP" + Umask 077 + User galaxyftp + Group galaxyftp + UseFtpUsers off + DefaultRoot ~ + AllowOverwrite on + AllowStoreRestart on + # .. Other rules for directories, etc + SQLEngine on + SQLGroupInfo sftp_groups name id members + # .. See above, the same SQL rules apply + + + + TLSEngine on + TLSLog /var/log/proftpd/tls.log + # Make sure that users know that you have to support TLS 1.2! This is very restrictive, but likely the best + TLSProtocol TLSv1.2 + TLSRSACertificateFile /etc/pki/tls/certs/your_cert.cer + TLSRSACertificateKeyFile /etc/pki/tls/private/your.key + TLSCertificateChainFile /etc/pki/tls/certs/your_intermediate.cer + TLSRenegotiate none + TLSCipherSuite ALL:!SSLv2:!SSLv3 + TLSVerifyClient off + TLSRequired auth+data + TLSOptions NoSessionReuseRequired + ServerName "Galaxy FTP" + ServerType standalone + DefaultServer on + Port 21 + Umask 077 + SyslogFacility DAEMON + SyslogLevel debug + MaxInstances 30 + User galaxyftp + Group galaxyftp + UseFtpUsers off + # Passive port range for the firewall - note that you must open these ports for this to work! + # Since the FTP traffic is now encrypted, your firewall can't peak to see that it is PASSV FTP + # and it will block it if you don't allow new connections one these ports. + PassivePorts 30000 30100 + # Cause every FTP user to be "jailed" (chrooted) into their home directory + DefaultRoot ~ + AllowOverwrite on + # Allow users to resume interrupted uploads + AllowStoreRestart on + # .. Other rules for directories, etc + SQLEngine on + # .. See above, the same SQL rules apply + +``` + +You may need to take some additional steps to get this working. Compile ProFTPD --with-modules=mod_sftp:mod_tls as well as sql. You may need to alter your PostGreSQL configuration (typically pg_hba.conf) to allow local IPv6 connections: + +``` +# IPv6 local connections: +host all all ::1/128 trust +``` + + +You may also need to add a table called 'groups' to allow the SFTP connection to your Galaxy database. + +``` +psql yourDB +# CREATE TABLE sftp_groups ( + id char(5) CONSTRAINT firstkey PRIMARY KEY, + name varchar(40) NOT NULL, + members varchar(100)); +``` + + +With these steps, you should be able to allow users to connect to your server using secure protocols. + +## Other Links + +- [Configuring ProFTP Galaxy Upload for Active Directory](https://galaxyproject.org/admin/config/proftpd-with-ad/) +- [Uploading Data to usegalaxy.org via FTP](https://galaxyproject.org/ftp-upload/) diff --git a/doc/source/admin/special_topics/index.rst b/doc/source/admin/special_topics/index.rst index 0abc9e283f5f..de56dbb527ea 100644 --- a/doc/source/admin/special_topics/index.rst +++ b/doc/source/admin/special_topics/index.rst @@ -5,9 +5,13 @@ Special Topics .. toctree:: :maxdepth: 2 + nginx + apache + ftp interactive_environments mulled_containers grt + job_metrics chat webhooks performance_tracking diff --git a/doc/source/admin/special_topics/job_metrics.md b/doc/source/admin/special_topics/job_metrics.md new file mode 100644 index 000000000000..dc9a6179899c --- /dev/null +++ b/doc/source/admin/special_topics/job_metrics.md @@ -0,0 +1,39 @@ +AGalaxy contains a plugin infrastructure for collecting and displaying metrics for job execution (see the original [pull request](https://bitbucket.org/galaxy/galaxy-central/pull-request/352) for implementation details). + +## Configuration + +An example `config/job_metrics_conf.xml.sample` is included in the Galaxy distribution that describes which plugins are enabled and how they are configured. This will be updated for each new plugin added and so should be considered the most updated, complete source of documentation for what plugins are available and how to configure them. + +### Per Destination + +If Galaxy targets different clusters or just different resources - it may make sense to configure different plugins for different resources, different jobs, etc.... To enable this - individual job destinations may disable, load a different job metrics file, or define metrics directly in `job_conf.xml` in an embedded fashion. + +### Pulsar + +Job metrics can be collected for Pulsar jobs - but there is added complexity. The deployer is responsible for ensuring the same plugins are available to both applications and the Pulsar app is configured with a `job_metrics_conf.xml` that matches Galaxy's configuration for that destination. In practice the Pulsar and Galaxy code bases should stay in-sync and if one is not using per destination metric configurations - this is simply a matter of ensuring the Pulsar and Galaxy are configured with identical `job_metrics_conf.xml` files. + +## Available Plugins + +See `job_metrics_conf.xml.sample` for the most updated information. + +The `core` plugin is one enabled by default and it is a cross-platform plugin used to caputre job run time and core allocation. `env` is the only other cross-platform plugin (should work with any *nix) and can be used to record all environment variables set at job runtime or just specific variables (e.g. `PATH`) - this can potentially be useful for debugging cluster issues. + +There are a number of remaining Linux-only plugins - there include `cpuinfo` and `meminfo` to collect CPU and memory about the node the Galaxy job runs on and `uname` to likewise collect host and operating system information. + +A more experimental and more powerful plugin providing deep integration with [Collectl](http://collectl.sourceforge.net/) is available. The possibilities for uses with this plugin are many and it contains many configuration options. Two of the more useful possibilities this plugin provides include aggregating statistics across the process tree generated by a job to provide per-job resource statistics such as detailed memory usage, detailed CPU breakdown, course I/O information, etc.... or alternatively simply recording a Collectl output file for each job with detailed time-series data for all processes on the runtime system. + +## Viewing Collected Data + +After a job complete successful, all information collected from job metric plugins should be visible to admin users under the dataset details for all of the dataset produces by the job. See the screenshots on the original [pull request](https://bitbucket.org/galaxy/galaxy-central/pull-request/352) for examples. + +## Developing New Plugins + +Galaxy ships with a good variety of plugins ranging from very simple to very complex - a good place to start is likely to just find the most similar plugin and use it as a template for creating a new plugin. Galaxy's existing plugins and new ones to be added should be placed in [lib/galaxy/jobs/metrics/instrumenters/](https://github.com/galaxyproject/galaxy/tree/dev/lib/galaxy/jobs/metrics/instrumenters) + +New plugins should subclass [galaxy.jobs.metrics.instrumenters:InstrumentPlugin](https://github.com/galaxyproject/galaxy/tree/dev/lib/galaxy/jobs/metrics/instrumenters/__init__.py). The general strategy for implementing plugins is to describe the commands that Galaxy should run on the remote server with `post_execute_instrument` and/or `pre_execute_instrument`. These commands should likely write output to files in the job's working directory (produce relative file names with `_instrument_file_name` or full paths with `_instrument_file_path` to ensure remote Galaxy job runners such as the Pulsar can determine what needs to be shipped back to Galaxy for processing). Finally, plugins must implement a `job_properties` method to parse these files created at runtime after the fact back on the Galaxy server and produce a dictionary of properties to summarize the job. + +Plugins can also affect how these properties are displayed to the user by overriding the `formatter` attribute on the `InstrumentPlugin` (see existing plugins for examples). + +## Future Plans + +Future plans can be tracked on this [Github Issue](https://github.com/galaxyproject/galaxy/issues/5134). diff --git a/doc/source/admin/special_topics/nginx.md b/doc/source/admin/special_topics/nginx.md new file mode 100644 index 000000000000..b279face30da --- /dev/null +++ b/doc/source/admin/special_topics/nginx.md @@ -0,0 +1,307 @@ +# Proxying Galaxy with NGINX + +[NGINX](http://nginx.org/en/) is a lightweight http server designed with high performance proxying in mind. The public Galaxy sites ([Main](https://galaxyproject.org/main/) and [Test](https://galaxyproject.org/test/)) as well as the [Docker Galaxy project](https://github.com/bgruening/docker-galaxy-stable) use nginx to proxy rather than Apache for its simple, fast load balancing and other features. + +Galaxy should _never_ be located on disk inside nginx's `root`. By default, this would expose all of Galaxy (including datasets) to anyone on the web. + +## Prerequisites + +Make sure that inbound (and outbound) traffic to the TCP protocol HTTP on port 80 (and HTTPS on port 443 if using SSL) is permitted by your server's firewall/security. + +```eval_rst +.. warning:: Please note that Galaxy should *never* be located on disk inside Nginx's document root. By default, this would expose all of Galaxy (including datasets) to anyone on the web. +``` + +## Basic configuration + +### Serving Galaxy at the web server root (/) + +For a default Galaxy configuration running on [http://localhost:8080/](http://localhost:8080/) (see [SSL](https://github.com/VJalili/galaxy-site/blob/patch-1/src/admin/config/nginxProxy/index.md#ssl) section for HTTPS), the following lines in the nginx configuration will proxy requests to the Galaxy application: + +```nginx +http { + ... + + upstream galaxy_app { + server localhost:8080; + } + + proxy_next_upstream off; + server { + client_max_body_size 10G; + # ... other server stuff ... + location / { + proxy_pass http://galaxy_app; + proxy_set_header X-Forwarded-Host $host; + proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; + } + + # serve static content for visualization and interactive environment plugins + location ~ ^/plugins/(?.+?)/(?.+?)/static/(?.*?)$ { + alias $GALAXY_ROOT/config/plugins/$plug_type/$vis_name/static/$static_file; + } + } +} +``` + + +**Notes:** + +Make sure that you either comment out or modify line containing default configuration for enabled sites. + +``` +include /etc/nginx/sites-enabled/*; +``` + +- The `proxy_next_upstream off;` disables nginx's round-robin scheme to prevent it from submitting POST requests more than once. This is unsafe, and is useful when using more than one upstream. +- Replace `$GALAXY_ROOT` with the path to your copy of Galaxy. +- The parameter `client_max_body_size` specifies the maximum upload size that can be handled by POST requests through nginx. You should set this to the largest file size that could be reasonable handled by your network. It defaults to 1M files, so will probably need to be increased if you are dealing with genome sized datasets. + +Since nginx is more efficient at serving static content, it is best to serve it directly, reducing the load on the Galaxy process and allowing for more effective compression (if enabled), caching, and pipelining. To do so, add the following to your existing `server { }` block, replacing `$GALAXY_ROOT` with the correct path.: + +```nginx +http { + server { + location /static { + alias $GALAXY_ROOT/static; + } + location /static/style { + alias $GALAXY_ROOT/static/style/blue; + } + location /static/scripts { + alias $GALAXY_ROOT/static/scripts; + } + location /favicon.ico { + alias $GALAXY_ROOT/static/favicon.ico; + } + location /robots.txt { + alias $GALAXY_ROOT/static/robots.txt; + } + } +} +``` + +You'll need to ensure that filesystem permissions are set such that the user running your nginx server has access to the Galaxy static/ directory. + + +### Serving Galaxy at a sub directory (such as /galaxy) + +It may be necessary to house Galaxy at an address other than the web server root (`http://www.example.org/galaxy`), instead of `http://www.example.org`). To do this, you need to make the following changes: + +1. In the nginx config, prefix all of the location directives with your prefix, like so: + +```nginx +http { + server { + ... + + location /galaxy { + proxy_pass http://galaxy_app; + proxy_set_header X-Forwarded-Host $host; + proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; + } + location /galaxy/static { + alias $GALAXY_ROOT/static; + } + + ... + } +} +``` + +2. The Galaxy application needs to be aware that it is running with a prefix (for generating URLs in dynamic pages). This is accomplished by configuring a Paste proxy-prefix filter in the `[app:main]` section of `config/galaxy.ini` and restarting Galaxy: + + +```ini +[filter:proxy-prefix] +use = egg:PasteDeploy#prefix +prefix = /galaxy + +[app:main] +filter-with = proxy-prefix +cookie_path = /galaxy +``` + +`cookie_prefix` should be set to prevent Galaxy's session cookies from clobbering each other if running more than one instance of Galaxy in different subdirectories on the same hostname. + + +### SSL + +If you place Galaxy behind a proxy address that uses SSL (i.e., `https://` URLs), edit your galaxy location block (e.g. `location /` when served at the root, or something else like `location /galaxy` when served under a prefix) + +```nginx +location / { + ... + proxy_set_header X-URL-SCHEME https; + ... +} +``` + +Setting `X-URL-SCHEME` makes Galaxy aware of what type of URL it should generate for external sites like Biomart. This should be added to the existing `location / { } ` block if you already have one, and adjusted accordingly if you're serving Galaxy from a subdirectory. + +## Advanced Configuration Topics + +### Compression and caching + +All of Galaxy's static content can be cached on the client side, and everything (including dynamic content) can be compressed on the fly. This will decrease download and page load times for your clients, as well as decrease server load and bandwidth usage. To enable, you'll need nginx gzip support (which is standard unless compiled with `--without-http_gzip_module`), and the following in your `nginx.conf`: + +```nginx +http { + ... + gzip on; + gzip_http_version 1.1; + gzip_vary on; + gzip_comp_level 4; + gzip_proxied any; + gzip_types text/plain text/css application/x-javascript text/xml application/xml text/javascript application/json application/javascript; + gzip_buffers 16 8k; + gzip_disable "MSIE [1-6].(?!.*SV1)"; + ... +} +``` + +For caching, you'll need to add an `expires` directive to the `location /static { }` blocks: + +```nginx +http { + server { + location /static { + alias $GALAXY_ROOT/static; + expires 24h; + } + location /static/style { + alias $GALAXY_ROOT/static/style/blue; + expires 24h; + } + } +} +``` + +The contents of `location /static { }` should be adjusted accordingly if you're serving Galaxy from a subdirectory. + +### Sending files using Nginx + +Galaxy sends files (e.g. dataset downloads) by opening the file and streaming it in chunks through the proxy server. However, this ties up the Galaxy process, which can impact the performance of other operations (see [Production Server Configuration](/src/admin/config/performance/production-server/index.md) for a more in-depth explanation). + +Nginx can assume this task instead and as an added benefit, speed up downloads. This is accomplished through the use of the special `X-Accel-Redirect` header. Dataset security is maintained in this configuration because nginx will still check with Galaxy to ensure that the requesting user has permission to access the dataset before sending it. + +To enable it, add the following to your `nginx.conf`: + +```nginx +http { + server { + location /_x_accel_redirect/ { + internal; + alias /; + } + } +} +``` + +Finally edit your `$GALAXY_ROOT/config/galaxy.ini` and make the following change before restarting Galaxy: + +```ini +[app:main] +nginx_x_accel_redirect_base = /_x_accel_redirect +``` + +For this to work, the user under which your nginx server runs will need read access to Galaxy's `$GALAXY_ROOT/database/files/` directory and its contents. + +### Receiving files using nginx + +Galaxy receives files (e.g. dataset uploads) by streaming them in chunks through the proxy server and writing the files to disk. However, this again ties up the Galaxy process. nginx can assume this task instead and as an added benefit, speed up uploads. This is accomplished through the use of `nginx_upload_module`, a 3rd-party nginx module. + +To enable it, you must first [download](http://www.grid.net.ru/nginx/upload.en.html), compile and install `nginx_upload_module`. This means recompiling nginx. Once done, add the necessary directives to `nginx.conf`: + +```nginx +user galaxy; +http { + server { + ... + + location /_upload { + upload_store $GALAXY_ROOT/database/tmp/upload_store; + upload_pass_form_field ""; + upload_set_form_field " __${upload_field_name}__ is_composite" "true"; + upload_set_form_field " __${upload_field_name}__ keys" "name path"; + upload_set_form_field "${upload_field_name}_name" "$upload_file_name"; + upload_set_form_field "${upload_field_name}_path" "$upload_tmp_path"; + upload_pass_args on; + upload_pass /_upload_done; + } + + location /_upload_done { + set $dst /api/tools; + if ($args ~ nginx_redir=([^&]+)) { + set $dst $1; + } + rewrite "" $dst; + } + + ... + } +} +``` + +Note the `user` directive. To ensure that Galaxy has write permission on the uploaded files, nginx's workers will need to run as the same user as Galaxy. + +Finally edit your `$GALAXY_ROOT/config/galaxy.ini` and make the following change before restarting Galaxy: + +```ini +[app:main] +nginx_upload_store = database/tmp/upload_store +nginx_upload_path = /_upload +``` + +When serving Galaxy with a prefix, as described in the serving Galaxy in a sub-directory section above, you will need to change one line in the `\_upload\_done` section. If your galaxy instance is available from `/galaxy`, then the first line should include this prefix: + +```nginx +set $dst /galaxy/api/tools; +``` + +### Protect Galaxy Reports + +Galaxy can run a separate reports app which gives useful information about your Galaxy instance. +To setup this reports app, have a look here https://docs.galaxyproject.org/en/master/admin/reports.html and here http://galacticengineer.blogspot.de/2015/06/exposing-galaxy-reports-via-nginx-in.html + +After succesfully following the blogpost you will have your galaxy reports available at e.g. http://yourgalaxy/reports +To secure this page to only galaxy administrators, adjust your nginx config with the following snippets: + +```nginx +# Add these snippets to your galaxy nginx configuration to make the reports +# daemon running on the default port 9001 available under the same address +# as your galaxy instance. In addition, the reports app will only be available +# to admins logged in to the galaxy instance. + +upstream reports { + server localhost:9001; +} + +server { # This you should already have. + listen 80; + (..) # The rest of your nginx configuration for galaxy + + location /reports { # the section to make reports available + proxy_pass http://reports; # on the same host as your galaxy at e.g. http://galaxy/reports + proxy_set_header X-Forwarded-Host $host; + proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; + satisfy any; # Restrict access + deny all; + auth_request /auth; + } + location /auth { + # The used galaxy api endpoint is only available to galaxy admins and thus limits the access + # to only logged in admins. + proxy_pass http://localhost/api/configuration/dynamic_tool_confs; + proxy_pass_request_body off; + proxy_set_header Content-Length ""; + proxy_set_header X-Original-URI $request_uri; + } + (..) +} +``` + +### External User Authentication + +- [Nginx for External Authentication](https://galaxyproject.org/admin/config/nginx-external-user-auth/) +- [Built-in Galaxy External Authentication](../authentication.md) diff --git a/doc/source/conf.py b/doc/source/conf.py index c9cad61ce922..180e84fa323f 100644 --- a/doc/source/conf.py +++ b/doc/source/conf.py @@ -19,11 +19,17 @@ # Library to make .md to slideshow from recommonmark.parser import CommonMarkParser +from recommonmark.transform import AutoStructify source_parsers = { '.md': CommonMarkParser, } +# Set GALAXY_DOCS_SKIP_SOURCE=1 to skip building source and release information and +# just build primary documentation. (Quicker to debug issues in most frequently updated +# docs). +SKIP_SOURCE = os.environ.get("GALAXY_DOCS_SKIP_SOURCE", False) == "1" + # REQUIRED GALAXY INCLUDES sys.path.insert(1, os.path.abspath(os.path.join(os.path.dirname(__file__), os.pardir, os.pardir, 'lib'))) @@ -40,7 +46,9 @@ # Add any Sphinx extension module names here, as strings. They can be extensions # coming with Sphinx (named 'sphinx.ext.*') or your custom ones. -extensions = ['sphinx.ext.autodoc', 'sphinx.ext.doctest', 'sphinx.ext.todo', 'sphinx.ext.coverage', 'sphinx.ext.viewcode'] +extensions = ['sphinx.ext.autodoc'] +if not SKIP_SOURCE: + extensions += ['sphinx.ext.doctest', 'sphinx.ext.todo', 'sphinx.ext.coverage', 'sphinx.ext.viewcode'] # Add any paths that contain templates here, relative to this directory. templates_path = ['_templates'] @@ -58,8 +66,12 @@ def dont_skip_init(app, what, name, obj, skip, options): def setup(app): + doc_root = 'https://docs.galaxyproject.org/en/master/' app.connect("autodoc-skip-member", dont_skip_init) - + app.add_config_value('recommonmark_config', { + "url_resolver": lambda url: doc_root + url, + }, True) + app.add_transform(AutoStructify) # The suffix of source filenames. source_suffix = ['.rst', '.md'] @@ -95,7 +107,10 @@ def setup(app): # List of patterns, relative to source directory, that match files and # directories to ignore when looking for source files. -exclude_patterns = [] +if SKIP_SOURCE: + exclude_patterns = ['lib', 'releases'] +else: + exclude_patterns = [] # The reST default role (used for this markup: `text`) to use for all documents. #default_role = None