Merge remote-tracking branch 'origin/release_18.01' into usegalaxy

doc/source/admin/nginx.md

@@ -296,7 +296,7 @@ nginx did not support shared modules, and the upload module is not yet shared-co
 and complicated process, the Galaxy Committers team maintains (for some platforms) versions of nginx modified from their
 upstream package sources (APT, EPEL, etc.) to include the upload module:
 
-- [Ubuntu (PPA)](https://launchpad.net/~galaxyproject/+archive/ubuntu/nginx)
+- [Ubuntu (PPA)](https://launchpad.net/~galaxyproject/+archive/ubuntu/nginx) (Be sure to install `nginx-extras`, not `nginx`)
 - [Enterprise Linux](https://depot.galaxyproject.org/yum/)
 
 To contribute support for additional platforms, please see the [Galaxy

doc/source/releases/18.01_announce.rst

@@ -14,10 +14,8 @@ Highlights
 
 **Web Server and Configuration**
   The default web server used by Galaxy has changed from Paste to `uWSGI <https://uwsgi-docs.readthedocs.io/>`__ and the default configuration file for Galaxy is now ``config/galaxy.yml`` instead of ``config/galaxy.ini``.
-  uWSGI is more production ready and allows Galaxy to scale better in its default
-  configuration. In the future uWSGI will allow Galaxy to setup GIE proxies without additional
-  configuration and use modern web technologies such as web sockets.
-  Read more about the server, configuration, and documentation changes in the `uWSGI details`_ section of this document.
+  To minimize the impact of this change on existing Galaxy instances, if a Galaxy has a ``galaxy.ini`` file configured, it will continue to use Paste by default unless additional steps are taken by the administrator.
+  uWSGI is more production ready and allows Galaxy to scale better in its default configuration. Read more about the server, configuration, and documentation changes in the `uWSGI details`_ section of this document.
 
 **Dataset Collection Usability**
   This release has significantly improved the usability of Galaxy dataset collections. Dozens of improvements
@@ -78,14 +76,29 @@ To get a new Galaxy repository run:
 To update an existing Galaxy repository run:
   .. code-block:: shell
 
-      $ git checkout release_18.01 && git pull --ff-only origin release_18.01
+      $ git fetch origin && git checkout release_18.01 && git pull --ff-only origin release_18.01
 
 See the `community hub <https://galaxyproject.org/develop/source-code/>`__ for additional details regarding the source code locations.
 
 
 Security
 ========
 
+Unsecure GenomeSpace token exposure
+-----------------------------------
+Tracked as ``GX-2018-0002``.
+
+We have found and fixed a medium-level security issue concering the GenomeSpace importer/exporter tools that were updated in the Galaxy release 17.09. These tools did not handle the GenomeSpace access token securely and stored it as a job parameter which made it accessible to anybody with access to the datasets created by these tools.
+This means that any user that used a GenomeSpace token to access these tools and subsequently shared the output dataset (or history that contains it) with another user shared their GenomeSpace token also.
+
+These tools are both included in the ``tool_conf.xml.sample`` and are therefore *enabled on every new Galaxy by default*.
+
+Administrators please see the `GenomeSpace security sanitization`_ section of this document for the details on how to sanitize the tokens stored in the Galaxy database created prior to this fix.
+
+The vulnerability has been resolved by removing the token functionality until a proper implementation is in place. The GenomeSpace tools continue to work using the OpenID authentication as before.
+
+The fix for this issue has been applied back to Galaxy release 17.09 and can be found in this `pull request <https://github.com/galaxyproject/galaxy/pull/5631>`__.
+
 Breaking Changes
 ================
 
@@ -113,17 +126,16 @@ In addition to that, the PlantTribes datatypes have been commented out in the so
 
 uWSGI details
 =============
-
-To minimize the impact of this change on existing Galaxy instances, if a Galaxy has a ``galaxy.ini``
-file configured, it will continue to use Paste by default unless additional steps are
-taken by the administrator (Galaxy can be forced to start under uWSGI even with an older configuration file
-by setting ``APP_WEBSERVER=uwsgi`` in the environment). As part of the transition to YAML-based
+Galaxy can be forced to start under uWSGI even with an older configuration file
+by setting ``APP_WEBSERVER=uwsgi`` in the environment. As part of the transition to YAML-based
 configuration files, we have implemented a schema to validate Galaxy configuration files. Run
 ``make config-validate`` from Galaxy's root directory to validate a schema and ``make config-lint`` to
 check for best practices. While there is no need to convert your configuration file (``galaxy.ini`` hasn't
 been deprecated), you can run ``make config-convert-dry-run`` and ``make config-convert``
 to respectively test and perform the conversion of an ``ini`` configuration file to a YAML one.
 
+In the future uWSGI will allow Galaxy to setup GIE proxies without additional configuration and use modern web technologies such as web sockets.
+
 These are big changes that affect many parts of Galaxy's administration documentation and makes
 this documentation very dependent on which Galaxy version they are targeting. To address this, we have
 moved a significant amount of administration documentation into Galaxy's code
@@ -140,8 +152,74 @@ Release Notes
 .. include:: 18.01.rst
    :start-after: announce_start
 
-Security patch details
-======================
+GenomeSpace security sanitization
+=================================
+
+Outputs of these tools may require sanitization: ``genomespace_importer`` and ``genomespace_exporter``
+
+The following SQL commands will help you **identify** the datasets in your Galaxy's database.
+
+Finding bad GenomeSpace importer params:
+
+.. code-block:: sql
+
+  SELECT j.id,
+         j.create_time,
+         j.user_id,
+         jp.value
+  FROM job_parameter jp
+  JOIN job j ON j.id = jp.job_id
+  WHERE jp.job_id IN
+      (SELECT id
+       FROM job
+       WHERE tool_id = 'genomespace_importer')
+    AND jp.name = 'URL'
+    AND jp.value LIKE '%^%'
+  ORDER BY j.id DESC;
+
+Finding bad GenomeSpace exporter params:
+
+.. code-block:: sql
+
+  SELECT j.id,
+         j.create_time,
+         j.user_id,
+         jp.value
+  FROM job_parameter jp
+  JOIN job j ON j.id = jp.job_id
+  WHERE jp.job_id IN
+      (SELECT id
+       FROM job
+       WHERE tool_id = 'genomespace_exporter')
+    AND jp.name = 'genomespace_browser'
+    AND jp.value LIKE '%^%'
+  ORDER BY j.id DESC;
+
+The following SQL commands will help you **sanitize** the datasets in your Galaxy's database.
+
+Sanitizing GenomeSpace importer params:
+
+.. code-block:: sql
+
+  UPDATE job_parameter jp
+  SET value = split_part(jp.value, '^', 1) || '"'
+  FROM job j
+  WHERE jp.job_id = j.id
+    AND j.tool_id = 'genomespace_importer'
+    AND jp.name = 'URL'
+    AND jp.value LIKE '%^%';
+
+Sanitizing GenomeSpace exporter params:
+
+.. code-block:: sql
+
+  UPDATE job_parameter jp
+  SET value = split_part(jp.value, '^', 1) || '"'
+  FROM job j
+  WHERE jp.job_id = j.id
+    AND j.tool_id = 'genomespace_exporter'
+    AND jp.name = 'genomespace_browser'
+    AND jp.value LIKE '%^%';
 
 .. include:: _thanks.rst
 

lib/galaxy/managers/libraries.py

@@ -145,7 +145,7 @@ def list(self, trans, deleted=False):
             allowed_library_manage_ids = set([])
             for action in all_actions:
                 if action.action == library_access_action:
-                    accessible_restricted_library_ids.add(action.library_id)
+                    accessible_restricted_library_ids.append(action.library_id)
                 if action.action == library_add_action:
                     allowed_library_add_ids.add(action.library_id)
                 if action.action == library_modify_action:

lib/galaxy/model/__init__.py

@@ -4075,7 +4075,7 @@ def add_output(self, workflow_output, step, output_object):
             output_assoc.dataset_collection = output_object
             self.output_dataset_collections.append(output_assoc)
         else:
-            raise Exception("Uknown output type encountered")
+            raise Exception("Unknown output type encountered")
 
     def to_dict(self, view='collection', value_mapper=None, step_details=False):
         rval = super(WorkflowInvocation, self).to_dict(view=view, value_mapper=value_mapper)
@@ -4199,7 +4199,7 @@ def add_output(self, output_name, output_object):
             output_assoc.output_name = output_name
             self.output_dataset_collections.append(output_assoc)
         else:
-            raise Exception("Uknown output type encountered")
+            raise Exception("Unknown output type encountered")
 
     @property
     def jobs(self):

lib/galaxy/security/__init__.py

@@ -777,15 +777,15 @@ def get_private_user_role(self, user, auto_create=False):
         return role
 
     def get_role(self, name, type=None):
-        type = type or self.model.Role.types.ADMIN
+        type = type or self.model.Role.types.SYSTEM
         # will raise exception if not found
         return self.sa_session.query(self.model.Role) \
             .filter(and_(self.model.Role.table.c.name == name,
                      self.model.Role.table.c.type == type)) \
             .one()
 
     def create_role(self, name, description, in_users, in_groups, create_group_for_role=False, type=None):
-        type = type or self.model.Role.types.ADMIN
+        type = type or self.model.Role.types.SYSTEM
         role = self.model.Role(name=name, description=description, type=type)
         self.sa_session.add(role)
         # Create the UserRoleAssociations

lib/galaxy/tools/actions/__init__.py

@@ -7,7 +7,8 @@
 
 from galaxy import model
 from galaxy.exceptions import ObjectInvalid
-from galaxy.model import LibraryDatasetDatasetAssociation
+from galaxy.jobs.actions.post import ActionBox
+from galaxy.model import LibraryDatasetDatasetAssociation, WorkflowRequestInputParameter
 from galaxy.tools.parameters import update_dataset_ids
 from galaxy.tools.parameters.basic import DataCollectionToolParameter, DataToolParameter, RuntimeValue
 from galaxy.tools.parameters.wrapped import WrappedParameters
@@ -36,7 +37,8 @@ class ToolAction(object):
     been converted and validated).
     """
 
-    def execute(self, tool, trans, incoming={}, set_output_hid=True):
+    def execute(self, tool, trans, incoming=None, set_output_hid=True):
+        incoming = incoming or {}
         raise TypeError("Abstract method")
 
 
@@ -213,12 +215,13 @@ def _collect_inputs(self, tool, trans, incoming, history, current_user_roles):
 
         return history, inp_data, inp_dataset_collections, preserved_tags
 
-    def execute(self, tool, trans, incoming={}, return_job=False, set_output_hid=True, history=None, job_params=None, rerun_remap_job_id=None, execution_cache=None, dataset_collection_elements=None, completed_job=None):
+    def execute(self, tool, trans, incoming=None, return_job=False, set_output_hid=True, history=None, job_params=None, rerun_remap_job_id=None, execution_cache=None, dataset_collection_elements=None, completed_job=None):
         """
         Executes a tool, creating job and tool outputs, associating them, and
         submitting the job to the job queue. If history is not specified, use
         trans.history as destination for tool's output datasets.
         """
+        incoming = incoming or {}
         self._check_access(tool, trans)
         app = trans.app
         if execution_cache is None:
@@ -491,7 +494,6 @@ def handle_output(name, output, hidden=None):
                                      rerun_remap_job_id=rerun_remap_job_id,
                                      current_job=job,
                                      out_data=out_data)
-
         log.info("Setup for job %s complete, ready to flush %s" % (job.log_str(), job_setup_timer))
 
         job_flush_timer = ExecutionTimer()
@@ -544,6 +546,15 @@ def _remap_job_on_rerun(self, trans, galaxy_session, rerun_remap_job_id, current
             # Duplicate PJAs before remap.
             for pjaa in old_job.post_job_actions:
                 current_job.add_post_job_action(pjaa.post_job_action)
+            if old_job.workflow_invocation_step:
+                replacement_dict = {}
+                for parameter in old_job.workflow_invocation_step.workflow_invocation.input_parameters:
+                    if parameter.type == WorkflowRequestInputParameter.types.REPLACEMENT_PARAMETERS:
+                        replacement_dict[parameter.name] = parameter.value
+                for pja in old_job.workflow_invocation_step.workflow_step.post_job_actions:
+                    # execute immediate actions here, with workflow context.
+                    if pja.action_type in ActionBox.immediate_actions:
+                        ActionBox.execute(trans.app, trans.sa_session, pja, current_job, replacement_dict)
             for p in old_job.parameters:
                 if p.name.endswith('|__identifier__'):
                     current_job.parameters.append(p.copy())
@@ -598,7 +609,7 @@ def _wrapped_params(self, trans, tool, incoming, input_datasets=None):
 
     def _get_on_text(self, inp_data):
         input_names = []
-        for name, data in reversed(inp_data.items()):
+        for data in reversed(inp_data.values()):
             if getattr(data, "hid", None):
                 input_names.append('data %s' % data.hid)
 
@@ -782,7 +793,7 @@ def create_collection(self, output, name, **element_kwds):
         if "elements" in element_kwds:
             elements = element_kwds["elements"]
             if hasattr(elements, "items"):  # else it is ELEMENTS_UNINITIALIZED object.
-                for key, value in elements.items():
+                for value in elements.values():
                     # Either a HDA (if) or a DatasetCollection (the else)
                     if getattr(value, "history_content_type", None) == "dataset":
                         assert value.history is not None

lib/galaxy/version.py

@@ -1,3 +1,3 @@
 VERSION_MAJOR = "18.01"
-VERSION_MINOR = "rc1"
+VERSION_MINOR = None
 VERSION = VERSION_MAJOR + ('.' + VERSION_MINOR if VERSION_MINOR else '')

lib/galaxy/workflow/run_request.py

@@ -44,15 +44,15 @@ class WorkflowRunConfig(object):
     def __init__(self, target_history,
                  replacement_dict,
                  copy_inputs_to_history=False,
-                 inputs={},
-                 param_map={},
+                 inputs=None,
+                 param_map=None,
                  allow_tool_state_corrections=False,
                  use_cached_job=False):
         self.target_history = target_history
         self.replacement_dict = replacement_dict
         self.copy_inputs_to_history = copy_inputs_to_history
-        self.inputs = inputs
-        self.param_map = param_map
+        self.inputs = inputs or {}
+        self.param_map = param_map or {}
         self.allow_tool_state_corrections = allow_tool_state_corrections
         self.use_cached_job = use_cached_job
 
@@ -168,7 +168,8 @@ def _flatten_step_params(param_dict, prefix=""):
     return new_params
 
 
-def _get_target_history(trans, workflow, payload, param_keys=[], index=0):
+def _get_target_history(trans, workflow, payload, param_keys=None, index=0):
+    param_keys = param_keys or []
     history_name = payload.get('new_history_name', None)
     history_id = payload.get('history_id', None)
     history_param = payload.get('history', None)

lib/tool_shed/util/hg_util.py

@@ -62,7 +62,7 @@ def clone_repository(repository_clone_url, repository_file_dir, ctx_rev):
         error_message = 'Error cloning repository: %s' % e
         if isinstance(e, subprocess.CalledProcessError):
             error_message += "\nOutput was:\n%s" % stdouterr
-        log.debug(error_message)
+        log.error(error_message)
         return False, error_message
 
 

scripts/cleanup_datasets/admin_cleanup_datasets.py

@@ -53,6 +53,8 @@
 from six.moves import configparser
 from sqlalchemy import and_, false
 
+sys.path.insert(1, os.path.abspath(os.path.join(os.path.dirname(__file__), os.pardir, os.pardir, 'lib')))
+
 import galaxy.config
 import galaxy.model.mapping
 import galaxy.util

scripts/cleanup_datasets/update_dataset_size.py

@@ -10,6 +10,8 @@
 
 from six.moves import configparser
 
+sys.path.insert(1, os.path.abspath(os.path.join(os.path.dirname(__file__), os.pardir, os.pardir, 'lib')))
+
 import galaxy.app
 
 assert sys.version_info[:2] >= (2, 6)
@@ -24,7 +26,7 @@ def usage(prog):
 
 
 def main():
-    if len(sys.argv) != 1 or sys.argv[1] == "-h" or sys.argv[1] == "--help":
+    if len(sys.argv) != 2 or sys.argv[1] == "-h" or sys.argv[1] == "--help":
         usage(sys.argv[0])
         sys.exit()
     ini_file = sys.argv.pop(1)