Permalink
Browse files

Allow finer grain specification of requirements.

This adapts CWL-style ``SoftwareRequirement`` ``specs`` to solve #1927. Here I'm trying to implement the CWL specification in a way that helps enable the feasibility of Conda packaging in Galaxy. It is a delicate balancing act aimed to upset as many interested parties as I can.

To understand the problem - consider the ``blast+`` requirement found in the Galaxy wrappers. It looks something like this:

```
<requirement type="package" version="2.2.31" name="blast+">
```

Great, that works for Galaxy and Tool Shed packages. It doesn't work for bioconda at all. I think this problem is fairly uncommon - most packages have simple names shared across Debian, Brew, Conda, etc... - but it does happen in some cases that there are inconsistencies. Some people have taken to duplicating the requirement - this is bad and should not be done since they are mutually exclusive and Galaxy will attempt to resolve both.

This introduces the following syntax for tools with profile >= 16.10:

```
<requirement type="package" version="2.2.31" name="blast+">
    <specification uri="https://anaconda.org/bioconda/blast" />
    <specification uri="https://packages.debian.org/sid/ncbi-blast+" version="2.2.31-3" />
</requirement>
```

This allows finer grain resolution of the requirement without sacrificing the abstract name at the top. It allows the name and the version to be adapted by resolvers as needed (hopefully rarely so).

This syntax is the future facing one, but obviously this tool would not work on older Galaxy versions. To remedy this - an alternative syntax can be used for tools targetting Galaxy verions pre-16.10:

```
<requirement type="package" version="2.2" specification_uris="https://anaconda.org/bioconda/blast@2.2.31,https://packages.debian.org/jessie/ncbi-blast+@2.2.29-3">blast+</requirement>
```

This syntax sucks - but it does give newer Galaxies the ability to resolve the specifications without breaking the more simple functionality for older Galaxies.

For more information on the CWL side of this - checkout the discussion on common-workflow-language/cwltool#214. The CWL specification information is defined at http://www.commonwl.org/v1.0/CommandLineTool.html#SoftwarePackage.

Conflicts:
	lib/galaxy/tools/deps/__init__.py
	test/functional/tools/samples_tool_conf.xml
  • Loading branch information...
jmchilton committed Jan 18, 2017
1 parent 29cdd60 commit 81d71d2e740ee07754785306e4448f8425f890bc
@@ -143,6 +143,7 @@ def _requirements_to_dependencies_dict(self, requirements, **kwds):
# Check requirements all at once
all_unmet = len(requirement_to_dependency) == 0
if all_unmet and hasattr(resolver, "resolve_all"):
# TODO: Handle specs.
dependencies = resolver.resolve_all(resolvable_requirements, **kwds)
if dependencies:
assert len(dependencies) == len(resolvable_requirements)
@@ -158,7 +159,17 @@ def _requirements_to_dependencies_dict(self, requirements, **kwds):
continue
if requirement.type in ['package', 'set_environment']:
dependency = resolver.resolve( requirement.name, requirement.version, requirement.type, **kwds )
name = requirement.name
version = requirement.version
specs = requirement.specs
if hasattr(resolver, "find_specification"):
spec = resolver.find_specification(specs)
if spec is not None:
name = spec.short_name
version = spec.version or version
dependency = resolver.resolve( name, version, requirement.type, **kwds )
if require_exact and not dependency.exact:
continue
@@ -10,20 +10,48 @@ class ToolRequirement( object ):
run (for example, a program, package, or library). Requirements can
optionally assert a specific version.
"""
def __init__( self, name=None, type=None, version=None ):
def __init__( self, name=None, type=None, version=None, specs=[] ):
self.name = name
self.type = type
self.version = version
self.specs = specs
def to_dict( self ):
return dict(name=self.name, type=self.type, version=self.version)
specs = [s.to_dict() for s in self.specs]
return dict(name=self.name, type=self.type, version=self.version, specs=specs)
@staticmethod
def from_dict( dict ):
version = dict.get( "version", None )
name = dict.get("name", None)
type = dict.get("type", None)
return ToolRequirement( name=name, type=type, version=version )
specs = [RequirementSpecification.from_dict(s) for s in dict.get("specs", [])]
return ToolRequirement( name=name, type=type, version=version, specs=specs )
class RequirementSpecification(object):
"""Refine a requirement using a URI."""
def __init__(self, uri, version=None):
self.uri = uri
self.version = version
@property
def specifies_version(self):
return self.version is not None
@property
def short_name(self):
return self.uri.split("/")[-1]
def to_dict(self):
return dict(uri=self.uri, version=self.version)
@staticmethod
def from_dict(dict):
uri = dict.get["uri"]
version = dict.get("version", None)
return RequirementSpecification(uri=uri, version=version)
def __eq__(self, other):
return self.name == other.name and self.type == other.type and self.version == other.version
@@ -110,10 +138,29 @@ def parse_requirements_from_xml( xml_root ):
requirements = []
for requirement_elem in requirement_elems:
name = xml_text( requirement_elem )
if "name" in requirement_elem.attrib:
name = requirement_elem.get( "name" )
spec_elems = requirement_elem.findall("specification")
specs = map(specification_from_element, spec_elems)
else:
name = xml_text( requirement_elem )
spec_uris_raw = requirement_elem.attrib.get("specification_uris", "")
specs = []
for spec_uri in spec_uris_raw.split(","):
if not spec_uri:
continue
version = None
if "@" in spec_uri:
uri, version = spec_uri.split("@", 1)
else:
uri = spec_uri
uri = uri.strip()
if version:
version = version.strip()
specs.append(RequirementSpecification(uri, version))
type = requirement_elem.get( "type", DEFAULT_REQUIREMENT_TYPE )
version = requirement_elem.get( "version", DEFAULT_REQUIREMENT_VERSION )
requirement = ToolRequirement( name=name, type=type, version=version )
requirement = ToolRequirement( name=name, type=type, version=version, specs=specs )
requirements.append( requirement )
container_elems = []
@@ -125,6 +172,12 @@ def parse_requirements_from_xml( xml_root ):
return requirements, containers
def specification_from_element(specification_elem):
uri = specification_elem.get("uri", None)
version = specification_elem.get("version", None)
return RequirementSpecification(uri, version)
def container_from_element(container_elem):
identifier = xml_text(container_elem)
type = container_elem.get("type", DEFAULT_CONTAINER_TYPE)
@@ -67,6 +67,34 @@ def _to_requirement(self, name, version=None):
return ToolRequirement(name=name, type="package", version=version)
class SpecificationAwareDependencyResolver:
"""Mix this into a :class:`DependencyResolver` to implement URI specification matching.
Allows adapting generic requirements to more specific URIs - to tailor name
or version to specified resolution system.
"""
__metaclass__ = ABCMeta
@abstractmethod
def find_specification(self, specs):
"""Find closest matching specification for discovered resolver."""
class SpecificationPatternDependencyResolver:
"""Implement the :class:`SpecificationAwareDependencyResolver` with a regex pattern."""
@abstractproperty
def _specification_pattern(self):
"""Pattern of URI to match against."""
def find_specification(self, specs):
pattern = self._specification_pattern
for spec in specs:
if pattern.match(spec.uri):
return spec
return None
class InstallableDependencyResolver:
""" Mix this into a ``DependencyResolver`` and implement to indicate
the dependency resolver can attempt to install new dependencies.
@@ -5,6 +5,7 @@
import logging
import os
import re
import galaxy.tools.deps.installable
@@ -29,6 +30,7 @@
InstallableDependencyResolver,
ListableDependencyResolver,
NullDependency,
SpecificationPatternDependencyResolver,
)
@@ -39,9 +41,10 @@
log = logging.getLogger(__name__)
class CondaDependencyResolver(DependencyResolver, ListableDependencyResolver, InstallableDependencyResolver):
class CondaDependencyResolver(DependencyResolver, ListableDependencyResolver, InstallableDependencyResolver, SpecificationPatternDependencyResolver):
dict_collection_visible_keys = DependencyResolver.dict_collection_visible_keys + ['conda_prefix', 'versionless', 'ensure_channels', 'auto_install']
resolver_type = "conda"
_specification_pattern = re.compile(r"https\:\/\/anaconda.org\/\w+\/\w+")
def __init__(self, dependency_manager, **kwds):
self.versionless = _string_as_bool(kwds.get('versionless', 'false'))
@@ -227,7 +227,7 @@ complete descriptions of the runtime of a tool.
</xs:sequence>
</xs:complexType>
<xs:complexType name="Requirement">
<xs:complexType name="Requirement" mixed="true">
<xs:annotation>
<xs:documentation xml:lang="en"><![CDATA[
@@ -276,20 +276,29 @@ resolver.
]]></xs:documentation>
</xs:annotation>
<xs:simpleContent>
<xs:extension base="xs:string">
<xs:attribute name="type" type="RequirementType" use="required">
<xs:annotation>
<xs:documentation xml:lang="en"> This value defines the which type of the 3rd party module required by this tool. </xs:documentation>
</xs:annotation>
</xs:attribute>
<xs:attribute name="version" type="xs:string">
<xs:annotation>
<xs:documentation xml:lang="en"> For package type requirements this value defines a specific version of the tool dependency. </xs:documentation>
</xs:annotation>
</xs:attribute>
</xs:extension>
</xs:simpleContent>
<xs:sequence>
<xs:element name="specification" minOccurs="0" maxOccurs="unbounded" type="xs:anyType" />
</xs:sequence>
<xs:attribute name="type" type="RequirementType" use="required">
<xs:annotation>
<xs:documentation xml:lang="en"> This value defines the which type of the 3rd party module required by this tool. </xs:documentation>
</xs:annotation>
</xs:attribute>
<xs:attribute name="version" type="xs:string">
<xs:annotation>
<xs:documentation xml:lang="en"> For package type requirements this value defines a specific version of the tool dependency. </xs:documentation>
</xs:annotation>
</xs:attribute>
<xs:attribute name="name" type="xs:string">
<xs:annotation>
<xs:documentation xml:lang="en">Name of requirement (if body of ``requirement`` element contains specification URIs).</xs:documentation>
</xs:annotation>
</xs:attribute>
<xs:attribute name="specification_uris" type="xs:string">
<xs:annotation>
<xs:documentation xml:lang="en">URIs and versions of requirement specification.</xs:documentation>
</xs:annotation>
</xs:attribute>
</xs:complexType>
<xs:complexType name="Container">
<xs:annotation>
@@ -0,0 +1,18 @@
<tool id="requirement_specification_1" name="requirement_specification_1" version="0.1.0" profile="16.10">
<command><![CDATA[
blastn -help > $out_file1 ;
echo "Moo" >> $out_file1 ;
]]></command>
<requirements>
<requirement type="package" version="2.2.31" name="blast+">
<specification uri="https://anaconda.org/bioconda/blast" />
<specification uri="https://packages.debian.org/sid/ncbi-blast+" version="2.2.31-3" />
</requirement>
</requirements>
<inputs>
<param name="input1" type="data" optional="true" />
</inputs>
<outputs>
<data name="out_file1" format="txt" />
</outputs>
</tool>
@@ -0,0 +1,16 @@
<tool id="requirement_specification_2" name="requirement_specification_2" version="0.1.0" profile="16.01">
<command><![CDATA[
blastn -help > $out_file1 ;
echo "Moo" >> $out_file1 ;
]]></command>
<requirements>
<!-- Demonstrate backward-compatible-ish specification_uri syntax. -->
<requirement type="package" version="2.2" specification_uris="https://anaconda.org/bioconda/blast@2.2.31,https://packages.debian.org/jessie/ncbi-blast+@2.2.29-3">blast+</requirement>
</requirements>
<inputs>
<param name="input1" type="data" optional="true" />
</inputs>
<outputs>
<data name="out_file1" format="txt" />
</outputs>
</tool>

0 comments on commit 81d71d2

Please sign in to comment.