SoftwareRequirement fails if in a Python3 virtualenv

[mr-c]

From https://www.biostars.org/p/344955/

Expected Behavior

Tell us what should happen

Actual Behavior

openjdk-8.0.12 100% |##############################################################| Time: 0:00:01  59.85 MB/s
rtg-tools-3.8. 100% |##############################################################| Time: 0:00:00  15.25 MB/s
Extracting packages ...
[      COMPLETE      ]|#################################################################################| 100%
Linking packages ...
[      COMPLETE      ]|#################################################################################| 100%
#
# To activate this environment, use:
# > source activate __rtg-tools@3.8.2
#
# To deactivate this environment, use:
# > source deactivate __rtg-tools@3.8.2
#

Exception while running job
Traceback (most recent call last):
  File "/nethome/usevani/miniconda2/lib/python2.7/site-packages/cwltool/job.py", line 278, in _execute
    name=self.name
  File "/nethome/usevani/miniconda2/lib/python2.7/site-packages/cwltool/job.py", line 575, in _job_popen
    mode="wb") as job_file:
ValueError: binary mode doesn't take an encoding argument
[job rtg.cwl] completed permanentFail
{}
Final process status is permanentFail

Workflow Code

cwlVersion: v1.0
class: CommandLineTool
baseCommand: rtg

requirements:
  EnvVarRequirement:
    envDef:
      TOIL_GRIDENGINE_PE: smp

hints:
  SoftwareRequirement:
    packages:
    - package: 'rtg'
      version:
      - '3.8.2'
      specs:
      - https://anaconda.org/bioconda/rtg-tools

arguments:
- valueFrom: vcfeval
  position: 2
- valueFrom: 20G
  position: 1
  prefix: RTG_MEM=
  separate: false
- valueFrom: "$(runtime.outdir)/out"
  prefix: -o
  position: 3
- valueFrom: $(runtime.cores)
  prefix: -T
  position: 4

inputs:
  rtg_reference:
    type: Directory
    inputBinding:
      position: 5
      prefix: -t
  gold_standard:
    type: File
    inputBinding:
      position: 6
      prefix: -b
    secondaryFiles:
    - .tbi
    doc: Genome-In-A-Bottle gold start VCF.
  input:
    type: File
    inputBinding:
      position: 7
      prefix: -c
    secondaryFiles:
    - .tbi
    doc: input compressed and index VCF
  all_records:
    type: boolean
    inputBinding:
      position: 8
      prefix: --all-records
  sample_names:
    type: string
    inputBinding:
      prefix: --sample=
      separate: false
      position: 9
  evaluation_regions:
    type: File
    inputBinding:
      prefix: --evaluation-regions=
      separate: false
      position: 10

outputs:
  output:
    type: Directory
    outputBinding:
      glob: "$(runtime.outdir)/out"

Full Traceback

Paste the full traceback in case there is an exception
Run the workflow with ``--debug`` flag for more verbose logging 

Your Environment

• cwltool version: 1.0.20180809224403

[usevani]

This is working in cwltool version 1.0.20181012180214 using Python 3.5.5. In Python 2.7, it's working using the --beta-conda-dependencies flag.

Here is the error when using --beta-dependency-resolvers-configuration using modules.

[job rtg.cwl] /tmp/tmp8aejiH$ rtg \
    RTG_MEM=20G \
    vcfeval \
    -o \
    /tmp/tmp8aejiH/out \
    -T \
    1 \
    -t \
    /tmp/tmpKVrxeO/stgafaade76-79d5-4600-b539-0f0c9d2a4dfe/GRCh38_full_analysis_set_plus_decoy_hla.sdf \
    -b \
    /tmp/tmpKVrxeO/stge929c0b4-1e54-4d47-9a20-393fe327b3d1/HG001_GRCh38_GIAB_highconf_CG-IllFB-IllGATKHC-Ion-10X-SOLID_CHROM1-X_v.3.3.2_highconf_PGandRTGphasetransfer.vcf.gz \
    -c \
    /tmp/tmpKVrxeO/stgd8c1a267-3c9c-4494-88ab-9403c7498306/rtg_test.vcf.gz \
    --all-records \
    --sample=HG001,NA12878 \
    --evaluation-regions=/tmp/tmpKVrxeO/stg5c829e7e-cfc4-4efe-8e6a-91da20d2642b/HG001_GRCh38_GIAB_highconf_CG-IllFB-IllGATKHC-Ion-10X-SOLID_CHROM1-X_v.3.3.2_highconf_nosomaticdel_noCENorHET7.bed
Exception while running job
Traceback (most recent call last):
  File "/nethome/usevani/miniconda2/lib/python2.7/site-packages/cwltool/job.py", line 293, in _execute
    job_script_contents = builder.build_job_script(commands)
  File "/nethome/usevani/miniconda2/lib/python2.7/site-packages/cwltool/builder.py", line 196, in build_job_script
    return build_job_script_method(self, commands)
  File "/nethome/usevani/miniconda2/lib/python2.7/site-packages/cwltool/software_requirements.py", line 75, in build_job_script
    dependencies, job_directory=builder.tmpdir))
  File "/nethome/usevani/miniconda2/lib/python2.7/site-packages/galaxy/tools/deps/__init__.py", line 111, in dependency_shell_commands
    requirement_to_dependency = self.requirements_to_dependencies(requirements, **kwds)
  File "/nethome/usevani/miniconda2/lib/python2.7/site-packages/galaxy/tools/deps/__init__.py", line 120, in requirements_to_dependencies
    requirement_to_dependency = self._requirements_to_dependencies_dict(requirements, **kwds)
  File "/nethome/usevani/miniconda2/lib/python2.7/site-packages/galaxy/tools/deps/__init__.py", line 168, in _requirements_to_dependencies_dict
    log.debug(dependency.resolver_msg)
  File "/nethome/usevani/miniconda2/lib/python2.7/site-packages/galaxy/tools/deps/resolvers/__init__.py", line 258, in resolver_msg
    return "Using dependency %s version %s of type %s" % (self.name, self.version, self.dependency_type)
AttributeError: 'ModuleDependency' object has no attribute 'name'
[job rtg.cwl] completed permanentFail
[job rtg.cwl] {}
[job rtg.cwl] Removing input staging directory /tmp/tmpKVrxeO
[job rtg.cwl] Removing temporary directory /tmp/tmphfpneO
{}
Final process status is permanentFail

[davidjsherman]

The specific error AttributeError: 'ModuleDependency' object has no attribute 'name' seems to come from an ancient bug in galaxy.tools.deps: in we log name and version https://github.com/galaxyproject/galaxy/blob/952847aad1eb00f6150761127cce4456e2f880a5/lib/galaxy/tools/deps/resolvers/__init__.py#L257 but for an environment modules resolver the attributes are module_name and module_version https://github.com/galaxyproject/galaxy/blob/d2b4dd1e655324e88f367e7f146458357972beb7/lib/galaxy/tools/deps/resolvers/modules.py#L130-L131

[davidjsherman]

Apparently this has been fixed in galaxy-lib 18.09, so a fix might be to bump

cwltool/setup.py

Line 68 in dda97d7

'deps': ["galaxy-lib >= 17.09.3"]

[mr-c]

@davidjsherman Thank you for your research! Since the dependency has no upper bound, it seems that 18.09 did not fix it yet.

[davidjsherman]

The precise version where this was fixed is galaxy-lib 17.9.9 (2017-09-27), incorporating galaxyproject/galaxy@69a8700

[davidjsherman]

This is off-topic for this issue (virtualenv and Python 3), but to close off my comment above, the bug in resolver_msg reported by @usevani #954 (comment) is reported in #983 and corrected by PR #982.

PR #982 adds a test for resolving dependencies using Environment Modules and bumps galaxy-lib to minimum 17.09.9.