Install Dependencies Error

[pdellorusso]

I have been following the tutorial for running the pipeline on SGE using conda. When I install_dependencies I run into the following error and the conda environment is not created:

`Verifying transaction: failed

PaddingError: Placeholder of length '80' too short in package /ifs/scratch/columbia/CSCI/Passegue/Paul_Dellorusso/miniconda3/envs/encode-atac-seq-pipeline-python3/bin/hb-ot-shape-closure.
The package must be rebuilt with conda-build > 2.0.`

I looked into my conda-build version, and it appears to be using conda-build 3.17.5. So I don't see how that could be contributing to this error (unless the read and write filter support indicated in the output matters?):

`ha4c6n8:atac-seq-pipeline(master)] conda build -V
read filter "zstd" is not supported
write filter "zstd" is not supported
conda-build 3.17.5`

I deleted the pipeline and re-started the tutorial from scratch, and still ran into the same error. This is the entire run information:

`ha4c6n8:atac-seq-pipeline(master)] bash conda/uninstall_dependencies.sh
/ifs/scratch/columbia/CSCI/Passegue/Paul_Dellorusso/bin/miniconda3/bin/conda
=== Found Conda (conda 4.5.12).
=== Pipeline's py3 Conda env (encode-atac-seq-pipeline-python3) does not exist or has already been removed.
=== Pipeline's Conda env (encode-atac-seq-pipeline) does not exist or has already been removed.
ha4c6n8:atac-seq-pipeline(master)] bash conda/install_dependencies.sh 
/ifs/scratch/columbia/CSCI/Passegue/Paul_Dellorusso/bin/miniconda3/bin/conda
=== Found Conda (conda 4.5.12).
=== Installing packages for python3 env...
Solving environment: done

## Package Plan ##

  environment location: /ifs/scratch/columbia/CSCI/Passegue/Paul_Dellorusso/miniconda3/envs/encode-atac-seq-pipeline-python3

  added / updated specs: 
    - bedtools==2.26.0
    - idr==2.0.4.2
    - java-jdk==8.0.92
    - libgcc==5.2.0
    - matplotlib==1.5.1
    - ncurses==6.1
    - numpy==1.11.3
    - openblas==0.2.20
    - python==3.5.0
    - tabix==0.2.6


The following NEW packages will be INSTALLED:

    bedtools:        2.26.0-0                              bioconda   
    blas:            1.1-openblas                          conda-forge
    ca-certificates: 2018.11.29-ha4d7672_0                 conda-forge
    cairo:           1.14.6-0                              conda-forge
    certifi:         2018.8.24-py35_1001                   conda-forge
    cycler:          0.10.0-py_1                           conda-forge
    fontconfig:      2.11.1-6                              conda-forge
    freetype:        2.6.3-1                               conda-forge
    gettext:         0.19.8.1-h5e8e0c9_1                   conda-forge
    glib:            2.56.2-h464dc38_1                     conda-forge
    harfbuzz:        1.0.6-1                               conda-forge
    icu:             56.1-4                                conda-forge
    idr:             2.0.4.2-py35h24bf2e0_0                bioconda   
    java-jdk:        8.0.92-1                              bioconda   
    jpeg:            9c-h470a237_1                         conda-forge
    libffi:          3.2.1-hfc679d8_5                      conda-forge
    libgcc:          5.2.0-0                               conda-forge
    libgcc-ng:       7.2.0-hdf63c60_3                      conda-forge
    libgfortran:     3.0.0-1                               conda-forge
    libiconv:        1.15-h470a237_3                       conda-forge
    libpng:          1.6.36-ha92aebf_0                     conda-forge
    libstdcxx-ng:    7.2.0-hdf63c60_3                      conda-forge
    libtiff:         4.0.6-5                               conda-forge
    libxml2:         2.9.3-8                               conda-forge
    matplotlib:      1.5.1-np111py35_4                     conda-forge
    ncurses:         6.1-hfc679d8_2                        conda-forge
    numpy:           1.11.3-py35_blas_openblashd3ea46f_205 conda-forge [blas_openblas]
    openblas:        0.2.20-8                              conda-forge
    openssl:         1.0.2p-h470a237_1                     conda-forge
    pango:           1.40.1-0                              conda-forge
    pcre:            8.41-hfc679d8_3                       conda-forge
    pip:             18.0-py35_1001                        conda-forge
    pixman:          0.34.0-h470a237_3                     conda-forge
    pyparsing:       2.3.0-py_0                            conda-forge
    pyqt:            4.11.4-py35_3                         conda-forge
    python:          3.5.0-1                                          
    python-dateutil: 2.7.5-py_0                            conda-forge
    pytz:            2018.7-py_0                           conda-forge
    qt:              4.8.7-6                               conda-forge
    readline:        6.2-2                                            
    scipy:           1.1.0-py35_blas_openblash7943236_201  conda-forge [blas_openblas]
    setuptools:      40.4.3-py35_0                         conda-forge
    sip:             4.18-py35_1                           conda-forge
    six:             1.11.0-py35_1                         conda-forge
    sqlite:          3.19.3-1                              conda-forge
    tabix:           0.2.6-ha92aebf_0                      bioconda   
    tk:              8.5.19-2                              conda-forge
    wheel:           0.32.0-py35_1000                      conda-forge
    xz:              5.0.5-1                               conda-forge
    zlib:            1.2.11-h470a237_3                     conda-forge

Preparing transaction: done
Verifying transaction: failed

PaddingError: Placeholder of length '80' too short in package /ifs/scratch/columbia/CSCI/Passegue/Paul_Dellorusso/miniconda3/envs/encode-atac-seq-pipeline-python3/bin/hb-ot-shape-closure.
The package must be rebuilt with conda-build > 2.0.`

Thanks for any guidance.

Paul

[leepc12]

This is Conda's issue. conda/conda#4088
Full path for Conda's installation should not be long (~80). Try installing Conda on your home (e.g. /home/your_username, whose path is much shorter than /ifs/scratch/columbia/CSCI/Passegue/Paul_Dellorusso/miniconda3).

Or try with SGE + singularity method.

[pdellorusso]

Thanks for the quick reply.

My home directory on the cluster is: /ifs/scratch/columbia/CSCI/Passegue/Paul_Dellorusso/

Is there any way around this using Conda? I would prefer it over singularity if possible. Otherwise I will try singularity if there is nothing to do about it

[leepc12]

There might be an workaround but I am not sure. In my experience, I could increase maximum path ($CONDA_PREFIX) length to 255 from 80. But It's long long time ago so I don't remember how I fixed it.

Please install the latest Conda and read through similar issues on Conda (like conda/conda-build#1484).

[leepc12]

Closing this.

[pdellorusso]

I've been trying to make this work as I can't get singularity installed on my cluster (working on this issue in a parallel).

I've searched through the similar issues but I can't follow along with what the suggested workarounds are. If you happen to know a specific set of commands that will make this work I would appreciate it. I'm not sure if I need to directly edit the install dependencies script to do so?

[leepc12]

This will be fixed in the next release v1.1.5. That limit will increase to 255 from 80. You may need to uninstall (conda/uninstall_dependencies.sh)/re-install(conda/install_dependencies.sh) Conda dependencies.

[pdellorusso]

great thanks a lot, I will try again when v1.1.5 is released

[leepc12]

https://github.com/ENCODE-DCC/atac-seq-pipeline/blob/master/conda/requirements_py3.txt#L11

You can actually fix it by commenting out version information in line 11 of requirements_py3.txt.
matplotlib ==1.5.1 -> matplotlib #==1.5.1. Please uninstall (bash/uninstall_dependencies.sh)/reinstalling(bash/install_dependencies.sh) dependencies after this.

[pdellorusso]

this worked! thank you!