error with clipper and scipy version

[giardine]

I downloaded the older clipper 1.3 that works with Python 2.7 and installed. clipper -h works as expected. But when I run the pipeline I get an error about scipy.
pkg_resources.DistributionNotFound: The 'scipy>=0.11.0' distribution was not found and is required by clipper

but according to pip
Requirement already satisfied: scipy in local/lib/python2.7/site-packages (1.2.3)

[giardine]

I tried changing the requirement to scipy>=1.2.1 to be closer to the installed version (what was listed under requirements in another file). I ran the test that is with the code and it failed 3 but did not get this error. Running on real data in the workflow I still got the same error. It seems like it may be an environment problem. But it does find clipper so the conda environment is activated. When I run the command from the output manually I do not get the scipy error, but I do get an error about missing gene files.

[giardine]

I fixed the missing gene files, and retried the workflow. Still get the same error about scipy version. I took the clipper command that failed from the workflow output, found the bam file and included the correct location for it and ran the command in the same conda environment as the workflow. When run this way it finishes with no errors. There are potentially 2 problems, whatever is making it not find scipy, and that the bam file is expected to be in /tmp instead of the directory I gave it with the -cache variable.
[job step_clipper] Output of job will be cached in /eclip/K562/hg38/100373/cache/f3ec8926a61e5ee48b0aa0bfe51426d5
[job step_clipper] /eclip/K562/hg38/100373/cache/f3ec8926a61e5ee48b0aa0bfe51426d5$ clipper
--species
GRCh38
--bam
/tmp/tmpKgBuYz/stgbda45e2e-34fa-4f91-b6b5-8f31cb608301/ENCSR981WKN.rep1_clip.A01.r1.fq.genome-mappedSo.rmDupSo.merged.r2.bam
--outfile
ENCSR981WKN.rep1_clip.A01.r1.fq.genome-mappedSo.rmDupSo.merged.r2.peakClusters.bed
...
pkg_resources.DistributionNotFound: The 'scipy>=1.2.1' distribution was not found and is required by clipper

[byee4]

Hi,

Thanks for troubleshooting and sorry that you're having trouble. I agree with you and think there might be an issue with clipper, but in case we can rule out CWL being the issue, are you in addition getting "not found" errors with the BAM file being inside the /tmp directory instead of /eclip? Or are you getting the same "scipy" error each time? If CWL is having trouble finding the BAM file, can you let me know if you've followed the documentation for getting the pipeline deployed/running on AWS found here?

For the clipper problem, it's possible that because that version of clipper uses python2, that some of the dependencies are no longer supported and so conda may have trouble resolving the correct package versions. If you cannot use the latest version of clipper, I've made a singularity image of that version for you to download here. Let me know if that works for you, otherwise I've referenced the issue over at the clipper repo (YeoLab/clipper#97) and can follow up with any further issues over there

[giardine]

I get the same scipy error every time I run it within CWL. I am assuming that if it got past that error there would be one about the bam file. I followed documentation I found on github and ENCODE, but the file you provided the link to does not open for me. I see a bunch of blank pages, then a page 8 with text.

I was using the python 2.7 version because I thought that was what was needed from the Readme. It does not list python 3. I am new to CWL, can it handle 2 versions of Python? Clipper does run for me though, just not within CWL.

[byee4]

Hi,

The python2.7 version of Clipper was used to call peaks for data submitted to the ENCODE DCC. We've since updated Clipper to be python3+ compatible, but either one should work. We've provided container definitions for each pipeline step within each CWL document, so it shouldn't have a problem dealing with multiple versions of python. The link works for me, perhaps you might need to download the file or update the PDF viewer?

https://github.com/YeoLab/eclip/blob/master/documentation/Zero_to_peaks.pdf

[giardine]

So if I create a new conda environment for clipper with python3, where do I indicate which environment to use?

[byee4]

CWL should have container definitions so you don't need to define an environment, however as long as clipper is in your $PATH, it should work

[giardine]

If by container definitions you mean docker, I can not use docker.

[byee4]

Can you use singularity to use the link provided? For example:

singularity exec clipper_1.3.sif clipper -b...

Otherwise, I would recommend installing Python3 Clipper and ensuring that "clipper" can be found by your $PATH

[giardine]

It works from the command line but not within cwl.

[byee4]

I cannot think of a good reason as to why clipper would work outside CWL but gives a missing package error when using the CWL wrapper. Can you please let me know the full cwl command you're using and the complete log file that you get? I will try and reproduce the error on my end and see what is happening.

[giardine]

Sorry, I was on vacation, then getting caught up.
The command is
cwltool --no-container --cachedir /eclip/K562/hg38/100373/cache/ --outdir /eclip/K562/hg38/100373/output/ /eclipWorkflow/eclip/cwl/wf_get_peaks_scatter_pe.cwl paired_end_clip.yaml

I prefer not to post the log to a public server as it contains full paths on our system. I edited the paths above. Besides it is 13G.

[giardine]

I did find one problem. In star.cwl it looks for http://schema.org/docs/schema_org_rdfa.html which no longer exists. Is there a different url that should go there? Could that be why clipper is looking in /tmp for the bam files instead of using the cachedir?

[byee4]

No, I don't think so. I'm confused now as I thought the problem was with a scipy version and not /tmp, so let's make sure clipper can run first inside CWL. Can you send me the log to my email? bay001 (at) health.ucsd.edu. It might also help to know what is inside your environments, or if you're using Singularity to access the tools needed.