PySlice_Unpack error #16

cooketho · 2019-01-27T04:53:34Z

oyster.mk main fails on the test data set with the following error. Quick google search reveals similar errors in conda, sometimes reported fixed by using newer Python.

======= SPAdes pipeline finished.

SPAdes log can be found here: /home/tc/sampledata/assemblies/test.spades_k55/spades.log

Thank you for using SPAdes!
Traceback (most recent call last):
File "/lab/pr/tc/ORP/software/transabyss/transabyss", line 18, in
from utilities.adj_utils import has_edges
File "/lab/pr/tc/ORP/software/transabyss/utilities/adj_utils.py", line 6, in
import igraph
File "/lab/pr/tc/ORP/software/anaconda/install/envs/orp_v2/lib/python3.6/site-packages/igraph/init.py", line 34, in
from igraph._igraph import *
ImportError: /lab/pr/tc/ORP/software/anaconda/install/envs/orp_v2/lib/python3.6/site-packages/igraph/_igraph.cpython-36m-x86_64-linux-gnu.so: undefine
d symbol: PySlice_Unpack
make: *** [/home/tc/sampledata/assemblies/test.transabyss.fasta] Error 1

cooketho · 2019-01-27T07:38:44Z

Fixed it by editing py36_env.yml to specify a newer python version (>=3.6.1), adding git to dependencies, and getting busco directly from the gitlab repository (v3.1) instead of the out-of-date anaconda repository (v3.02). Don't know if this latter bit was necessary, but I saw in the commits that there were problems with busco when using python>3.6. Anyway, it didn't give me any problems on the included test data. Here is my new py36_env.yml:

name: orp_v2
channels:
  - bioconda
  - conda-forge
  - defaults
  - r
  - antoined
dependencies:
  - python>=3.6.1
  - git
  - zsh
  - ruby
  - make
  - glib
  - zlib==1.2.11=0
  - libgcc>=7.2.0
  - salmon>=0.10.2
  - seqtk
  - trinity
  - spades
  - blast
  - hmmer
  - samtools
  - blat
  - parallel
  - last
  - rcorrector==1.0.3
  - mcl
  - abyss>=2.0.1
  - python-igraph
  - diamond
  - cd-hit
  - bwa
  - pip:
    - biopython
    - cvxopt
    - cython
    - numpy
    - pandas
    - pip
    - scipy
    - bashplotlib
    - "git+https://gitlab.com/ezlab/busco"

macmanes · 2019-01-28T12:31:20Z

So just to confirm, the test dataset now runs properly, and gives an output something like

 6| #
 5| ##
 4| ###
 3| ###
 2| ###       #
 1| ### ###   #
   -----------

------------------------------------
|             Summary              |
------------------------------------
|         observations: 20         |
|       min value: -1.000000       |
|         mean : -0.986450         |
|       max value: -0.935000       |
------------------------------------


*****  See the following link for interpretation *****
*****  https://oyster-river-protocol.readthedocs.io/en/latest/strandexamine.html *****



*****  QUALITY REPORT FOR: test4 using the ORP version 2.1.0 ****
*****  THE ASSEMBLY CAN BE FOUND HERE: /mnt/lustre/macmaneslab/macmanes/fastq_files/assemblies/test4.ORP.fasta ****

*****  BUSCO SCORE ~~~~~>           	C:0.0%[S:0.0%,D:0.0%],F:0.3%,M:99.7%,n:303
*****  TRANSRATE SCORE ~~~~~>           0.42269
*****  TRANSRATE OPTIMAL SCORE ~~~~~>   0.57644
*****  UNIQUE GENES ORP ~~~~~>          39
*****  UNIQUE GENES TRINITY ~~~~~>      31
*****  UNIQUE GENES SPADES55 ~~~~~>     22
*****  UNIQUE GENES SPADES75 ~~~~~>     25
*****  UNIQUE GENES TRANSABYSS ~~~~~>   36

cooketho · 2019-01-28T14:10:04Z

Here is the output:

[bam_sort_core] merging from 0 files and 10 in-memory blocks...
-parsing file: test.sorted.bam
-done parsing file, examining orientations of reads.

 7|  #         
 6| ##         
 5| ##         
 4| ##         
 3| ###        
 2| ###       #
 1| ###  ##   #
   -----------

------------------------
|       Summary        |
------------------------
|   observations: 20   |
| min value: -1.000000 |
|   mean : -0.987400   |
| max value: -0.935000 |
------------------------


*****  See the following link for interpretation ***** 
*****  https://oyster-river-protocol.readthedocs.io/en/latest/strandexamine.html ***** 



*****  QUALITY REPORT FOR: test using the ORP version 2.1.0 ****
*****  THE ASSEMBLY CAN BE FOUND HERE: /home/tc/sampledata2/assemblies/test.ORP.fasta **** 

*****  BUSCO SCORE ~~~~~>           	C:0.0%[S:0.0%,D:0.0%],F:0.3%,M:99.7%,n:303
*****  TRANSRATE SCORE ~~~~~>           0.41937
*****  TRANSRATE OPTIMAL SCORE ~~~~~>   0.58487
*****  UNIQUE GENES ORP ~~~~~>          39
*****  UNIQUE GENES TRINITY ~~~~~>      31
*****  UNIQUE GENES SPADES55 ~~~~~>     22
*****  UNIQUE GENES SPADES75 ~~~~~>     23
*****  UNIQUE GENES TRANSABYSS ~~~~~>   35
 

DeprecationWarning: 'source deactivate' is deprecated. Use 'conda deactivate'.

macmanes · 2019-01-28T14:12:13Z

ok good.. the other issues aside, have you been able to successfully run a real dataset through the pipeline?

AntonioBaeza · 2019-01-31T19:13:16Z

Sample problem not solved following the instructions above.
This is the way py36_env.yml looks after editing with nano (note:

(orp_v2) [ant@hillary Oyster_River_Protocol]$ cat py36_env.yml
name: orp_v2
channels:

bioconda
conda-forge
defaults
r
antoined
dependencies:
python==3.6.0 {note: I check the version in my download}
git
zsh
ruby
make
glib
zlib==1.2.11=0
libgcc>=7.2.0
salmon>=0.10.2
seqtk
trinity
spades
busco
blast
hmmer
samtools
blat
parallel
last
rcorrector==1.0.3
mcl
abyss>=2.0.1
python-igraph
diamond
cd-hit
bwa
pip:
- biopython
- cvxopt
- cython
- numpy
- pandas
- pip
- scipy
- bashplotlib
- "git+https://gitlab.com/ezlab/busco"
  (orp_v2) [ant@hillary Oyster_River_Protocol]$

The error message I get is the same than before editing py36 but slightly different to the one reported above by cooketho:

Thank you for using SPAdes!
Traceback (most recent call last):
File "/home/ant/Oyster_River_Protocol/software/transabyss/transabyss", line 18, in
from utilities.adj_utils import has_edges
File "/home/ant/Oyster_River_Protocol/software/transabyss/utilities/adj_utils.py", line 6, in
import igraph
File "/home/ant/Oyster_River_Protocol/software/anaconda/install/envs/orp_v2/lib/python3.6/site-packages/igraph/init.py", line 34, in
from igraph._igraph import *
ImportError: /home/ant/Oyster_River_Protocol/software/anaconda/install/envs/orp_v2/lib/python3.6/site-packages/igraph/_igraph.cpython-36m-x86_64-linux-gnu.so: undefined symbol: PySlice_Unpack
make: *** [/home/ant/Oyster_River_Protocol/sampledata/assemblies/test.transabyss.fasta] Error 1
(orp_v2) [ant@hillary sampledata]$

Any thoughts? Thanks beforehand

cooketho · 2019-01-31T19:25:05Z

You have python==3.6.0. You probably need python>=3.6.1, which is what I used (see previous post).

AntonioBaeza · 2019-01-31T19:31:23Z

Yes, I did try it too:
I edited again py36 so that it looks like:

(orp_v2) [ant@hillary Oyster_River_Protocol]$ cat py36_env.yml
name: orp_v2
channels:

bioconda
conda-forge
defaults
r
antoined
dependencies:
python>=3.6.1
git
zsh
ruby
make
glib
zlib==1.2.11=0
libgcc>=7.2.0
salmon>=0.10.2
seqtk
trinity
spades
busco
blast
hmmer
samtools
blat
parallel
last
rcorrector==1.0.3
mcl
abyss>=2.0.1
python-igraph
diamond
cd-hit
bwa
pip:
- biopython
- cvxopt
- cython
- numpy
- pandas
- pip
- scipy
- bashplotlib
- "git+https://gitlab.com/ezlab/busco"

But I get the same error:

Thank you for using SPAdes!
Traceback (most recent call last):
File "/home/ant/Oyster_River_Protocol/software/transabyss/transabyss", line 18, in
from utilities.adj_utils import has_edges
File "/home/ant/Oyster_River_Protocol/software/transabyss/utilities/adj_utils.py", line 6, in
import igraph
File "/home/ant/Oyster_River_Protocol/software/anaconda/install/envs/orp_v2/lib/python3.6/site-packages/igraph/init.py", line 34, in
from igraph._igraph import *
ImportError: /home/ant/Oyster_River_Protocol/software/anaconda/install/envs/orp_v2/lib/python3.6/site-packages/igraph/_igraph.cpython-36m-x86_64-linux-gnu.so: undefined symbol: PySlice_Unpack
make: *** [/home/ant/Oyster_River_Protocol/sampledata/assemblies/test.transabyss.fasta] Error 1
(orp_v2) [ant@hillary sampledata]$

Thanks!

cooketho · 2019-01-31T19:35:41Z

You still have - busco in the main dependencies list. That will install the outdated anaconda repository for busco. Try deleting it. That's why I used the gitlab busco repository installed via pip. You can just cut and paste the code from my previous post.

AntonioBaeza · 2019-01-31T19:36:29Z

got it! I will try

AntonioBaeza · 2019-01-31T19:41:49Z

Nope. Still the same error! Not sure what else to do. Thanks

cooketho · 2019-01-31T19:51:52Z

Hmm. Sorry! I don't have any more advice, except you might want to try deleting the entire installation and starting from scratch by downloading from github again, and following the installation instructions. But before you enter make, copy-paste my post above into your py36_env.yml

AntonioBaeza · 2019-01-31T19:53:34Z

ok, great, will do, thanks again

macmanes · 2019-02-01T12:04:20Z

I've updated the repo with the Python changes. You should be able to git update -> make at this point.

Let me know if starting from scratch works - I'm committed to helping you get this up and running.

AntonioBaeza · 2019-02-01T13:09:04Z

Thanks Matt. I will keep you posted. Regards!

AntonioBaeza · 2019-02-01T18:48:37Z

I uninstalled everything, and all went smoothly. Now, before doing make, I modified the py36 file, but then I am getting this error message after make:

Collecting git+https://gitlab.com/ezlab/busco (from -r /home/ant/Oyster_River_Protocol/condaenv._g9hpm_0.requirements.txt (line 9))
Cloning https://gitlab.com/ezlab/busco to /tmp/pip-req-build-kzyqy7sh
error: RPC failed; result=22, HTTP code = 404

This problem was solved by adding '.git' at the end of the web address in the py36 file

AntonioBaeza · 2019-02-01T21:51:47Z

The installation went smoothly: I modified the py36 file according to the post above before 'make' and all run well. The previous error was resolved. However, I have a new issue. When I test the protocol, this is the message:

(orp_v2) [ant@hillary Oyster_River_Protocol]$ cd sampledata/

(orp_v2) [ant@hillary sampledata]$ /home/ant/Oyster_River_Protocol/oyster.mk main \

STRAND=RF
MEM=15
CPU=8
READ1=test.1.fq.gz
READ2=test.2.fq.gz
RUNOUT=test
which: no run_BUSCO.py in which: no run_BUSCO.py in (/home/ant/Oyster_River_Protocol/software/anaconda/install/envs/orp_v2/bin:/home/ant/Oyster_River_Protocol/software/anaconda/install/condabin:/usr/local/rvm/gems/ruby-1.9.3-p551/bin:/usr/local/rvm/gems/ruby-1.9.3-p551@global/bin:/usr/local/rvm/rubies/ruby-1.9.3-p551/bin:/usr/lib64/qt-3.3/bin:/usr/local/bin:/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/sbin:/usr/local/rvm/bin:/home/ant/bin:/home/ant/Oyster_River_Protocol/software/anaconda/install/bin:/home/ant/Oyster_River_Protocol/software/OrthoFinder/orthofinder:/home/ant/Oyster_River_Protocol/software/orp-transrate:/home/ant/Oyster_River_Protocol/software/transabyss)
/home/ant/Oyster_River_Protocol/oyster.mk:96: *** "\n\n*** BUSCO is not installed, must fix ***". Stop.
(orp_v2) [ant@hillary sampledata]$

So, I went to github and downloaded the script 'run_BUSCO.py' to anaconda/bin
When I run the test sample, the protocol is getting stuck in orthofuse with the following message:

parallel: Error: --tollef has been retired.
parallel: Error: Remove --tollef or use --gnu to override --tollef.
make: *** [/home/ant/Oyster_River_Protocol/sampledata/orthofuse/test/orthotransrate.done] Error 255
(orp_v2) [ant@hillary sampledata]$

trying to solve this issue now: problem solved by adding flag --gnu to each parallel command found in each *.mk file.

AntonioBaeza · 2019-02-14T18:29:19Z

I wonder about what can be done if you are working on a linux centos machine with no administrator privileges. I got stuck because the version of coreutils in the machine I am using is old enough so that the command sort does not have --parallel. I installed a new version of coreutils with conda in the ORP_v2 environment [together with other mandatory updates requested by conda when installing coreutils]. Of course, this last action [mandatory script updates] broke the pipeline. I assume that is too much work to rework the pipeline so to have coreutils be integral part of it? Any comment will be appreciated. Thanks

macmanes · 2019-02-14T23:07:16Z

I will address ASAP! #

macmanes added the question label Feb 14, 2019

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

PySlice_Unpack error #16

PySlice_Unpack error #16

cooketho commented Jan 27, 2019

cooketho commented Jan 27, 2019 •

edited

Loading

macmanes commented Jan 28, 2019

cooketho commented Jan 28, 2019

macmanes commented Jan 28, 2019

AntonioBaeza commented Jan 31, 2019

cooketho commented Jan 31, 2019

AntonioBaeza commented Jan 31, 2019

cooketho commented Jan 31, 2019

AntonioBaeza commented Jan 31, 2019

AntonioBaeza commented Jan 31, 2019

cooketho commented Jan 31, 2019

AntonioBaeza commented Jan 31, 2019

macmanes commented Feb 1, 2019

AntonioBaeza commented Feb 1, 2019

AntonioBaeza commented Feb 1, 2019 •

edited

Loading

AntonioBaeza commented Feb 1, 2019 •

edited

Loading

AntonioBaeza commented Feb 14, 2019

macmanes commented Feb 14, 2019

PySlice_Unpack error #16

PySlice_Unpack error #16

Comments

cooketho commented Jan 27, 2019

cooketho commented Jan 27, 2019 • edited Loading

macmanes commented Jan 28, 2019

cooketho commented Jan 28, 2019

macmanes commented Jan 28, 2019

AntonioBaeza commented Jan 31, 2019

cooketho commented Jan 31, 2019

AntonioBaeza commented Jan 31, 2019

cooketho commented Jan 31, 2019

AntonioBaeza commented Jan 31, 2019

AntonioBaeza commented Jan 31, 2019

cooketho commented Jan 31, 2019

AntonioBaeza commented Jan 31, 2019

macmanes commented Feb 1, 2019

AntonioBaeza commented Feb 1, 2019

AntonioBaeza commented Feb 1, 2019 • edited Loading

AntonioBaeza commented Feb 1, 2019 • edited Loading

AntonioBaeza commented Feb 14, 2019

macmanes commented Feb 14, 2019

cooketho commented Jan 27, 2019 •

edited

Loading

AntonioBaeza commented Feb 1, 2019 •

edited

Loading

AntonioBaeza commented Feb 1, 2019 •

edited

Loading