"buffer overflow detected" when using psipred. #13

panganqi · 2021-07-07T15:00:48Z

here is the error:

./input_prep/make_ss.sh: line 18: 11775 Aborted (core dumped) psipred $ID.mtx $DATADIR/weights.dat $DATADIR/weights.dat2 $DATADIR/weights.dat3 > $ID.ss

I am running the fasta file in example/input.fa, I think this might be the gcc version error. Can you help me? and which version of gcc do you use?

The text was updated successfully, but these errors were encountered:

minkbaek · 2021-07-07T20:40:59Z

Could you provide more details for your machine environment? We used gcc 9.3.0 (see below)

gcc --version
gcc (Ubuntu 9.3.0-17ubuntu1~20.04) 9.3.0
Copyright (C) 2019 Free Software Foundation, Inc.
This is free software; see the source for copying conditions. There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

moonblue333 · 2021-07-08T07:02:49Z

This version is OK for me, @ubuntu 18.0.4.
wget https://github.com/soedinglab/hh-suite/releases/download/v3.2.0/hhsuite-3.2.0-AVX2-Linux.tar.gz --no-check-certificate
tar xvfz hhsuite-3.2.0-AVX2-Linux.tar.gz

moonblue333 · 2021-07-08T07:03:34Z

hhsuite-3.3.0 is NOT OK.

li-ziang · 2021-07-15T13:57:59Z

Hi there, I had the same problem as @panganqi ,
2920146 Aborted (core dumped) psipred $ID.mtx $DATADIR/weights.dat $DATADIR/weights.dat2 $DATADIR/weights.dat3 > $ID.ss
I installed hhsuite by using:

$ wget https://github.com/soedinglab/hh-suite/releases/download/v3.2.0/hhsuite-3.2.0-AVX2-Linux.tar.gz --no-check-certificate
$ tar xvfz hhsuite-3.2.0-AVX2-Linux.tar.gz
$ export PATH="$(pwd)/bin:$(pwd)/scripts:$PATH"

Here is my machine environment:

$ hhblits
HHblits 3.1.0:
HMM-HMM-based lightning-fast iterative sequence search
HHblits is a sensitive, general-purpose, iterative sequence search tool that represents
both query and database sequences by HMMs. You can search HHblits databases starting
with a single query sequence, a multiple sequence alignment (MSA), or an HMM. HHblits
prints out a ranked list of database HMMs/MSAs and can also generate an MSA by merging
the significant database HMMs/MSAs onto the query MSA.

$ gcc --version
gcc (Ubuntu 9.4.0-1ubuntu1~18.04) 9.4.0
Copyright (C) 2019 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.  There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

I have already tried to use the file in this repo (https://github.com/RosettaCommons/RoseTTAFold/blob/main/example/pyrosetta/t000_.msa0.a3m) as the input of step 2 in https://github.com/RosettaCommons/RoseTTAFold/blob/main/run_pyrosetta_ver.sh#L44, but still got the same result.

It seems that it is the psipred package which goes wrong, according to the things front.

Can you offer some advice or help @minkbaek?
Thank you!

minkbaek · 2021-07-15T15:17:40Z

Not sure what exactly causes the problem here as it works fine for my Linux machine. One thing you can try is to install psipred from the source code (https://github.com/psipred/psipred). For easy-install, I included conda installation for psipred, but it might be not compatible with your Linux environment.

andrewcchang · 2021-07-15T17:11:28Z

I am facing the same 'buffer overflow' error while running PSIPRED. I did try installing from the source code with no success.

Just curious, is this secondary structure info crucial to the prediction, can we ignore it and just use the MSA from hhblits? If we still have no luck getting psipred to work, maybe we can ignore this step?

Thank you!

li-ziang · 2021-07-16T01:39:35Z

I have already tried to install psipred from the source code (https://github.com/psipred/psipred), but it did not succeed either. I am using ubuntu1~18.04, so this means that maybe I can try ubuntu1~20.04?

li-ziang · 2021-07-16T04:18:55Z

I found that the bug occurs at https://github.com/psipred/psipred/blob/master/src/sspred_avpred.c#L241, buffer size is 256 when I change it to 512, it can run, but I got the following output which seems not correct:

>ss_pred
CCCCCCCCCCCCCCCCCCCCCEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEECCEECCCCCCCCCCCCCCCC
>ss_conf
572999999899030000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000001000999998999999

Could you give me an example of the t000_.msa0.tmp.mtx file in step 2?

Here is mine:t000_.msa0.tmp.mtx.txt

Line 15 and the following lines of the file seems to be wrong because there are many -2147483648? Is that true?

pp3ndwich · 2021-07-20T17:35:08Z

In my case, updating Ubuntu from 18.04 to 20.04 perfectly solved the buffer overflow problem. Guess the latest version of Ubuntu and gcc somehow would not encounter this problem. Hope this helps.

cuiyoutian · 2021-07-21T07:13:22Z

I have the same issue. I am using Ubuntu 16.04.7 and gcc 5.4.0. I am using HHblits 3.3.0.

li-ziang · 2021-07-21T07:57:35Z

In my case, updating Ubuntu from 18.04 to 20.04 perfectly solved the buffer overflow problem. Guess the latest version of Ubuntu and gcc somehow would not encounter this problem. Hope this helps.
@pp3ndwich

That would be perfect!
Could you please give your gcc version too? I will try to find a docker to solve the problem.
By the way, is HHblits of version 3.3.0 installed by conda ok?
THANKS IN ADVANCE!!!

pp3ndwich · 2021-07-21T08:41:11Z

In my case, updating Ubuntu from 18.04 to 20.04 perfectly solved the buffer overflow problem. Guess the latest version of Ubuntu and gcc somehow would not encounter this problem. Hope this helps.
@pp3ndwich

That would be perfect!
Could you please give your gcc version too? I will try to find a docker to solve the problem.
By the way, is HHblits of version 3.3.0 installed by conda ok?
THANKS IN ADVANCE!!!

Here is the gcc version on our computer:

gcc (Ubuntu 9.3.0-17ubuntu1~20.04) 9.3.0

This version of gcc is bundled with the latest version of Ubuntu.
I checked my version of hhblits, the same as yours, 3.3.0. The package is installed in the RoseTTAFold environment in conda, so if you do "conda install hhsuite" under the base environment, I guess the script would fail to detect the package. Anyway, I doubt hhblits has anything related to the problem.
Good luck!

li-ziang · 2021-07-21T17:02:37Z

Good news!
Thanks to the nvidia docker (nvidia/cuda:11.0.3-devel-ubuntu20.04), I perfectly solve the problem.
This docker has the right environment.
Many thanks again!!!

qiyubio · 2021-07-21T22:12:03Z

The working around works for me is by adding
export BLASTMAT=your_path_to/RoseTTAFold/your_conda_env/share/blast-2.2.26/data/ to input_prep/make_ss.sh

Without it, makemat will generate the wrong size of mtx file.
makemat -P t000_.msa0.tmp
44K Jul 21 13:50 t000_.msa0.tmp.mtx
19 Jul 21 13:50 t000_.msa0.tmp.mn
74 Jul 21 13:50 t000_.msa0.tmp.aux

With BLASTMAT:
24K Jul 21 13:44 t000_.msa0.tmp.mtx
19 Jul 21 13:44 t000_.msa0.tmp.mn
74 Jul 21 13:44 t000_.msa0.tmp.aux

pfreezer · 2021-07-27T01:02:08Z

The working around works for me is by adding
export BLASTMAT=your_path_to/RoseTTAFold/your_conda_env/share/blast-2.2.26/data/ to input_prep/make_ss.sh

Without it, makemat will generate the wrong size of mtx file.
makemat -P t000_.msa0.tmp
44K Jul 21 13:50 t000_.msa0.tmp.mtx
19 Jul 21 13:50 t000_.msa0.tmp.mn
74 Jul 21 13:50 t000_.msa0.tmp.aux

With BLASTMAT:
24K Jul 21 13:44 t000_.msa0.tmp.mtx
19 Jul 21 13:44 t000_.msa0.tmp.mn
74 Jul 21 13:44 t000_.msa0.tmp.aux

Hi, could you specify what is "your_path_to" and "your_conda_env" here,
you_path_to is the path where I pull the code from github?
and "your_conda_env" is the name the conda environment, like RoseTTAFold or folding?
I tried this but it cannot solve the problem

xubeisi · 2021-07-28T01:09:41Z

The working around works for me is by adding
export BLASTMAT=your_path_to/RoseTTAFold/your_conda_env/share/blast-2.2.26/data/ to input_prep/make_ss.sh

Without it, makemat will generate the wrong size of mtx file.
makemat -P t000_.msa0.tmp
44K Jul 21 13:50 t000_.msa0.tmp.mtx
19 Jul 21 13:50 t000_.msa0.tmp.mn
74 Jul 21 13:50 t000_.msa0.tmp.aux

With BLASTMAT:
24K Jul 21 13:44 t000_.msa0.tmp.mtx
19 Jul 21 13:44 t000_.msa0.tmp.mn
74 Jul 21 13:44 t000_.msa0.tmp.aux

this works for me too,
you could either
conda install --force-reinstall biocore::blast-legacy=2.2.26

or used "data" in

https://ftp.ncbi.nlm.nih.gov/blast/executables/legacy.NOTSUPPORTED/2.2.26/blast-2.2.26-x64-linux.tar.gz

thanks!

qiyubio · 2021-07-28T20:47:39Z

The working around works for me is by adding
export BLASTMAT=your_path_to/RoseTTAFold/your_conda_env/share/blast-2.2.26/data/ to input_prep/make_ss.sh
Without it, makemat will generate the wrong size of mtx file.
makemat -P t000_.msa0.tmp
44K Jul 21 13:50 t000_.msa0.tmp.mtx
19 Jul 21 13:50 t000_.msa0.tmp.mn
74 Jul 21 13:50 t000_.msa0.tmp.aux
With BLASTMAT:
24K Jul 21 13:44 t000_.msa0.tmp.mtx
19 Jul 21 13:44 t000_.msa0.tmp.mn
74 Jul 21 13:44 t000_.msa0.tmp.aux

Hi, could you specify what is "your_path_to" and "your_conda_env" here,
you_path_to is the path where I pull the code from github?
and "your_conda_env" is the name the conda environment, like RoseTTAFold or folding?
I tried this but it cannot solve the problem

The path to your conda env, if you didn't change anything in the yaml file, then "RoseTTAFold" env. blast should be installed already in the environment:

RoseTTAFold/RoseTTAFold-linux.yml

Line 12 in 5510443

- biocore::blast-legacy=2.2.26

.

pfreezer · 2021-07-30T03:46:40Z

The working around works for me is by adding
export BLASTMAT=your_path_to/RoseTTAFold/your_conda_env/share/blast-2.2.26/data/ to input_prep/make_ss.sh
Without it, makemat will generate the wrong size of mtx file.
makemat -P t000_.msa0.tmp
44K Jul 21 13:50 t000_.msa0.tmp.mtx
19 Jul 21 13:50 t000_.msa0.tmp.mn
74 Jul 21 13:50 t000_.msa0.tmp.aux
With BLASTMAT:
24K Jul 21 13:44 t000_.msa0.tmp.mtx
19 Jul 21 13:44 t000_.msa0.tmp.mn
74 Jul 21 13:44 t000_.msa0.tmp.aux

Hi, could you specify what is "your_path_to" and "your_conda_env" here,
you_path_to is the path where I pull the code from github?
and "your_conda_env" is the name the conda environment, like RoseTTAFold or folding?
I tried this but it cannot solve the problem

The path to your conda env, if you didn't change anything in the yaml file, then "RoseTTAFold" env. blast should be installed already in the environment:

RoseTTAFold/RoseTTAFold-linux.yml

Line 12 in 5510443

- biocore::blast-legacy=2.2.26

.

Thanks, it turns out if you use RoseTTAFold/RoseTTAFold-linux-cu101.yml instead of RoseTTAFold/RoseTTAFold-linux.yml, you can not find $CONDA_PREFIX/env/RoseTTAFold/share/blast-2.2.26/data/, but if I download the package mention by xubeisi,
it works well, thanks @xubeisi

Overall, the problem is not related with psipred but makemat, with BLASTMAT defined, correct mtx file can be generated and so psipred can work well.

thuxugang · 2021-08-15T07:14:54Z

The working around works for me is by adding
export BLASTMAT=your_path_to/RoseTTAFold/your_conda_env/share/blast-2.2.26/data/ to input_prep/make_ss.sh
Without it, makemat will generate the wrong size of mtx file.
makemat -P t000_.msa0.tmp
44K Jul 21 13:50 t000_.msa0.tmp.mtx
19 Jul 21 13:50 t000_.msa0.tmp.mn
74 Jul 21 13:50 t000_.msa0.tmp.aux
With BLASTMAT:
24K Jul 21 13:44 t000_.msa0.tmp.mtx
19 Jul 21 13:44 t000_.msa0.tmp.mn
74 Jul 21 13:44 t000_.msa0.tmp.aux

this works for me too,
you could either
conda install --force-reinstall biocore::blast-legacy=2.2.26

or used "data" in

https://ftp.ncbi.nlm.nih.gov/blast/executables/legacy.NOTSUPPORTED/2.2.26/blast-2.2.26-x64-linux.tar.gz

thanks!

works for me, thanks!

wmustafaAwad · 2021-09-15T10:45:38Z

Running conda install --force-reinstall biocore::blast-legacy=2.2.26 as suggested by xubeisi did it for me.

BJWiley233 · 2023-01-27T00:21:59Z

No. All wrong :0. This is because we all followed the install.md from this post. It indicates to switch NCBI's default environmental variable $BLASTMAT to point to the csblast database under step 16. This is incorrect. Just look at the ouput of $ID.mtx when you do this. It looks like this below and the bits overload the program. makemat is a C program made by NCBI and it uses the BLASTMAT environmental variable to get the BLOSUM/PAM matrices. See section 10.2.4 Environment Variables

-32768  -2147483648  -32768  -2147483648  -2147483648  -2147483648  -2147483648  -2147483648  -2147483648  -2147483648  -2147483648  -2147483648

To test this, deactivate with conda deactivate and then conda activate RoseTTAFold

$ echo $BLASTMAT
/opt/conda/envs/RoseTTAFold/share/blast-2.2.26/data/

TADA!!!!

xubeisi mentioned this issue Jul 29, 2021

Update RoseTTAFold-linux-cu101.yml #47

Open

JonathanHungerland mentioned this issue Oct 19, 2023

there is no t000_.e2e.pdb #60

Open

RumoisteinKartenspiel mentioned this issue Mar 11, 2024

About Conda Environment baker-laboratory/RoseTTAFold-All-Atom#5

Open

leodoned mentioned this issue Mar 13, 2024

psipred terminated baker-laboratory/RoseTTAFold-All-Atom#28

Closed

YaoYinYing mentioned this issue Apr 9, 2024

rf2aa as a Python module baker-laboratory/RoseTTAFold-All-Atom#78

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

"buffer overflow detected" when using psipred. #13

"buffer overflow detected" when using psipred. #13

panganqi commented Jul 7, 2021

minkbaek commented Jul 7, 2021

moonblue333 commented Jul 8, 2021

moonblue333 commented Jul 8, 2021

li-ziang commented Jul 15, 2021

minkbaek commented Jul 15, 2021

andrewcchang commented Jul 15, 2021 •

edited

li-ziang commented Jul 16, 2021 •

edited

li-ziang commented Jul 16, 2021

pp3ndwich commented Jul 20, 2021

cuiyoutian commented Jul 21, 2021

li-ziang commented Jul 21, 2021

pp3ndwich commented Jul 21, 2021

li-ziang commented Jul 21, 2021

qiyubio commented Jul 21, 2021 •

edited

pfreezer commented Jul 27, 2021

xubeisi commented Jul 28, 2021 •

edited

qiyubio commented Jul 28, 2021

pfreezer commented Jul 30, 2021

thuxugang commented Aug 15, 2021

wmustafaAwad commented Sep 15, 2021 •

edited

BJWiley233 commented Jan 27, 2023 •

edited

"buffer overflow detected" when using psipred. #13

"buffer overflow detected" when using psipred. #13

Comments

panganqi commented Jul 7, 2021

minkbaek commented Jul 7, 2021

moonblue333 commented Jul 8, 2021

moonblue333 commented Jul 8, 2021

li-ziang commented Jul 15, 2021

minkbaek commented Jul 15, 2021

andrewcchang commented Jul 15, 2021 • edited

li-ziang commented Jul 16, 2021 • edited

li-ziang commented Jul 16, 2021

pp3ndwich commented Jul 20, 2021

cuiyoutian commented Jul 21, 2021

li-ziang commented Jul 21, 2021

pp3ndwich commented Jul 21, 2021

li-ziang commented Jul 21, 2021

qiyubio commented Jul 21, 2021 • edited

pfreezer commented Jul 27, 2021

xubeisi commented Jul 28, 2021 • edited

qiyubio commented Jul 28, 2021

pfreezer commented Jul 30, 2021

thuxugang commented Aug 15, 2021

wmustafaAwad commented Sep 15, 2021 • edited

BJWiley233 commented Jan 27, 2023 • edited

andrewcchang commented Jul 15, 2021 •

edited

li-ziang commented Jul 16, 2021 •

edited

qiyubio commented Jul 21, 2021 •

edited

xubeisi commented Jul 28, 2021 •

edited

wmustafaAwad commented Sep 15, 2021 •

edited

BJWiley233 commented Jan 27, 2023 •

edited