failing tests with t/test_pipeline.pl

[zmahiri]

Hi, I am trying to install orthomcl-pipeline. I have successfully set up the orthomcl orthomcl-setup-database.pl, but was unable to create the database with this file, I recieved the error message as follows

$ perl orthomcl-setup-database.pl --user orthomcl --password password --host localhost --database orthomcl --outfile orthomcl.conf
Warning: file orthomcl.conf already exists ... overwrite? (Y/N) Y

Config file, orthomcl.conf will be overwritten
Connecting to mysql and creating database orthomcl on host localhost with user orthomcl ...
DBI connect('mysql:localhost:mysql_local_infile=1','orthomcl',...) failed: Access denied for user 'orthomcl'@'localhost' to database 'mysql' at orthomcl-setup-database.pl line 89.
error connecting to database at orthomcl-setup-database.pl line 93, line 1..

I was able to create the orthomcl database manually in mysql and grant access. I can access mysql from the command line with the user 'orthomcl' that i created, but it seems like execution form with orthomcl-pipeline is having trouble.

I passed the orthomcl-pipeline-setup.pl script successfully, but when I try the test run

$ perl t/test_pipeline.pl -m orthomcl.conf -s fork -t /tmp

Test using scheduler fork

TESTING NON-COMPLIANT INPUT
TESTING FULL PIPELINE RUN 5
README:
Tests case of one gene (in 1.fasta) not present in any other files but with a paralog in 1.fasta.
/Path_to_programs/orthomcl-pipeline/t/data/basic/5/groups/groups.txt contains entries (1|a 2|a 3|b 1|b 2|b 3|c 1|c 2|c 3|d 1|d 2|d 3|e 1|e 2|e 3|a 1|f 1|g) not in /tmp/orthomcl-pipeline.dD7Xqn/output/groups/groups.txt
not ok 1 - Expected matched returned groups file
Failed test 'Expected matched returned groups file'
at t/test_pipeline.pl line 204.

TESTING FULL PIPELINE RUN 3
README:
Tests case of one gene (in 1.fasta and 2.fasta) not present in other files.
/Path_to_programs/orthomcl-pipeline/t/data/basic/3/groups/groups.txt contains entries (1|a 2|a 3|b 1|b 2|b 3|c 1|c 2|c 3|d 1|d 2|d 3|e 1|e 2|e 3|a 1|f 2|f) not in /tmp/orthomcl-pipeline.oJnvbO/output/groups/groups.txt
not ok 2 - Expected matched returned groups file
Failed test 'Expected matched returned groups file'
at t/test_pipeline.pl line 204.

And it continues to fail 7 out of 8 tests. However, I cannot locate a err file to see what is going wrong. I have gathered that it is not generating files in the -t directory (i.e /tmp/orthomcl-pipeline.dD7Xqn/output/groups/groups.txt doesn't exist in the above error message). Is this a problem related to the mysql configuration? Any advice would be greatly appreciated.

Thanks
Zaid

[apetkau]

Hello Zaid,

When you ran the initial script to setup the OrthoMCL database did you first run the GRANT ... on *.* to orthomcl; command in mysql? This just gives permissions for the user orthomcl@localhost to run any command on any database. This was mainly implemented for us to have different users who aren't as familiar with mysql to create databases with a single command, but may not be what you want (since it grants nearly all permissions on every database to a single user). Creating the user manually will work just as well.

For the failing test, by default the test script cleans up temporary files after execution. To turn this off please remote the rmtree lines in the test script. That is delete line

orthomcl-pipeline/t/test_pipeline.pl

Line 208 in 4877cce

rmtree($tempdir);

and

orthomcl-pipeline/t/test_pipeline.pl

Line 252 in 4877cce

rmtree($tempdir);

. Then, try re-running the tests and you should have access to all the produced output files. In particular, a directory that looks like orthomcl-pipeline.pWOWqc/output/log/ (first directory name different for each run) will have any of the produced log files when can be inspected to see what went wrong.

Let me know if you have any more questions,

Aaron

[zmahiri]

Hello Aaron,

Thanks for the fast response. I had created the user, the database, and granted the privileges directly in mysql. I thought it may have been the perl module DBD::mysql, but I tried to test that with some sample database queries it worked fine so I don't think that is the problem.

Obtaining the log files I see directly in the orthomcl-pipeline.err.log file for each of the test temp directories a few instances of Odd number of elements in hash assignment at /usr/local/share/perl5/Bio/SeqIO.pm line 378.. When I look in /tmp/orthomcl-pipeline.7NlXUa/output/log I see that 13.mcl.log.stderr appears to have created 0 clusters.

[mclxIOstreamIn] no assignments yield void/empty matrix [mcl] new tab created ___ [mclAlgorithmStart] attempting to cluster the void [mcl] pid 30280 ite chaos time hom(avg,lo,hi) m-ie m-ex i-ex fmv 1 0.00 0.00 0.00/340282346638528859811704183484516925440.00/0.00 0.00 0.00 0.00 -2147483648 [mcl] jury pruning marks: <100,100,100>, out of 100 [mcl] jury pruning synopsis: <100.0 or really really really good> (cf -scheme, -do log) [mcl] output is in /tmp/orthomcl-pipeline.7NlXUa/output/pairs/mclOutput [mcl] 0 clusters found [mcl] output is in /tmp/orthomcl-pipeline.7NlXUa/output/pairs/mclOutput

I have attached one of the temp log folders. if you take a look and give me some suggestions, it would be greatly appreciated.

Zaid

ZMK.orthomcl.pipeline.err.tar.gz

[apetkau]

Hello Zaid,

Thanks for attaching the error logs. It helps me out for trying to diagnose the problem. However, I'm not really sure what the exact cause of the issue you are having is. The message Odd number of elements ... appears for me after a successful run of the tests so that is not the issue.

The main difference I noticed between our two runs are that you are using BLAST version 2.2.11 whereas I've been using BLAST version 2.2.26. There could be differences in the output between each version. The directories in output/blast* contain most of the blast results and you can check to see if they have valid blast results (or send them to me).

[zmahiri]

Hello Aaron,

I figured that Odd number of elements ... was not the problem. However, I do have BLAST 2.2.26 installed, but when using it I was getting an error that formatdb was not found, so I tried to use a legacy version of BLAST. Also I have attached the blast output logs from one of the tests.

ZMK_orthoMCL_blast.tar.gz

[apetkau]

Hello Zaid,

Can you try out again with blast 2.2.26? Please make sure that the correct paths to blastall and formatdb are set in etc/orthomcl-pipeline.conf (see, the example file at https://github.com/apetkau/orthomcl-pipeline/blob/master/etc/orthomcl-pipeline.conf.default#L6).

Based on the blast files you sent me, it appears to be an issue with the blast results. Here is a line from one of my blast results:

1|a 1|a 100.00  400 0   0   1   400 1   400 1e-104   310

Compared to an equivalent line from your results:

1|a 1|a 100.00  400 0   0   1   400 1   400 0.0  775

That is, all your e-value and bitscore fields are much different. In particular, every single e-value is 0.0, and OrthoMCL requires at least one e-value to be non-zero or it will not work. This makes me think that the issue you are experiencing is due to blast.

Let me know how this works,

Aaron

[zmahiri]

Hello Aaron,

It worked! t/test_pipeline.pl returned no errors! Thanks!

Zaid

[apetkau]

Awesome. I've updated the documentation to reflect this. I'm closing this issue. Let me know if you have any more questions.

[HernanVM]

Dear Aaron,

I was able to successfuly install orthomcl-pipeline on my Mac laptop (OSX Mavericks). While trying to install it on a different Mac desktop, I'm running into this similar problem to the original referred here even after I give permissions to user orthomcl on mysql. I copied the .conf file I generated in the laptop:

Bracken-Grissoms-iMac:orthomcl-pipeline Odontodactylus$ perl scripts/orthomcl-setup-database.pl --user orthomcl --password orthomcl --host localhost --database orthomcl --outfile orthomcl.conf
Warning: file orthomcl.conf already exists ... overwrite? (Y/N) Y

Config file, orthomcl.conf will be overwritten
Connecting to mysql and creating database orthomcl on host localhost with user orthomcl ...
DBI connect('mysql:localhost:mysql_local_infile=1','orthomcl',...) failed: Access denied for user 'orthomcl'@'localhost' (using password: YES) at scripts/orthomcl-setup-database.pl line 89.
error connecting to database at scripts/orthomcl-setup-database.pl line 93, <STDIN> line 1.

I'm using mysql5.7.18. After reading several forums on this it seems that there's a new way to give permissions to users and the one in the instructions is no longer a valid syntax. I gave permissions like this at the mysql> promtp:

GRANT SELECT, INSERT, UPDATE, DELETE, CREATE, CREATE VIEW, INDEX, DROP ON *.* TO orthomcl@'localhost';

ALTER USER 'orthomcl'@'localhost' IDENTIFIED BY 'orthomcl';

GRANT PROXY ON ''@'' TO 'orthomcl'@'localhost' WITH GRANT OPTION;

As you can see my permissions seem correct but for whatever reason it is still not working:

mysql> SHOW GRANTS FOR 'root'@'localhost';
+---------------------------------------------------------------------+
| Grants for root@localhost                                           |
+---------------------------------------------------------------------+
| GRANT ALL PRIVILEGES ON *.* TO 'root'@'localhost' WITH GRANT OPTION |
| GRANT PROXY ON ''@'' TO 'root'@'localhost' WITH GRANT OPTION        |
+---------------------------------------------------------------------+
2 rows in set (0.00 sec)

mysql> SHOW GRANTS FOR 'orthomcl'@'localhost';
+-------------------------------------------------------------------------+
| Grants for orthomcl@localhost                                           |
+-------------------------------------------------------------------------+
| GRANT ALL PRIVILEGES ON *.* TO 'orthomcl'@'localhost' WITH GRANT OPTION |
| GRANT PROXY ON ''@'' TO 'orthomcl'@'localhost' WITH GRANT OPTION        |
+-------------------------------------------------------------------------+
2 rows in set (0.00 sec)

Any ideas as to what could be happening? Like I mentioned, I was already succesful in installing everything on a mac laptop.

Thanks! - Hernán

[apetkau]

Hello Hernán,

Are you able to log into mysql directly using the orthomcl user. E.g., mysql -u orthomcl -p. If you can log in, can you create a new orthomcl database directly from mysql (e.g., CREATE DATABASE orthomcl;).

The new permission you are referring to is PROXY am I correct? I do not think that is needed for orthomcl to run as it relates to being able to impersonate other users.

Also, note that you don't have to run orthomcl with the user orthomcl@localhost. You can run it with any user you can log into (even root) so long as you have permission to create/drop the particular database used by orthomcl.

Aaron

[gianninimail]

Hello, I speak from Brazil and my English is terrible. So I'm sorry for any mistake. I'm trying to install the orthomcl pipeline, but when I run the "orthomcl-pipeline-setup.pl" script I get the following error message: "Checking for formatdb ... error: formatdb could not be found on PATH". What would be the BLAST + database?

My installation is with the following setting:
BLAST = /home/thiago/Softwares/ncbi-blast-2.6.0+
OrthoMCL = /home/thiago/Softwares/orthomclSoftware-v2.0.9
BLAST Database = /home/thiago/Softwares/ncbi-blast-2.6.0+/blastdb

Thanks for listening!

[apetkau]

Hello,

You are going to want to use an older version of blast (legacy blast) which you can download from ftp://ftp.ncbi.nlm.nih.gov/blast/executables/legacy/2.2.26/.

You will want to add the directory blast-2.2.26/bin to your PATH (e.g., by running export PATH=/path/to/blast-2.2.26/bin:$PATH) and then run through https://github.com/apetkau/orthomcl-pipeline/blob/master/INSTALL.md#step-2-other-dependencies if you haven't done so already. This should ask you to run a small Perl script to automatically set the paths for all dependency software in a configuration file.

Also, for any new issues you have with the software could you create a new issue in github instead of replying to a closed issue? This makes it easier to keep issues organized.

Thank you and I hope you have a nice day.

Aaron

[gianninimail]

Thank you very much! It worked, I was able to run the first analysis.

[rillaxy]

Dear Aaron,
I have successfully set up the orthomcl orthomcl-setup-database.pl, but was unable to create the database with this file, I recieved the same error message as @zmahiri .I also passed the orthomcl-pipeline-setup.pl script successfully, but when I try the test run
perl t/test_pipeline.pl -m orthomcl.conf -s fork -t /tmp,I recieved error message as follows:

Test using scheduler fork

TESTING NON-COMPLIANT INPUT
TESTING FULL PIPELINE RUN 1
README:
Test using with no splitting of fasta files.
Could not execute command /home/xiaoye/app/orthomcl-pipeline/t/../bin/orthomcl-pipeline --nocompliant --scheduler fork --yes -c /home/xiaoye/app/orthomcl-pipeline/t/data/basic/1/etc/orthomcl-pipeline.conf -i /home/xiaoye/app/orthomcl-pipeline/t/data/basic/1/input -o /tmp/orthomcl-pipeline.FJMV_U/output -m /tmp/orthomcl-pipeline.FJMV_U/orthomcl.config 2>/tmp/orthomcl-pipeline.FJMV_U/orthomcl-pipeline.err.log 1>/tmp/orthomcl-pipeline.FJMV_U/orthomcl-pipeline.out.log

less /tmp/orthomcl-pipeline.FJMV_U/orthomcl-pipeline.err.log
DBI connect('orthomcl','orthomcl',...) failed: Access denied for user 'orthomcl'@'localhost' (using password: YES) at /home/xiaoye/app/orthomcl-pipeline/bin/../scripts/orthomcl-pipeline.pl line 182.

less /tmp/orthomcl-pipeline.FJMV_U/orthomcl-pipeline.out.log

Starting OrthoMCL pipeline on: Fri Jul 23 16:04:59 2021
Git commit: d0bacb3bd0f655406e09bc7fc3f776a40a57c75c


=Stage 1: Validate Files =
Validating 1.fasta ... 5 sequences
Validating 2.fasta ... 5 sequences
Validating 3.fasta ... 5 sequences
Validated 3 files
Stage 1 took 0.00 minutes 

=Stage 2: Validate Database=

I create the orthomcl database manually in mysql and grant access. I'm using mysql5.7.31 and I'm able to log into mysql directly using the orthomcl user.

create user 'orthomcl'@'localhost' identified by 'orthomcl';
GRANT ALL PRIVILEGES on *.* to 'orthomcl'@'localhost' WITH GRANT OPTION;
create database orthomcl;
FLUSH PRIVILEGES; quit;

My installation is with the following setting:

blastall: /home/xiaoye/app/blast-2.2.26/bin/blastall
formatdb: /home/xiaoye/app/blast-2.2.26/bin/formatdb
mcl: /home/xiaoye/app/mcl-14-137/bin/mcl
orthomcl: /home/xiaoye/app/orthomclSoftware-v2.0.9/bin

Here is my orthomcl.conf,

# this config assumes a mysql database named 'orthomcl'.  adjust according
# to your situation.
dbVendor=mysql 
dbConnectString=dbi:mysql:orthomcl
dbLogin=orthomcl
dbPassword=orthomcl
similarSequencesTable=SimilarSequences
orthologTable=Ortholog
inParalogTable=InParalog
coOrthologTable=CoOrtholog
interTaxonMatchView=InterTaxonMatch
percentMatchCutoff=50
evalueExponentCutoff=-5
oracleIndexTblSpc=NONE

Thanks for listening, I'm looking forward to your reply!
Rilla

[apetkau]

Hello @rillaxy ,

Try changing the dbConnectString in orthomcl.conf to include the host name of your mysql server along with an additional required parameter mysql_local_infile.

That is in orthomcl.conf:

• Change dbConnectString=dbi:mysql:orthomcl to dbConnectString=dbi:mysql:orthomcl:localhost:mysql_local_infile=1

I hope this will work for you.

[rillaxy]

Hello Aaron,
Thanks for your fast responses. I have changed the dbConnectString in orthomcl.conf ,but I still get the same error message described above.
It's so frustrating! Do you have any other suggestions?
Best wishes,
Rilla

[apetkau]

Hello @rillaxy,

I'm not really sure what else it could be. You can try directly calling the Perl code to connect to the database:

perl -MDBI -e 'DBI->connect("dbi:mysql:orthomcl:localhost:mysql_local_infile=1","orthomcl","orthomcl");'

And you could try switching localhost with 127.0.0.1.