Request to update gmap within pasapipeline

[tamuanand]

Hi

Right now, PASA (from anaconda or quay) uses GMAP version 2017-11-15

Is it possible to update PASA to use GMAP version 2021-05-27

http://research-pub.gene.com/gmap/

Lot of changes/improvements and fatal bug fixes (as mentioned in above page) have been implemente with GMAP version 2021-05-27

Thanks

[brianjohnhaas]

If it's a drop-in fit with the current code, I can remove the recommendation to use the older version. Just let me know if it works. I won't have time to work on the pasa code for a few months so if it needs additional code updates for it (by me), it'll have to wait.

…

On Sun, Jul 11, 2021 at 7:29 PM tamuanand ***@***.***> wrote: Hi Right now, PASA (from anaconda or quay) uses GMAP version 2017-11-15 Is it possible to update PASA to use GMAP version 2021-05-27 http://research-pub.gene.com/gmap/ Lot of changes/improvements and fatal bug fixes (as mentioned in above page) have been implemente with GMAP version 2021-05-27 Thanks — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#197>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ABZRKX5YSAG45Y3GJBHJUNDTXISOPANCNFSM5AFYDRMA> .

-- -- Brian J. Haas The Broad Institute http://broadinstitute.org/~bhaas <http://broad.mit.edu/~bhaas>

[tamuanand]

Hi @brianjohnhaas

I would think so

If it's a drop-in fit with the current code, I can remove the
recommendation to use the older version. Just let me know if it works.

I can test it - if ok with you, probably you can deploy a pasa-dev via bioconda and I can download the same to do the tests.

Alternatively, if you can let me know where I can change the path for gmap on my pasa install (which I did via bioconda conda install -c bioconda -y pasa) - I can try changing it and let you know how it goes

[brianjohnhaas]

I expect it should just use whatever version of gmap is first found within your PATH setting, so you could just upgrade to the latest gmap and then update your PATH so it's found first. many thanks, ~brian

…

On Tue, Jul 13, 2021 at 8:35 PM tamuanand ***@***.***> wrote: Hi @brianjohnhaas <https://github.com/brianjohnhaas> I would think so If it's a drop-in fit with the current code, I can remove the recommendation to use the older version. Just let me know if it works. I can test it - if ok with you, probably you can deploy a pasa-dev via bioconda and I can download the same to do the tests. Alternatively, if you can let me know where I can change the path for gmap on my pasa install (which I did via bioconda conda install -c bioconda -y pasa) - I can try changing it and let you know how it goes — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#197 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ABZRKX677M2EINIAWGZNSW3TXTLUJANCNFSM5AFYDRMA> .

-- -- Brian J. Haas The Broad Institute http://broadinstitute.org/~bhaas <http://broad.mit.edu/~bhaas>

[tamuanand]

A related question: If I have already built a genome index with gmap, is it possible to tell pasa to use this gmap_build/index instead of pasa trying to build it

[tamuanand]

hi @brianjohnhaas

I have some more updates

1 - I used your Dockerfile and changed it to download the latest GMAP

Find below the sections I needed to update (GMAP and FASTA)

## GMAP installation
WORKDIR $SRC
RUN wget http://research-pub.gene.com/gmap/src/gmap-gsnap-2021-05-27.tar.gz && \
        tar xvf gmap-gsnap-2021-05-27.tar.gz && \
        cd gmap-2021-05-27 && \
        ./configure && \
        make && \
        make install

## Fasta3
WORKDIR $SRC
RUN wget https://github.com/wrpearson/fasta36/archive/refs/tags/v36.3.8h_04-May-2020.tar.gz && \
        tar zxvf v36.3.8h_04-May-2020.tar.gz && \
        cd ./fasta36-36.3.8h_04-May-2020/src && \
        make -f ../make/Makefile.linux_sse2 all && \
        cp ../bin/fasta36 /usr/local/bin/fasta

2 - I successfully built the docker image, called it pasa_gmap_2021:latest and tried to do a docker run as suggested here: https://github.com/PASApipeline/PASApipeline/wiki/PASA_Docker

• I think there is a typo in there -g genome_sample.fasta should actually be -g genome_sample.fasta.gz or the sample_data directory should have this file genome_sample.fasta.gz as gunzipped

I encounter some problems when I try to do a docker run - I have git cloned your repo and I am pointing it to the right places in my command below:

sudo docker run --rm -it -v /tmp:/tmp \
-v <path_to>/sample_data:/home/bhaas/GITHUB/pasapipeline/sample_data \
pasa_gmap_2021:latest \
bash -c \
'cd /home/bhaas/GITHUB/pasapipeline/sample_data && \ /usr/local/src/PASApipeline/Launch_PASA_pipeline.pl \
-c sqlite.confs/alignAssembly.config \
-C -R --ALIGNER gmap \
-g genome_sample.fasta -t all_transcripts.fasta.clean'

-connecting to SQLite db: /tmp/sample_mydb_pasa.sqlite
-*** Running PASA pipeine:
* [Wed Jul 14 05:03:13 2021] Running CMD: /usr/local/src/PASApipeline/scripts/create_sqlite_cdnaassembly_db.dbi
 -c sqlite.confs/alignAssembly.config -S '/usr/local/src/PASApipeline/schema/cdna_alignment_sqliteschema'
* [Wed Jul 14 05:03:15 2021] Running CMD: samtools faidx genome_sample.fasta
* [Wed Jul 14 05:03:15 2021] Running CMD: samtools faidx all_transcripts.fasta.clean
* [Wed Jul 14 05:03:15 2021] Running CMD: /usr/local/src/PASApipeline/scripts/upload_transcript_data.dbi -M '/tmp/sample_mydb_pasa.sqlite' -t all_transcripts.fasta.clean  -f NULL
Committing...
* [Wed Jul 14 05:03:16 2021] Running CMD: /usr/local/src/PASApipeline/scripts/run_spliced_aligners.pl --aligners gmap --genome genome_sample.fasta --transcripts all_transcripts.fasta.clean -I 500000 -N 1 --CPU 2
CMD: /usr/local/src/PASApipeline/scripts/process_GMAP_alignments_gff3_chimeras_ok.pl --genome genome_sample.fasta  --transcripts all_transcripts.fasta.clean --CPU 2 -N 1  -I 500000  > gmap.spliced_alignments.gff3
CMD: gmap_build -D . -T . -d genome_sample.fasta.gmap -k 13 genome_sample.fasta >&2
-j flag not specified, so building regional hash tables with default 6-mers
Writing files under .
Not sorting chromosomes.  To sort chromosomes other ways, use the -s flag.
A transcript FASTA or genes file was given, but no transcriptome name was provided with the -c flag at /usr/local/bin/gmap_build line 201.
Error, cmd: gmap_build -D . -T . -d genome_sample.fasta.gmap -k 13 genome_sample.fasta >&2 died with ret (512)
at /usr/local/src/PASApipeline/scripts/process_GMAP_alignments_gff3_chimeras_ok.pl line 115.
Thread 1 terminated abnormally: Error, cmd:
/usr/local/src/PASApipeline/scripts/process_GMAP_alignments_gff3_chimeras_ok.pl --genome genome_sample.fasta  -
-transcripts all_transcripts.fasta.clean --CPU 2 -N 1  -I 500000  > gmap.spliced_alignments.gff3
 died with ret (512) at /usr/local/src/PASApipeline/scripts/../PerlLib/Process_cmd.pm line 18 thread 1.
        Process_cmd::process_cmd("/usr/local/src/PASApipeline/scripts/process_GMAP_alignments_g"...) called at
/usr/local/src/PASApipeline/scripts/run_spliced_aligners.pl line 109 thread 1
        main::run_gmap() called at /usr/local/src/PASApipeline/scripts/run_spliced_aligners.pl line 70 thread 1
        eval {...} called at /usr/local/src/PASApipeline/scripts/run_spliced_aligners.pl line 70 thread 1
ERROR, thread 1 exited with error Error, cmd:
/usr/local/src/PASApipeline/scripts/process_GMAP_alignments_gff3_chimeras_ok.pl --genome genome_sample.fasta  -
-transcripts all_transcripts.fasta.clean --CPU 2 -N 1  -I 500000  > gmap.spliced_alignments.gff3
 died with ret (512) at /usr/local/src/PASApipeline/scripts/../PerlLib/Process_cmd.pm line 18 thread 1.
        Process_cmd::process_cmd("/usr/local/src/PASApipeline/scripts/process_GMAP_alignments_g"...) called at
/usr/local/src/PASApipeline/scripts/run_spliced_aligners.pl line 109 thread 1
        main::run_gmap() called at /usr/local/src/PASApipeline/scripts/run_spliced_aligners.pl line 70 thread 1
        eval {...} called at /usr/local/src/PASApipeline/scripts/run_spliced_aligners.pl line 70 thread 1

Error, 1 threads failed.
Error, cmd: /usr/local/src/PASApipeline/scripts/run_spliced_aligners.pl --aligners gmap --genome genome_sample.
fasta --transcripts all_transcripts.fasta.clean -I 500000 -N 1 --CPU 2 died with ret 7424 No such file or direc
tory at /usr/local/src/PASApipeline/PerlLib/Pipeliner.pm line 187.
        Pipeliner::run(Pipeliner=HASH(0x26805b8)) called at /usr/local/src/PASApipeline/Launch_PASA_pipeline.pl
 line 1044

A similar job run using docker pull pasapipeline/pasapipeline runs fine without any issues using the same command line structure as above.

[tamuanand]

An update on the above

Turns out:

• gmap_build 2017-11-15 has -T to be Temporary build directory (may need to specify if you run out of space in your current directory)
• gmap_build 2021-05-27 has -T to be --transcripts=FILE FASTA file containing transcripts (required if specifying --transcriptomedb)

Hence, I had to change https://github.com/PASApipeline/PASApipeline/blob/master/scripts/process_GMAP_alignments_gff3_chimeras_ok.pl#L75
from
my $cmd = "gmap_build -D $genomeBaseDir -T $genomeBaseDir -d $genomeDir -k 13 $genome >&2";
to
my $cmd = "gmap_build -D $genomeBaseDir -d $genomeDir -k 13 $genome >&2";

After making the above changes to the local copy of process_GMAP_alignments_gff3_chimeras_ok.pl, I went about it like this

sudo docker run -td pasa_gmap_2021:latest

sudo docker ps  # to grab the container id and let's say that container id was 1234

sudo docker cp process_GMAP_alignments_gff3_chimeras_ok.pl 1234:/usr/local/src/PASApipeline/scripts/

sudo docker commit  1234 pasa_gmap_2021:v2

sudo docker kill 1234

Now, when I run it, the job runs fine without errors that I encountered above

sudo docker run --rm -it -v /tmp:/tmp \
-v <path_to>/sample_data:/home/bhaas/GITHUB/pasapipeline/sample_data \
pasa_gmap_2021:v2 \
bash -c \
'cd /home/bhaas/GITHUB/pasapipeline/sample_data && \ /usr/local/src/PASApipeline/Launch_PASA_pipeline.pl \
-c sqlite.confs/alignAssembly.config \
-C -R --ALIGNER gmap \
-g genome_sample.fasta -t all_transcripts.fasta.clean'

I still have to do other checks to make sure everything looks OK

I will keep updating this thread with my findings and hope it benefits you (and others)

[brianjohnhaas]

sounds good. If you want to put in some github pull requests for any changes that you need to make, that would be great!

…

On Wed, Jul 14, 2021 at 2:33 AM tamuanand ***@***.***> wrote: An update on the above <#197 (comment)> Turns out: - gmap_build 2017-11-15 has -T to be Temporary build directory (may need to specify if you run out of space in your current directory) - gmap_build 2021-05-27 has -T to be --transcripts=FILE FASTA file containing transcripts (required if specifying --transcriptomedb) Hence, I had to change https://github.com/PASApipeline/PASApipeline/blob/master/scripts/process_GMAP_alignments_gff3_chimeras_ok.pl#L75 from my $cmd = "gmap_build -D $genomeBaseDir -T $genomeBaseDir -d $genomeDir -k 13 $genome >&2"; to my $cmd = "gmap_build -D $genomeBaseDir -d $genomeDir -k 13 $genome >&2"; After making the above changes to the local copy of process_GMAP_alignments_gff3_chimeras_ok.pl, I went about it like this sudo docker run -td pasa_gmap_2021:latest sudo docker ps # to grab the container id and let's say that container id was 1234 sudo docker cp process_GMAP_alignments_gff3_chimeras_ok.pl 1234:/usr/local/src/PASApipeline/scripts/ sudo docker commit 1234 pasa_gmap_2021:v2 sudo docker kill 1234 Now, when I run it, the job runs fine without errors that I encountered above <#197 (comment)> sudo docker run --rm -it -v /tmp:/tmp \ -v <path_to>/sample_data:/home/bhaas/GITHUB/pasapipeline/sample_data \ pasa_gmap_2021:v2 \ bash -c \ 'cd /home/bhaas/GITHUB/pasapipeline/sample_data && \ /usr/local/src/PASApipeline/Launch_PASA_pipeline.pl \ -c sqlite.confs/alignAssembly.config \ -C -R --ALIGNER gmap \ -g genome_sample.fasta -t all_transcripts.fasta.clean' I still have to do other checks to make sure everything looks OK I will keep updating this thread with my findings and hope it benefits you (and others) — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#197 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ABZRKXYDYRCR43S2XHHBFXTTXUVVNANCNFSM5AFYDRMA> .

-- -- Brian J. Haas The Broad Institute http://broadinstitute.org/~bhaas <http://broad.mit.edu/~bhaas>

[tamuanand]

Hi @brianjohnhaas

I would like to seek your inputs on this:

A related question: If I have already built a genome index with gmap, is it possible to tell pasa to use this gmap_build/index instead of pasa trying to build it

[tamuanand]

As regards a PR, my suggestion would be if you could clone this in a separate branch (like dev), I can work off it and have the Dockerfile etc stuff updated where pasa uses GMAP 2021. I saw this because there might be folks who would like to use the current codebase

sounds good. If you want to put in some github pull requests for any changes that you need to make, that would be great!

[tamuanand]

Hi @brianjohnhaas

A question: If I want to use Docker and use mysql, do I need to have any instructions in the Dockerfile to start the mysql service.

Also, can I use this file as is and change my docker run command to point to this

Thanks

[brianjohnhaas]

It's not set up to use an existing gmap database currently. You could add this feature though.

…

On Wed, Jul 14, 2021 at 4:26 PM tamuanand ***@***.***> wrote: Hi @brianjohnhaas <https://github.com/brianjohnhaas> I would like to seek your inputs on this: A related question: If I have already built a genome index with gmap, is it possible to tell pasa to use this gmap_build/index instead of pasa trying to build it — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#197 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ABZRKXZSAHKBBFP4K4O5MHLTXXXIRANCNFSM5AFYDRMA> .

-- -- Brian J. Haas The Broad Institute http://broadinstitute.org/~bhaas <http://broad.mit.edu/~bhaas>

[brianjohnhaas]

I'd suggest just upgrading. The older releases are still available if someone wants to use the older version of gmap.

…

On Wed, Jul 14, 2021 at 4:31 PM tamuanand ***@***.***> wrote: As regards a PR, my suggestion would be if you could clone this in a separate branch (like dev), I can work off it and have the Dockerfile etc stuff updated where pasa uses GMAP 2021. I saw this because there might be folks who would like to use the current codebase sounds good. If you want to put in some github pull requests for any changes that you need to make, that would be great! — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#197 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ABZRKX4XWZB5LT65XJADNDLTXXX3HANCNFSM5AFYDRMA> .

-- -- Brian J. Haas The Broad Institute http://broadinstitute.org/~bhaas <http://broad.mit.edu/~bhaas>

[brianjohnhaas]

I haven't tried using the mysql connection through docker... only the sqlite. I figure it should be possible though. It might involve opening up some ports through the docker to connect to a separately running msyql instance.

…

On Wed, Jul 14, 2021 at 7:32 PM tamuanand ***@***.***> wrote: Hi @brianjohnhaas <https://github.com/brianjohnhaas> A question: If I want to use Docker and use mysql, do I need to have any instructions in the Dockerfile to start the mysql service. Also, can I use this file <https://github.com/PASApipeline/PASApipeline/blob/master/sample_data/mysql.confs/alignAssembly.config> as is and change my docker run command to point to this Thanks — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#197 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ABZRKXZODFFOA6YBMCVZREDTXYNB5ANCNFSM5AFYDRMA> .

-- -- Brian J. Haas The Broad Institute http://broadinstitute.org/~bhaas <http://broad.mit.edu/~bhaas>

[tamuanand]

Do you want me to submit a PR on the master branch and then submit my changes?

I'd suggest just upgrading. The older releases are still available if someone wants to use the older version of gmap.

[tamuanand]

Would it be possible for you to give minimalistic guidelines on this (using the mysql connection via docker). I know you have mentioned lack of time etc and I totally understand.

Hence, if at all it is possible on your end, I would really appreciate it

• does it need a pre-configured conf.txt with dbname, db_username, db_password etc that can be used with a "COPY" command on Dockerfile to add it to /usr/local/src/PASApipeline/pasa_conf/conf.txt
• any other config tips on this

I haven't tried using the mysql connection through docker... only the sqlite. I figure it should be possible though. It might involve opening up some ports through the docker to connect to a separately running msyql instance.