Len file missing for indexed genomes at http://usegalaxy.org

[jennaj]

Problem

Data migration incomplete for new indexed genomes. Certain tools will fail.

Tools impacted

Trackster
Extract Genomic DNA
Wig/BedGraph-to-bigWig
Others that require a .len file

Genomes with issue

rn6
danRer10

Related: #2530

Master Genome Ticket: https://github.com/galaxyproject/galaxy/issues/1470

Workaround: Obtain the rn6 or danRer10 reference genome from UCSC, FTP it to Galaxy, and use it as a Custom Reference Genome. Help for FTP and Custom Genomes can be found on the Galaxy support wiki: https://wiki.galaxyproject.org/Support

ping @natefoo

[jennaj]

Hello - another ping @natefoo or maybe @martenson ?

[jennaj]

@galaxyproject/guac Could we please work to resolve this soon? Has been around a while, the issue is clear (just needs to be done), and will help users. Thx!

[natefoo]

I could not find any DM-generated length files so I set up a new space for these two and generated them. They should be available on Main now.

[jennaj]

@natefoo Trackster is not working for dm6 again (it is not in the trackster builds list). danRer10 is fine.

[natefoo]

I'm finally setting up the UCSC build and chrom length fetcher to run automatically and update them in CVMFS once a weekmonth, which should take care of this (including dm6). I don't think this has been run automatically since Test and Main moved to TACC.

[natefoo]

Okay, Test is now using the updated data from UCSC. This includes new builds for datasets, new builds recognized for the UCSC browsers ("display at" links) and new chrom length files. I'll do Main tomorrow morning after Anton's workshop.

I set the UCSC updating process up to run on the first Wednesday of the month. Any new genomes added to UCSC at that time will appear in Test/Main once they are restarted after that date (that part will still be manually, but they're typically restarted every few days for tool upgrades, etc.).

xref: galaxyproject/usegalaxy-playbook@f3257f3

[bgruening]

Thats cool @natefoo! Are in the same process also indices build? Is there a mechanism needed to build them or contribute them. Thinking about contribution possibilities, now that we have more users and mirrors that usegalaxy.org.

[natefoo]

This is now also done for Main.

xref: galaxyproject/usegalaxy-playbook@319ff47

[natefoo]

@bgruening No, no indices. This would be more complicated since they use DMs, I'd need to write a BioBlend script and wrap starting and stopping a dedicated Galaxy server with it. I'd like to do this, though!

[bgruening]

Oh @natefoo no! We have this for you: https://github.com/galaxyproject/ephemeris/blob/master/ephemeris/run_data_managers.py

And this is included in Galaxy Docker, so you can do: https://github.com/galaxyproject/training-material/blob/master/topics/chip-seq/docker/Dockerfile.old#L26

In combination with a mountpoint, you could start, run, deploy this automatically I think.

[natefoo]

!!!!!!!

You are my hero, @bgruening!

[jennaj]

Hi - Some tools use the fasta_indexes.loc file still (Freebayes in an example). The new DM databases are not being populated into this table (example: danRer10).

Any idea what is going wrong? Does the fetch DM need a change?

ping @blankenberg

[blankenberg]

@jennaj Freebayes uses samtool indexes, which are contained in the rather ambiguously named fasta_indexes table. Are you building samtool indexes from the all_fasta table with the SAM FASTA index data manager?

[jennaj]

This was from a user question. I reviewed the tip files and it calls the fasta_indexes.loc which is used to populate the fasta_indexes table, right? I checked our data and this is not populated for the genomes we had to do some manual fixes for. I did run SAMTools indexes on them with the DM but don't see the data in the sam_fa_indexes.loc/table now. It is in the picard_indexes.loc/table. I always run both and in order with other DMs: fetch (w/ create dbkey in this case), sam, picard, 2bit, then others. Rats - I wonder what went wrong or I did wrong. danRer10 is an example. Will look into what actually happened - the DM history is still intact.

[jennaj]

We can close this out.

Updating the processes for adding indexes to use ephemeris/run_data_managers.py on main is tracked in this ticket now: galaxyproject/usegalaxy-playbook#38

Duplicates in our data are in this ticket: galaxyproject/usegalaxy-playbook#55