JBrowse-in-Galaxy v0.4 #507

Merged
merged 76 commits into from Dec 31, 2015

Conversation

Projects
None yet
4 participants
@erasche
Member

erasche commented Dec 29, 2015

It's finally here! Just in time for the new year :) Opening for review of the tool (under WIP) while I fix the installation recipe.

What's new

  • Advanced colour control
    • see sexy photos below
  • Ability to specify genetic code and have it reflected in track colouring
  • Dataset Collection support
    • Now you can specify a dc:list of BigWig files and have them all styled identically in your plot.
    • Works with updated XMFA to gff3/bigWig tools
    • If biopython/biopython#544 ever gets merged, this means support for all other MSA formats such as MAF display in JBrowse.
  • Ability to build a "full" jbrowse instance, or just the data directory
    • for use in WebApollo tools which will be out probably by gcc2016
  • Now clicking the eye goes straight to JBrowse without the interstitial.
    • this means the tool now requires whitelisting, but the old version was probably violating some security stuff anyway. I should fix that. Hmm.
    • Also makes the $output.files_path a bit cleaner which is nice.
  • (soon) fixed installation recipe
  • test cases! There are two enabled tests, and one commented out because I couldn't get it to behave properly. If anyone has suggestions on that, please let me know. I followed peter's example.

TODO before out of WIP

  • fix installation recipe
  • add more help documentation

Roadmap

(This is apparently not a happy subject for some, but) we're almost at feature parity with trackster. v0.5 will likely be a minor update which adds support for len files and loading fasta genomes from the built-in indices, which will put JBrowse at essentially feature parity with Trackster

The other last real feature on the roadmap before I hit 1.0 is support for protein sequences. I have work in various places but this will necessitate patches to upstream.

Please comment on #508 if you have other features you'd like to see

Colour Control

Say you have four gff3 files from four different gene callers, and wish to plot them all under a single group. You can do that with the added multiple="True", and setting colour to "automatic", means you get lovely things like this. (You could also manually specify the colour, and all four tracks would be the same colour)

selection_516

There are a couple different "colour score scaling" methods built in and automatically used. Here we see a blast XML dataset that's automatically been processed into a GFF3 representation. It takes advantage of the "blast" scoring mechanism which writes a horrifically complex JBrowse config that automatically calculates opacity in relation to the e-value of the hit.

selection_521

Here we see the "linear" scoring mechanism with a manually specified min/max of 0/1000. The min/max values can be automatically detected from the GFF3 file, or specified in the configuration

selection_515

Lastly BigWig files are more easily configurable. You can specify the colour pivot points, the positive/negative colours, or you can let them automatically be determined. The below plots showcase features like custom min/maxes, custom pivot points, etc.

selection_517

Tool UI

A preview of the tool's interface. All of the data from the below images gets compiled in the background into a JBrowse instance with some crazy javascript color functions for track styling, as hinted by the options: :)

selection_503
selection_504

Would produce the following:

selection_505

The javascript color functions are a bit crazy, and worse in my contrived test dataset. If you don't manually select a colour, they're able to either use an automatically chosen colour, or to pull a "color=#..." attribute from the GFF3 data. The first CDS of Merlin_1_mRNA is the automatically chosen colour from along brewer palletes, the blue and green features are pulling that data from a gff3 attribute.

selection_506

Eric Rasche and others added some commits Jun 23, 2015

Updates to JBrowse for more colour configuration
- extensive updates to enable brewer palettes/custom colour config for
  bigwig tracks
- various bug fixes and re-architecting
- may re-work brewer palettes eventually--right now we just use the
  extrma. I don't know if JBrowse supports gradiations on the BigWig
  tracks, other than in "Density" mode
HTML, logging, tool XML updates
- removed useless redirect
- adjusted message
- moved logging statements to log.*, to be disabled by setting to INFO
- disabled pretty unused brewer code
- fixed # in sanitizer
- updated test case
Progress towards 0.4
- recursively search up/down feature tree for score & colour qualifiers
- reorg code
- temporarily disable all except gff3 tracks
- does not match up with current XML
Restructure Color module
gene features is pretty much done! Color is awesome
+ <install version="1.0">
+ <actions>
+ <action type="download_by_url" sha256sum="a8bb2a56f1ca0470df2d072811db1f90fd6756feed92d8656eba1e6f67f438d3">
+ http://jbrowse.org/wordpress/wp-content/plugins/download-monitor/download.php?id=101&amp;fmt=.zip

This comment has been minimized.

@bgruening

bgruening Dec 31, 2015

Member

What is our policy here? Cargo or not or only in a few month?
I try to urge people to use it currently ... maybe this is too early?

@bgruening

bgruening Dec 31, 2015

Member

What is our policy here? Cargo or not or only in a few month?
I try to urge people to use it currently ... maybe this is too early?

This comment has been minimized.

@erasche

erasche Dec 31, 2015

Member

This is sort of a special case

  • given that our download_proxy action is not widely available
  • and the upstream URL explicitly has "download-monitor" in the URL so they're obviously tracking downloads
  • and they have a very good history of maintaining old releases

I was willing to use the upstream URL.

@erasche

erasche Dec 31, 2015

Member

This is sort of a special case

  • given that our download_proxy action is not widely available
  • and the upstream URL explicitly has "download-monitor" in the URL so they're obviously tracking downloads
  • and they have a very good history of maintaining old releases

I was willing to use the upstream URL.

+ <repository name="package_perl_5_18" owner="iuc" prior_installation_required="True" />
+ </package>
+ <package name="bundle_jbrowse" version="1.12.0">
+ <repository name="package_perl_bundle_jbrowse_1_12_0" owner="iuc" prior_installation_required="True" />

This comment has been minimized.

@bgruening

bgruening Dec 31, 2015

Member

prior_installation_required="True" not needed in both cases

@bgruening

bgruening Dec 31, 2015

Member

prior_installation_required="True" not needed in both cases

This comment has been minimized.

@erasche

erasche Dec 31, 2015

Member

no, but there are plans to use that as the default action everywhere. https://botbot.me/freenode/galaxyproject/search/?q=prior_installation_required

@erasche

erasche Dec 31, 2015

Member

no, but there are plans to use that as the default action everywhere. https://botbot.me/freenode/galaxyproject/search/?q=prior_installation_required

This comment has been minimized.

@bgruening

bgruening Jan 1, 2016

Member

@gregvonkuster what do you think about this? I recall a discussion with you at the beginning of the TS that this might not be a good idea.

@bgruening

bgruening Jan 1, 2016

Member

@gregvonkuster what do you think about this? I recall a discussion with you at the beginning of the TS that this might not be a good idea.

This comment has been minimized.

@gregvonkuster

gregvonkuster Jan 1, 2016

Contributor

@erasche @bgruening I do not agree that this should be the default attribute for all repository dependency definitions. This flag is necessary only for compile time, so if a required package is not needed at compile time for a dependent package, the flag should not be used - I think doing so opens the door to potential problems. For example, if package A requires package B to be installed at A's compile time, and both packages have prior installation required set to True, then attempts to install package A can occur before package B is installed. The more complex the dependency chain, the more problems that can be introduced. This whole recipe process is obviously not ideal, but it looks like it should be replaced fairly soon with a better solution.

@gregvonkuster

gregvonkuster Jan 1, 2016

Contributor

@erasche @bgruening I do not agree that this should be the default attribute for all repository dependency definitions. This flag is necessary only for compile time, so if a required package is not needed at compile time for a dependent package, the flag should not be used - I think doing so opens the door to potential problems. For example, if package A requires package B to be installed at A's compile time, and both packages have prior installation required set to True, then attempts to install package A can occur before package B is installed. The more complex the dependency chain, the more problems that can be introduced. This whole recipe process is obviously not ideal, but it looks like it should be replaced fairly soon with a better solution.

+ <package sha256sum="cbfef8376d3d9026483360150e3745a27dad78e995348d0c08ef5b2e4374170c">
+ https://depot.galaxyproject.org/software/Bio-GFF3/Bio-GFF3_2_src_all.tar.gz
+ </package>
+

This comment has been minimized.

@bgruening

bgruening Dec 31, 2015

Member

Nice, it's so easy in the end :)

@bgruening

bgruening Dec 31, 2015

Member

Nice, it's so easy in the end :)

This comment has been minimized.

@erasche

erasche Dec 31, 2015

Member

will be even easier with your download by proxy action ;)

@erasche

erasche Dec 31, 2015

Member

will be even easier with your download by proxy action ;)

This comment has been minimized.

@bgruening

bgruening Dec 31, 2015

Member

Oh boy have I posted this already to galaxy-dev?

@bgruening

bgruening Dec 31, 2015

Member

Oh boy have I posted this already to galaxy-dev?

This comment has been minimized.

@erasche

erasche Dec 31, 2015

Member

I don't think so. If so, I don't remember any discussion.

@erasche

erasche Dec 31, 2015

Member

I don't think so. If so, I don't remember any discussion.

This comment has been minimized.

@bgruening

bgruening Jan 1, 2016

Member

ok, send it ... Oo

@bgruening

bgruening Jan 1, 2016

Member

ok, send it ... Oo

tools/gff3_rebase/gff3_rebase.py
+ subfeatures=False):
+
+ new_subfeatures = child_features[feature.id]
+ # TODO: update starts

This comment has been minimized.

@bgruening

bgruening Dec 31, 2015

Member

todo still relevant?

@bgruening

bgruening Dec 31, 2015

Member

todo still relevant?

+ # Then update the location of the actual feature
+ __update_feature_location(x, feature, protein2dna)
+
+ if interpro:

This comment has been minimized.

@bgruening

bgruening Dec 31, 2015

Member

Awesome Introproscan support :)

@bgruening

bgruening Dec 31, 2015

Member

Awesome Introproscan support :)

tools/jbrowse/jbrowse-fromdir.xml
@@ -0,0 +1,40 @@
+<tool id="jbrowse-upgrade" name="JBrowse - Data Directory to Standalone" version="0.3">

This comment has been minimized.

@bgruening

bgruening Dec 31, 2015

Member

jbrowse-upgrade or jbrowse_to_standalone.

@bgruening

bgruening Dec 31, 2015

Member

jbrowse-upgrade or jbrowse_to_standalone.

This comment has been minimized.

@bgruening

bgruening Dec 31, 2015

Member

What about a special JBrowse datatype and this as converter and not as tool?
Maybe a converter is too hidden?

@bgruening

bgruening Dec 31, 2015

Member

What about a special JBrowse datatype and this as converter and not as tool?
Maybe a converter is too hidden?

tools/jbrowse/jbrowse.xml
+--outdir $output.files_path;
+
+#if str($standalone) == "Complete":
+ cat $output.files_path/index.html > $output;

This comment has been minimized.

@bgruening

bgruening Dec 31, 2015

Member

mv is probably better

@bgruening

bgruening Dec 31, 2015

Member

mv is probably better

This comment has been minimized.

@erasche

erasche Dec 31, 2015

Member

sure, can do

@erasche

erasche Dec 31, 2015

Member

sure, can do

tools/jbrowse/jbrowse.xml
+
+## Ugly testing hack since I cannot get <extra_files> to test the files I want to test. Hmph.
+#if str($uglyTestingHack) == "enabled":
+ cp $trackxml $output

This comment has been minimized.

- <param label="This is match/match_part data"
- type="boolean" name="match_part" truevalue="true" falsevalue="false" />
+ <expand macro="input_conditional" label="GFF/GFF3/BED Track Data" format="gff,gff3,bed" />
+ <conditional name="match_part" label="match/match_part data">

This comment has been minimized.

@bgruening

bgruening Dec 31, 2015

Member

Is this complicated conditional really needed? You could simply do the text field optional.

@bgruening

bgruening Dec 31, 2015

Member

Is this complicated conditional really needed? You could simply do the text field optional.

This comment has been minimized.

@erasche

erasche Dec 31, 2015

Member

Just trying to hide UI for people who don't need it. The Yes/No for using match/match_part is very clear, and I don't mind writing a conditional if it makes the UI clearer for end users.

@erasche

erasche Dec 31, 2015

Member

Just trying to hide UI for people who don't need it. The Yes/No for using match/match_part is very clear, and I don't mind writing a conditional if it makes the UI clearer for end users.

+ token_select_name="color_select"
+ token_automatic_label="Automatically selected"
+ token_manual_label="Manual Color Selection">
+ <expand macro="jb_color"

This comment has been minimized.

@bgruening

bgruening Dec 31, 2015

Member

Indentation could be more clear.

@bgruening

bgruening Dec 31, 2015

Member

Indentation could be more clear.

This comment has been minimized.

@erasche

erasche Dec 31, 2015

Member

Fixed

@erasche

erasche Dec 31, 2015

Member

Fixed

@bgruening

This comment has been minimized.

Show comment
Hide comment
@bgruening

bgruening Dec 31, 2015

Member

I don't really understand the usefulness for the data-only result-dataset. If this will be used in Apollo later on, why can we not remove this one directory that is useless before handing it over to Apollo and drop this "filetype".

Member

bgruening commented Dec 31, 2015

I don't really understand the usefulness for the data-only result-dataset. If this will be used in Apollo later on, why can we not remove this one directory that is useless before handing it over to Apollo and drop this "filetype".

@@ -6,106 +6,317 @@
<expand macro="requirements"/>

This comment has been minimized.

@bgruening

bgruening Dec 31, 2015

Member

Increase the version number to 0.4

@bgruening

bgruening Dec 31, 2015

Member

Increase the version number to 0.4

This comment has been minimized.

@erasche

erasche Dec 31, 2015

Member

Done

@bgruening

This comment has been minimized.

Show comment
Hide comment
@bgruening

bgruening Dec 31, 2015

Member

This works great for me. To make it work for everyone we need to get this in: galaxyproject/galaxy#1410
Or people need to activate it by default.

Great work Eric! Happy new year!

Member

bgruening commented Dec 31, 2015

This works great for me. To make it work for everyone we need to get this in: galaxyproject/galaxy#1410
Or people need to activate it by default.

Great work Eric! Happy new year!

@erasche

This comment has been minimized.

Show comment
Hide comment
@erasche

erasche Dec 31, 2015

Member

I don't really understand the usefulness for the data-only result-dataset. If this will be used in Apollo later on, why can we not remove this one directory that is useless before handing it over to Apollo and drop this "filetype".

  • JBrowse outputs consist of a parent directory of stuff related to JBrowse, and within that directory, one folder named "data" that's the actual processed files.
  • JBrowse folders are not small. They can be 6-20 Mb, whereas the data directory is only 400k or so.
  • When Apollo is going to use these datasets, it ONLY needs the 400k data directory, and nothing else.
  • I solicited feedback on possible options in #346 but no one had comments.

So I decided it was best if JBrowse had the ability to produce both a "standalone" and the data dir without JBrowse. What I really wanted was that all of JBrowses outputs could be just the data directory rather than making a full copy of JBrowse every time you want to produce a visualization, but I didn't see any technical solutions to that problem which didn't involve really unpleasant things. E.g. hacking JBrowse into a visualization plugin which would require splitting out static resources and an index.html template that would require some really, really ugly hacks and a separate viz plugin to install just for JBrowse and a special JBrowse datatype which means people would have to be on 16.01, rather than 15.05 or so.

Member

erasche commented Dec 31, 2015

I don't really understand the usefulness for the data-only result-dataset. If this will be used in Apollo later on, why can we not remove this one directory that is useless before handing it over to Apollo and drop this "filetype".

  • JBrowse outputs consist of a parent directory of stuff related to JBrowse, and within that directory, one folder named "data" that's the actual processed files.
  • JBrowse folders are not small. They can be 6-20 Mb, whereas the data directory is only 400k or so.
  • When Apollo is going to use these datasets, it ONLY needs the 400k data directory, and nothing else.
  • I solicited feedback on possible options in #346 but no one had comments.

So I decided it was best if JBrowse had the ability to produce both a "standalone" and the data dir without JBrowse. What I really wanted was that all of JBrowses outputs could be just the data directory rather than making a full copy of JBrowse every time you want to produce a visualization, but I didn't see any technical solutions to that problem which didn't involve really unpleasant things. E.g. hacking JBrowse into a visualization plugin which would require splitting out static resources and an index.html template that would require some really, really ugly hacks and a separate viz plugin to install just for JBrowse and a special JBrowse datatype which means people would have to be on 16.01, rather than 15.05 or so.

@erasche erasche referenced this pull request in GMOD/jbrowse Dec 31, 2015

Closed

JBrowse/Analytics #674

@bgruening

This comment has been minimized.

Show comment
Hide comment
@bgruening

bgruening Dec 31, 2015

Member

I clearly missed #346, sorry. If you really care about the 20MB I guess this is the best way forward. +1 from me. Let me know if you consider this one out of WIP and I will merge this in 2015 :)

Awesome!

Member

bgruening commented Dec 31, 2015

I clearly missed #346, sorry. If you really care about the 20MB I guess this is the best way forward. +1 from me. Let me know if you consider this one out of WIP and I will merge this in 2015 :)

Awesome!

@erasche

This comment has been minimized.

Show comment
Hide comment
@erasche

erasche Dec 31, 2015

Member

@bgruening not a problem. I'm really hoping there's a better solution in the future, but I just haven't found anything I'm happy with yet. The data dir is a stopgap solution until we get to the point where datatypes, viz plugins, and tools are all easy to install and depend upon.

I'm almost done writing the help text, and then it should be good to go :)

Member

erasche commented Dec 31, 2015

@bgruening not a problem. I'm really hoping there's a better solution in the future, but I just haven't found anything I'm happy with yet. The data dir is a stopgap solution until we get to the point where datatypes, viz plugins, and tools are all easy to install and depend upon.

I'm almost done writing the help text, and then it should be good to go :)

erasche added some commits Dec 31, 2015

@erasche erasche added enhancement high priority and removed wip labels Dec 31, 2015

@erasche

This comment has been minimized.

Show comment
Hide comment
@erasche

erasche Dec 31, 2015

Member

Ok @bgruening, ready to go if you're OK with it :)

Member

erasche commented Dec 31, 2015

Ok @bgruening, ready to go if you're OK with it :)

@bgruening

This comment has been minimized.

Show comment
Hide comment
@bgruening

bgruening Dec 31, 2015

Member

amenable?

amenable?

This comment has been minimized.

Show comment
Hide comment
@erasche

erasche Dec 31, 2015

Member

ack! Forgot to turn on :set spell

Member

erasche replied Dec 31, 2015

ack! Forgot to turn on :set spell

@bgruening

This comment has been minimized.

Show comment
Hide comment
@bgruening

bgruening Dec 31, 2015

Member

preferable?

preferable?

@bgruening

This comment has been minimized.

Show comment
Hide comment
@bgruening

bgruening Dec 31, 2015

Member

palatable?

palatable?

@bgruening

This comment has been minimized.

Show comment
Hide comment
@bgruening

bgruening Dec 31, 2015

Member

I like it a lot. Also I'm wondering why the help is not at the bottom and planemo lint gives green light. But that’s it, ready to merge!

Member

bgruening commented Dec 31, 2015

I like it a lot. Also I'm wondering why the help is not at the bottom and planemo lint gives green light. But that’s it, ready to merge!

@erasche

This comment has been minimized.

Show comment
Hide comment
@erasche

erasche Dec 31, 2015

Member

It fails locally, I'll move it really quick.

Applying linter xml_order... WARNING
.. WARNING: Best practice violation [tests] elements should come before [help]
Member

erasche commented Dec 31, 2015

It fails locally, I'll move it really quick.

Applying linter xml_order... WARNING
.. WARNING: Best practice violation [tests] elements should come before [help]
@erasche

This comment has been minimized.

Show comment
Hide comment
@erasche

erasche Dec 31, 2015

Member

Planemo isn't failing because the .travis.yml says --fail_level error.

Member

erasche commented Dec 31, 2015

Planemo isn't failing because the .travis.yml says --fail_level error.

bgruening added a commit that referenced this pull request Dec 31, 2015

@bgruening bgruening merged commit 40321d8 into master Dec 31, 2015

2 checks passed

continuous-integration/travis-ci/pr The Travis CI build passed
Details
continuous-integration/travis-ci/push The Travis CI build passed
Details

@bgruening bgruening deleted the jbrowse04 branch Dec 31, 2015

@bgruening

This comment has been minimized.

Show comment
Hide comment
@bgruening

bgruening Dec 31, 2015

Member

Awesome!

Member

bgruening commented Dec 31, 2015

Awesome!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment