URL control of tracks using category name

[childers]

Is there support for category level track control via the URL? I've been going through the docs without luck, along with testing it on our instances, also without luck.

Thanks!

[enuggetry]

Thanks for the elaboration, Chris.

----------------------- from Chris.
We already use the URL controls for managing what tracks to display a lot. On our gene pages we are embedding a jbrowse view and want to enable the Official Gene Set track to give a visual window for the gene, but that is difficult to do programatically in a way that will scale, especially since some OGS have multiple tracks. If we could have an option similar to the URL tack controls for categories, we could use a string like "&categories=Official%2bGene%2bSet" to turn on all of the tracks in the "Official Gene Set" category. Here is an example from one of our instances with multiple OGS tracks:
https://apollo.nal.usda.gov/oncfas_training/jbrowse/?loc=Scaffold1%3A1182070..1288681&tracks=DNA%2CAnnotations%2COGS_v1.1%2COGS_v1.1_pseudogene%2COGS_v1.1_rRNA&highlight=

The issue we have is with implementing this programmatically when there is variation in the number of tracks and in the naming, both from one organism to the next and as OGS versions get incremented. We are already support dozens of organisms, and we're expecting to scale to hundreds, and somewhere far down the line, to thousands.

I hope this helps to better describe the request, and how it would help our specific needs. Categories have been critical to our project for organizing a large number of tracks in a meaningful way, and I'd love to have more flexibility in how we can use them. It would also be nice to have default categories enables, similar to the default tracks.

[enuggetry]

So, here are some questions we should try to resolve:

Right now, the track list is given on the URL query, If we can pass a category=, should we translate this to a track list? If we are dealing with track lists, then we could easily run into the HTTP GET limit as defined by the server. This will probably happen even use say a track ID instead of the track path that we currently use.

Next: what if, after displaying tracks of the category and we selectively prune x tracks from the category, or add tracks. Do we want to be able to share the modified result?

GET request size: We can probably tune the HTTP GET request size on the server, but there are no guarantees for passing through a URL shortener or other proxy. So, I think, for hundreds or thousands of tracks, listing the tracks in the GET request gets to be impractical, from a general perspective.

If we need to share the changed track list, maybe we need to employ the server to maintain a "selected" profile each time the selection is modified and that some kind of track-profile-id can be passed in the query instead of the explicit.

[childers]

@enuggetry Thank you for the interesting implementation ideas. I'm including a more detailed description, including my thoughts on how it might look from the user perspective and an example of how we're organizing our tracks.

We already use the URL controls for managing what tracks to display a lot. On our gene pages we are embedding a jbrowse view and want to enable the Official Gene Set track to give a visual window for the gene, but that is difficult to do programatically in a way that will scale, especially since some OGS have multiple tracks. If we could have an option similar to the URL tack controls for categories, we could use a string like "&categories=Official%2bGene%2bSet" to turn on all of the tracks in the "Official Gene Set" category. Here is an example from one of our instances with multiple OGS tracks:
https://apollo.nal.usda.gov/oncfas_training/jbrowse/?loc=Scaffold1%3A1182070..1288681&tracks=DNA%2CAnnotations%2COGS_v1.1%2COGS_v1.1_pseudogene%2COGS_v1.1_rRNA&highlight=

The issue we have is with implementing this programmatically when there is variation in the number of tracks and in the naming, both from one organism to the next and as OGS versions get incremented. We are already support dozens of organisms, and we're expecting to scale to hundreds, and somewhere far down the line, to thousands.

I hope this helps to better describe the request, and how it would help our specific needs. Categories have been critical to our project for organizing a large number of tracks in a meaningful way, and I'd love to have more flexibility in how we can use them. It would also be nice to have default categories enables, similar to the default tracks.

[enuggetry]

Got a first-pass implementation, here. It is implemented as a JBrowse plugin.
https://github.com/GMOD/jbrowse/tree/Eric_work_f1/plugins/CategoryUrl

Installation instructions in the readme file

Example:
http://jbrowse.org/code/sandbox-113/?data=sample_data%2Fjson%2Fvolvox&loc=ctgA%3A9891..32316&cat=Miscellaneous

I noticed one track in the example seems to be left out. So, needs a little work.
Not sure how it will handle a larger data set yet.

[childers]

I've tested this on a few different instances (and different versions of JBrowse), with no success. Here is Plugin section of the trackList.json file:
"plugins" : [
{
"location" : "./plugins/CategoryURL",
"name" : "CategoryURL"
},
{
"location" : "./plugins/WebApollo",
"name" : "WebApollo"
}
],

Our categories categories are more complex, will that affect things? Will it handle giving a top level category and load the sub sections? (for example can I use "cat=1. Official Gene Set" to load both sub-categories?) In your example, "cat=Lucy" and "cat=Lucy / Goosey / Juicy" don't work.
"category" : "1. Official Gene Set/1. Protein Coding Genes",
"category" : "1. Official Gene Set/2. Noncoding Genes",
"category" : "0. Reference Assembly",
"category" : "BCM_v0.5.3/1. Gene Sets/Primary Gene Sets: Protein Coding",
"category" : "BCM_v0.5.3/3. Mapped Proteins/Protein2genome",

Please see one of our live instances for an example of our current setup. This instance does not have the Category URL plugin enabled yet.
https://apollo.nal.usda.gov/oncfas_training/jbrowse/?loc=Scaffold1%3A1187139..1780755&tracks=DNA%2CAnnotations&highlight=

[enuggetry]

Check "CategoryUrl"
Not sure how it would deal with spaces in the category name, but try that
first.
Then try %20 in place of spaces in URL.

If not, should be a quick fix.

I've tested this on a few different instances (and different versions of
JBrowse), with no success. Here is Plugin section of the trackList.json
file:
"plugins" : [
{
"location" : "./plugins/CategoryURL",
"name" : "CategoryURL"
},
{
"location" : "./plugins/WebApollo",
"name" : "WebApollo"
}
],

Our categories categories are more complex, will that affect things? Will
it handle giving a top level category and load the sub sections? (for
example can I use "cat=1. Official Gene Set" to load both sub-categories?)
In your example, "cat=Lucy" and "cat=Lucy / Goosey / Juicy" don't work.
"category" : "1. Official Gene Set/1. Protein Coding Genes",
"category" : "1. Official Gene Set/2. Noncoding Genes",
"category" : "0. Reference Assembly",
"category" : "BCM_v0.5.3/1. Gene Sets/Primary Gene Sets: Protein Coding",
"category" : "BCM_v0.5.3/3. Mapped Proteins/Protein2genome",

Please see one of our live instances for an example of our current setup.
This instance does not have the Category URL plugin enabled yet.
https://apollo.nal.usda.gov/oncfas_training/jbrowse/?loc=Scaffold1%3A1187139..1780755&tracks=DNA%2CAnnotations&highlight=

—
Reply to this email directly or view it on GitHub
#618 (comment).

[bellerbrock]

Hi,
Is this plugin still being developed? I'm finishing up a project now where it would be very useful. Unfortunately when I attempt to use it I get the following error:

Does CategoryUrl assume I've run generate_names? generating a complete set would be difficult in my case . . .

[enuggetry]

Hmm... are there any earlier errors besides the ones shown in your screen
capture?

CategoryUrl is not associated with names at all. It is associated with the
track category that is used to group tracks in the hierarchical track
selector.

On Thu, Nov 5, 2015 at 3:20 PM, Bryan Ellerbrock notifications@github.com
wrote:

Hi,
Is this plugin still being developed? I'm finishing up a project now where
it would be very useful. Unfortunately when I attempt to use it I get the
following error:

[image: screenshot from 2015-11-05 18 14 50]
https://cloud.githubusercontent.com/assets/11297346/10984730/7b479922-83e9-11e5-8504-663f57a937ec.png

Does CategoryUrl assume I've run generate_names? generating a complete set
would be difficult in my case . . .

—
Reply to this email directly or view it on GitHub
#618 (comment).

[bellerbrock]

No earlier errors, but I think I was following an older plugin configuration by putting the following in my jbrowse.conf file

plugins . CategoryUrl ]
location = plugins/CategoryUrl

I'm using JBrowse 1.11.6, so I've removed those lines and just have "plugins": [ 'CategoryUrl' ], in my trackList.json file. When I try to load an instance with the reference sequence, gene exons, and all the tracks in a category called Diversity, I get this:

Anything else I can try to help figure this out?

[enuggetry]

It looks good, right now.

The 404 errors you are getting there are inconsequential to the plugin
operation.
The plugin does not use a CSS file, so you would expect that error.

The 404 on root.json is probably because you have not run generate-names.
But, that is also inconsequential to the plugin and generally not required,
either.
http://gmod.org/wiki/JBrowse_Configuration_Guide#generate-names.pl

The plugin should now work for you.

Perhaps it's not spelled out clearly enough yet. The cat= URL parameter
allows you display tracks with the given category defined in the track
metadata that are used to group tracks in the hierarchical track selector.
For example:

"category" : "Transcripts",

​

Please do report any problems. It is a rather new plugin.

On Fri, Nov 6, 2015 at 7:55 AM, Bryan Ellerbrock notifications@github.com
wrote:

No earlier errors, but I think I was following an older plugin
configuration by putting the following in my jbrowse.conf file

plugins . CategoryUrl ]
location = plugins/CategoryUrl

I'm using JBrowse 1.11.6, so I've removed those lines and just have "plugins":
[ 'CategoryUrl' ], in my trackList.json file. When I try to load an
instance with the reference sequence, gene exons, and all the tracks in a
category called Diversity, I get this:

[image: screenshot from 2015-11-06 10 53 16]
https://cloud.githubusercontent.com/assets/11297346/11001317/9f894f84-8474-11e5-9bd6-382eff6fc175.png

Anything else I can try to help figure this out?

—
Reply to this email directly or view it on GitHub
#618 (comment).

[bellerbrock]

Ok, thank you for the clarification. I'm glad the plugin even exists, considering it's so new! And I appreciate the time you've taken to respond.

I think the urls have been cut out of the screenshots,so maybe that has caused some confusion. Here is the instance I've been testing out if you want to look.

To clarify, the tracks I've specified individually in the url with &tracks=DNA%2CGene%20exons are loading, but &cat=Diversity isn't loading the 13 vcf tracks in the category Diversity. So far I've thought this could be due to the fact that Diversity has many sub categories, for example Diversity/NEXTGEN/Imputed, but when I try a single subcategory &cat=Diversity%2FNEXTGEN%2FImputed that doesn't work either. Getting rid of subcategories altogether, I set up 4 extra vcf tracks in a simple category called Test, but I haven't had any luck displaying those with &cat=Test.

I'm running generate_names.pl now to see if it makes a difference. But unfortunately the scale of the project means I have hundreds of separate jbrowse instances set to display a vcf track for each accession contained in an individual phenotyping trial. So it would be very helpful to be able to pass &cat=Diversity in the url to display them all, regardless of trial. And unfortunately quite difficult to run generate_names.pl for each instance.

[enuggetry]

Yeah, the subcategories are going to throw it off. It's not too smart
yet. Let me take a look and see if I can make it a little smarter.

On Fri, Nov 6, 2015 at 12:15 PM, Bryan Ellerbrock notifications@github.com
wrote:

Ok, thank you for the clarification. I'm glad the plugin even exists,
considering it's so new! And I appreciate the time you've taken to respond.

I think the urls have been cut out of the screenshots,so maybe that has
caused some confusion. Here
https://cassavabase.org/jbrowse_cassavabase/current/?data=data%2Fjson%2Ftrials%2F405&tracks=DNA%2CGene%20exons&cat=Test&highlight=&loc=Chromosome01%3A13983212..20975156
is the instance I've been testing out if you want to look.

To clarify, the tracks I've specified individually in the url with
&tracks=DNA%2CGene%20exons are loading, but &cat=Diversity isn't loading
the 13 vcf tracks in the category Diversity. So far I've thought this
could be due to the fact that Diversity has many sub categories, for
example Diversity/NEXTGEN/Imputed, but when I try a single subcategory
&cat=Diversity%2FNEXTGEN%2FImputed that doesn't work either. Getting rid
of subcategories altogether, I set up 4 extra vcf tracks in a simple
category called Test, but I haven't had any luck displaying those with
&cat=Test.

I'm running generate_names.pl now to see if it makes a difference. But
unfortunately the scale of the project means I have hundreds of separate
jbrowse instances set to display a vcf track for each accession contained
in an individual phenotyping trial. So it would be very helpful to be able
to pass &cat=Diversity in the url to display them all, regardless of trial.
And unfortunately quite difficult to run generate_names.pl for each
instance.

—
Reply to this email directly or view it on GitHub
#618 (comment).

[bellerbrock]

Thanks, I appreciate it!

On a different topic, is there a recommended way to copy and update the plugin code? I could just git clone it and pull updates if it were it's own repo, but there doesn't seem to be an easy way to do this with sub directories. And I didn't think to set up my overall JBrowse 1.11.6 installation with a git clone.

[enuggetry]

The new iteration supports category/sub-category.
Hopefully, this can address Chris Childer's requirement, too.

[bellerbrock]

Alright! It's working beautifully with my Diversity/NEXTGEN/Imputed subcategory now. Thanks for added functionality!