Skip to content

Commit

Permalink
Merge remote-tracking branch 'origin/dev' into 1147_npm_tabix
Browse files Browse the repository at this point in the history
  • Loading branch information
rbuels committed Sep 19, 2018
2 parents 52074df + 556129a commit 021e4a3
Show file tree
Hide file tree
Showing 23 changed files with 437 additions and 193 deletions.
2 changes: 1 addition & 1 deletion docs/site/config_canvas_features.md
Expand Up @@ -7,7 +7,7 @@ CanvasFeatures are an alternative to HTMLFeatures and display features on the ge

The flatfile-to-json.pl by can output CanvasFeatures using the --trackType CanvasFeatures option

For more info on loading with flatfile-to-json.pl see the[flatfile-to-json.pl](flatfile-to-json.pl) docs.
For more info on loading with flatfile-to-json.pl see the[flatfile-to-json.pl](flatfile-to-json.pl.html) docs.



Expand Down
4 changes: 2 additions & 2 deletions docs/site/config_html_features.md
Expand Up @@ -5,7 +5,7 @@ title: HTMLFeatures

HTMLFeatures tracks display features on the genome, including things with parents and child features, using div elements. The flatfile-to-json.pl by default outputs HTMLFeatures tracks (also called FeatureTrack in the config file)

For more info on loading with flatfile-to-json.pl see the[flatfile-to-json.pl](flatfile-to-json.pl) docs.
For more info on loading with flatfile-to-json.pl see the [flatfile-to-json.pl](flatfile-to-json.pl.html) docs.

## HTMLFeatures Configuration Options

Expand All @@ -18,7 +18,7 @@ JBrowse HTMLFeatures tracks, the default legacy track type for range-based featu
|`description`|Comma-separated list of fields in which to look for the description of a feature. Case-insensitive. If set to `false` or `null`, no feature description will be shown. Defaults to 'note, description'.|
|`maxDescriptionLength`|Maximum length, in characters, for displayed feature descriptions.|
|`minSubfeatureWidth`|Minimum width, in pixels, of the *top-level* feature for JBrowse to attempt to display its subfeatures. Default 6.|
|`menuTemplate`|Optional menu configuration for right-click menus on features. Can be as large and complicated as you want. See [Customizing_Right-click_Context_Menus](#customizing-right-click-context-menus "wikilink") below. If set to null or false, disables feature right-click menus.|
|`menuTemplate`|Optional menu configuration for right-click menus on features. Can be as large and complicated as you want. See [customizing right-click context menus](mouse_configs.html#customizing-right-click-context-menus). If set to null or false, disables feature right-click menus.|
|`hooks→create`|JavaScript function that creates the parent feature HTML element and returns it. By default this is: `function(track,feature) { return document.createElement('div'); }`, which creates an HTML `div` element.|
|`hooks→modify`|JavaScript function that can be used to modify the feature HTML element in any way desired. If set, the function is called with the track object, feature object, and feature HTML element as arguments (signature: `function(track, feature, featDiv)`).|
|`style→featureScale`|Minimum zoom scale (pixels/basepair) for displaying individual features in the track. Not set by default, and overrides the `maxFeatureScreenDensity`.|
Expand Down
10 changes: 4 additions & 6 deletions docs/site/faq.md
Expand Up @@ -193,9 +193,7 @@ Note that with [Apollo](http://genomearchitect.org), you can output the
data directories to some given directory and add the directories via the
user interface.

You can also setup the "dataset selector" see [\#What is the dataset
selector](dataset_selector.md) and
[JBrowse\_Configuration\_Guide\#Dataset\_Selector](JBrowse_Configuration_Guide#Dataset_Selector "wikilink")
You can also setup the "dataset selector" see [dataset selector](dataset_selector.html) and [below](#what-is-the-dataset-selector)

## What is the dataset selector

Expand All @@ -207,7 +205,7 @@ trackList.json or tracks.conf on your data directory, and then in
jbrowse.conf, add a list of all your datasets with the dataset\_ids that
you listed in the genome's data directory.

See <http://gmod.org/wiki/JBrowse_Configuration_Guide#Dataset_Selector>
See [dataset selector](dataset_selector.html)

## How do I change the name that is displayed on my features

Expand Down Expand Up @@ -395,7 +393,7 @@ Note: when you add 1 method to menuTemplate, it overwrites 1 of the
default elements in the default right-click menu. This is sort of a bug.
If you want to "add" your method to the list, then create a "blank"
menuTemplate items for the View details, etc. See
[JBrowse\_Configuration\_Guide\#Customizing\_Right-click\_Context\_Menus](JBrowse_Configuration_Guide#Customizing_Right-click_Context_Menus "wikilink")
[customizing right click context menus](mouse_configs.html#customizing-right-click-context-menus)
for more details.

## How do I access data about my features in my callback or plugin
Expand Down Expand Up @@ -1541,7 +1539,7 @@ This is normally due to a module called mime\_magic being enabled on
your Apache server. Two options for fixing this are 1. disable
mime\_magic or 2. configuring custom file types with AddType in your
apache configuration. See
[JBrowse\_Configuration\_Guide\#Apache\_Configuration\_Note](JBrowse_Configuration_Guide#Apache_Configuration_Note "wikilink")
[this Apache configuration note](alignments.html#apache-configuration-note)
for recommended fixes.

## What is the error "invalid BGZF header" on my VCF files?
Expand Down
2 changes: 1 addition & 1 deletion docs/site/installation.md
Expand Up @@ -42,7 +42,7 @@ Download a JBrowse release from GitHub

{@inject: setup_snip}

**IMPORTANT NOTE: if you are using JBrowse plugins, you must download the source codeor use git clone of https://github.com/GMOD/jbrowse instead of the minified release**
**IMPORTANT NOTE: if you are using JBrowse plugins, you must download the source code or use git clone of https://github.com/GMOD/jbrowse instead of the minified release. Then you must run ./setup.sh or npm run build/npm run watch when you are adding or building plugin code or the jbrowse source code**


Source code:
Expand Down
11 changes: 6 additions & 5 deletions docs/site/tutorial.md
Expand Up @@ -12,7 +12,7 @@ To begin, we'll pretend as though we are setting up the genome of *Volvox mythic


mkdir data
curl http://jbrowse.org/code/latest-release/docs/tutorial/data_files/volvox.fa > data/volvox.fa
curl -L http://jbrowse.org/code/latest-release/docs/tutorial/data_files/volvox.fa > data/volvox.fa

We are going to use samtools to create a "FASTA index" using their faidx command. FASTA indexing allows even very large FASTA files to be downloaded into JBrowse "on demand" e.g. only downloading the sequence required for a certain view.

Expand Down Expand Up @@ -47,13 +47,14 @@ At this point, you should be able to open up http://localhost/jbrowse/?data=data

We will use the newly generated "gene annotation" file that was generated for Volvox mythicus

curl http://jbrowse.org/code/latest-release/docs/tutorial/data_files/volvox.gff3 > data/volvox.gff3
curl -L http://jbrowse.org/code/latest-release/docs/tutorial/data_files/volvox.gff3 > data/volvox.gff3

When we are processing GFF3 for usage in JBrowse, we can aim to use GFF3Tabix format. Tabix allows random access to genomic regions similar to Indexed FASTA. We must first sort the GFF to prepare it for tabx

sort -k1,1 -k4,4n data/volvox.gff3 > data/volvox.sorted.gff3
(grep ^"#" data/volvox.gff3; grep -v ^"#" data/volvox.gff3 | grep -v "^$" | grep "\t" | sort -k1,1 -k4,4n) > data/volvox.sorted.gff3

Then run
This command extracts the header then sorts the GFF without the header. You can also use GenomeTools with `gt gff3 -sortlines data/volvox.gff3 > data/volvox.sorted.gff3` if that is easier. Then run

bgzip data/volvox.sorted.gff3
tabix -p gff data/volvox.sorted.gff3.gz
Expand All @@ -72,7 +73,7 @@ If you have been given sequenced alignments, you can also create a Alignments tr

For volvox, we are given a file

curl http://jbrowse.org/code/latest-release/docs/tutorial/data_files/volvox-sorted.bam > data/volvox-sorted.bam
curl -L http://jbrowse.org/code/latest-release/docs/tutorial/data_files/volvox-sorted.bam > data/volvox-sorted.bam

Note that this BAM file is already sorted. If your BAM is not sorted, it must be sorted to use in JBrowse. Next index this file

Expand Down Expand Up @@ -136,6 +137,6 @@ If you have troubles, send an email to gmod-ajax@lists.sourceforge.net or create

a) If the folder was not called data, e.g. you had your files in /var/www/html/jbrowse/otherdata, then you can visit http://localhost/jbrowse/?data=otherdata (this automatically lends a way to have "multiple data directories" since you could navigate to different ?data= URL paths this way. the "dataset selector" configuration contains more details)

b) The "configuration format" is called .conf, but JSON format is also allowed. See [configuration file formats](configuration_file_formats.md) guide for details.
b) The "configuration format" is called .conf, but JSON format is also allowed. See [configuration file formats](configuration_file_formats.html) guide for details.

c) If you are doing source code updates or using plugins, you must use a git clone or the "source code" release from GitHub pages. Also the "-dev" release from older versions is supplanted by using a git clone or "source code" release
2 changes: 1 addition & 1 deletion docs/site/tutorial_classic.md
Expand Up @@ -114,7 +114,7 @@ To use them, set type = JBrowse/View/Track/Wiggle/XYPlot or type = JBrowse/View/

Sometimes things just don't go well. It seems like nothing is ever as easy as it should be. Here are several resources you can use if it turns out that this guide is not enough to get JBrowse working for you.

* [JBrowse FAQ](faq_troubleshooting.md) - The FAQ contains extended documentation on many aspects JBrowse, including troubleshooting tips.
* [JBrowse FAQ](faq.html#troubleshooting) - The FAQ contains extended documentation on many aspects JBrowse, including troubleshooting tips.
* Mailing list gmod-ajax@lists.sourceforge.net - send an email to this mailing address and/or subscribe. If it is related to just getting started with jbrowse, please attach the setup.log file that is generated by setup.sh
* GitHub at GMOD/jbrowse - This is preferably for actual JBrowse issues and feature requests but some help issues can be addressed here too :)
* Gitter channel at https://gitter.im/GMOD/jbrowse
Expand Down
1 change: 1 addition & 0 deletions package.json
Expand Up @@ -42,6 +42,7 @@
"@gmod/cram": "^1.3.0",
"@gmod/gff": "^1.1.1",
"@gmod/tabix": "^1.0.2",
"@gmod/indexedfasta": "^1.0.2",
"@gmod/twobit": "^1.1.1",
"ajv": "^6.2.1",
"ava": "^0.25.0",
Expand Down
17 changes: 17 additions & 0 deletions release-notes.md
@@ -1,10 +1,27 @@
{{$NEXT}}

## Minor improvements

* Added support for bgzipped indexed FASTA. To use, bgzip your FASTA with
`bgzip -i file.fa`, which generates file.fa.gz and file.fa.gzi and then use
`samtools faidx file.fa.gz`. If you specify the .fa.gz in the track config e.g.
`"urlTemplate": "file.fa.gz"` and have all three files in your data directory,
then they will automatically be detected (issue #1152, pull #1200, @cmdcolin)

* Allow fna and mfa file extensions for FASTA to be recognized by default in
the Open sequence dialog (issue #1205, @cmdcolin)

## Bug fixes

* Fixed a bug in which feature labels would sometimes be repeated across the view,
in the wrong locations. (@rbuels)

* Fix error where a chunk size limit error during histogram display would not be
displayed (@cmdcolin)

* Fix issue where Open sequence dialog will open up the default "data" directory
instead of a blank instance (issue #1207, @cmdcolin)

# Release 1.15.3 2018-08-29 22:34:53 UTC

## Minor improvements
Expand Down
18 changes: 16 additions & 2 deletions src/JBrowse/Browser.js
Expand Up @@ -52,6 +52,7 @@ define( [
'JBrowse/View/StandaloneDatasetList',
'JBrowse/Store/SeqFeature/UnindexedFasta',
'JBrowse/Store/SeqFeature/IndexedFasta',
'JBrowse/Store/SeqFeature/BgzipIndexedFasta',
'JBrowse/Store/SeqFeature/TwoBit',
'dijit/focus',
'../lazyload.js', // for dynamic CSS loading
Expand Down Expand Up @@ -114,6 +115,7 @@ define( [
StandaloneDatasetList,
UnindexedFasta,
IndexedFasta,
BgzipIndexedFasta,
TwoBit,
dijitFocus,
LazyLoad,
Expand Down Expand Up @@ -1291,7 +1293,17 @@ openFastaElectron: function() {
refSeqOrder: results.refSeqOrder
};

if( confs[0].store.fasta && confs[0].store.fai ) {
if( confs[0].store.bgzfa && confs[0].store.fai && confs[0].store.gzi) {
var fasta = Util.replacePath( confs[0].store.bgzfa.url );
var fai = Util.replacePath( confs[0].store.fai.url );
var gzi = Util.replacePath( confs[0].store.gzi.url );
trackList.tracks[0].storeClass= 'JBrowse/Store/SeqFeature/BgzipIndexedFasta';
trackList.tracks[0].urlTemplate = fasta;
trackList.tracks[0].faiUrlTemplate = fai;
trackList.tracks[0].gziUrlTemplate = gzi;
trackList.refSeqs = fai;
}
else if( confs[0].store.fasta && confs[0].store.fai ) {
var fasta = Util.replacePath( confs[0].store.fasta.url );
var fai = Util.replacePath( confs[0].store.fai.url );
trackList.tracks[0].storeClass= 'JBrowse/Store/SeqFeature/IndexedFasta';
Expand Down Expand Up @@ -1367,7 +1379,8 @@ openFasta: function() {
this.teardown()
var newBrowser = new this.constructor({
refSeqs: { data: refSeqs },
refSeqOrder: results.refSeqOrder
refSeqOrder: results.refSeqOrder,
dataRoot: null
})
newBrowser.afterMilestone('completely initialized', () => {
storeConf.name = 'refseqs' // important to make it the refseq store
Expand Down Expand Up @@ -1481,6 +1494,7 @@ getTrackTypes: function() {
'JBrowse/Store/SeqFeature/StaticChunked' : 'JBrowse/View/Track/Sequence',
'JBrowse/Store/SeqFeature/UnindexedFasta': 'JBrowse/View/Track/Sequence',
'JBrowse/Store/SeqFeature/IndexedFasta' : 'JBrowse/View/Track/Sequence',
'JBrowse/Store/SeqFeature/BgzipIndexedFasta' : 'JBrowse/View/Track/Sequence',
'JBrowse/Store/SeqFeature/TwoBit' : 'JBrowse/View/Track/Sequence'
},

Expand Down
2 changes: 2 additions & 0 deletions src/JBrowse/ConfigAdaptor/JB_json_v1.js
Expand Up @@ -223,6 +223,7 @@ return declare('JBrowse.ConfigAdaptor.JB_json_v1',null,
/\.(bw|bigwig)$/i.test( urlTemplate ) ? 'JBrowse/Store/SeqFeature/BigWig' :
/\.(bb|bigbed)$/i.test( urlTemplate ) ? 'JBrowse/Store/SeqFeature/BigBed' :
/\.(fa|fasta)$/i.test( urlTemplate ) ? 'JBrowse/Store/SeqFeature/IndexedFasta' :
/\.(fa|fasta)\.gz$/i.test( urlTemplate ) ? 'JBrowse/Store/SeqFeature/BgzipIndexedFasta' :
/\.2bit$/i.test( urlTemplate ) ? 'JBrowse/Store/SeqFeature/TwoBit' :
/\/Sequence$/.test(trackConfig.type) ? 'JBrowse/Store/Sequence/StaticChunked' :
null
Expand All @@ -246,6 +247,7 @@ return declare('JBrowse.ConfigAdaptor.JB_json_v1',null,
if( (storeClass == 'JBrowse/Store/Sequence/StaticChunked' ||
storeClass == 'JBrowse/Store/Sequence/IndexedFasta' ||
storeClass == 'JBrowse/Store/SeqFeature/IndexedFasta' ||
storeClass == 'JBrowse/Store/SeqFeature/BgzipIndexedFasta' ||
storeClass == 'JBrowse/Store/SeqFeature/TwoBit' ||
storeClass == 'JBrowse/Store/Sequence/TwoBit' ||
trackConfig.useAsRefSeqStore
Expand Down
86 changes: 86 additions & 0 deletions src/JBrowse/Store/SeqFeature/BgzipIndexedFasta.js
@@ -0,0 +1,86 @@
const LRU = cjsRequire('lru-cache')
const { BgzipIndexedFasta } = cjsRequire('@gmod/indexedfasta')
const { Buffer } = cjsRequire('buffer')

const fastaIndexedFilesCache = LRU(5)

const BlobFilehandleWrapper = cjsRequire('../../Model/BlobFilehandleWrapper')

define( [ 'dojo/_base/declare',
'dojo/_base/lang',
'JBrowse/Model/XHRBlob',
'JBrowse/Store/SeqFeature/IndexedFasta'
],
function(
declare,
lang,
XHRBlob,
IndexedFasta
) {

return declare( IndexedFasta,
{

/**
* Storage backend for sequences in indexed fasta files
* served as static text files.
* @constructs
*/
constructor: function(args) {
let dataBlob
if (args.bgzfa)
dataBlob = new BlobFilehandleWrapper(args.bgzfa)
else if (args.urlTemplate)
dataBlob = new BlobFilehandleWrapper(new XHRBlob(this.resolveUrl(args.urlTemplate), { expectRanges: true }))
else
dataBlob = new BlobFilehandleWrapper(new XHRBlob('data.fa', { expectRanges: true }))

let indexBlob
if (args.fai)
indexBlob = new BlobFilehandleWrapper(args.fai)
else if (args.faiUrlTemplate)
indexBlob = new BlobFilehandleWrapper(new XHRBlob(this.resolveUrl(args.faiUrlTemplate)))
else if (args.urlTemplate)
indexBlob = new BlobFilehandleWrapper(new XHRBlob(this.resolveUrl(args.urlTemplate+'.fai')))
else throw new Error('no index provided, must provide a FASTA index')


let gziBlob
if(args.gzi)
gziBlob = new BlobFilehandleWrapper(args.gzi)
else if(args.gziUrlTemplate)
gziBlob = new BlobFilehandleWrapper(new XHRBlob(this.resolveUrl(args.gziUrlTemplate)))
else if (args.urlTemplate)
gziBlob = new BlobFilehandleWrapper(new XHRBlob(this.resolveUrl(args.urlTemplate+'.gzi')))
else throw new Error('no gzi index provided, must provide a GZI index')

this.source = dataBlob.toString()

// LRU-cache the CRAM object so we don't have to re-download the
// index when we switch chromosomes
const cacheKey = `data: ${dataBlob}, index: ${indexBlob}, gzi: ${gziBlob}`
this.fasta = fastaIndexedFilesCache.get(cacheKey)
if (!this.fasta) {
this.fasta = new BgzipIndexedFasta({
fasta: dataBlob,
fai: indexBlob,
gzi: gziBlob
})

fastaIndexedFilesCache.set(cacheKey, this.fasta)
}
this.fasta.getSequenceList().then(
() => this._deferred.features.resolve({success:true}),
() => this._failAllDeferred()
)
},
saveStore: function() {
return {
urlTemplate: (this.config.file||this.config.blob).url,
faiUrlTemplate: this.config.fai.url,
gziUrlTemplate: this.config.gzi.url
};
}

});
});

0 comments on commit 021e4a3

Please sign in to comment.