New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Use @gmod/tabix-js for tabix access #1209

Merged
merged 29 commits into from Oct 2, 2018

Conversation

Projects
None yet
3 participants
@rbuels
Collaborator

rbuels commented Sep 19, 2018

Uses the new @gmod/tabix npm module for all tabix code.

Improves performance, and fixes the issues that have been seen with large VCF headers (@cmdcolin's large VCF header test has been enabled now and is passing)

fixes #1147, fixes #1139, fixes #1195

rbuels and others added some commits Sep 4, 2018

Merge pull request #1206 from GMOD/1147_npm_tabix_fixtests
Fixed VCF with/without dummy tests

@wafflebot wafflebot bot added the in progress label Sep 19, 2018

@rbuels rbuels added this to the 1.15.4 milestone Sep 19, 2018

@cmdcolin

This comment has been minimized.

Show comment
Hide comment
@cmdcolin

cmdcolin Sep 19, 2018

Contributor

Has this been tested on the data dir in #1195? I think there are some glitches that show up in gene models after applying these changes

screenshot-twitter com-2018 09 19-06-20-52
before

screenshot-localhost-2018 09 19-06-35-15
after

Contributor

cmdcolin commented Sep 19, 2018

Has this been tested on the data dir in #1195? I think there are some glitches that show up in gene models after applying these changes

screenshot-twitter com-2018 09 19-06-20-52
before

screenshot-localhost-2018 09 19-06-35-15
after

rbuels added some commits Sep 24, 2018

add support for `topLevelFeaturesPercent` config variable that can be…
… set to tweak stat estimation when topLevelFeatures and feature aggregation make the stats off by a lot
@rbuels

This comment has been minimized.

Show comment
Hide comment
@rbuels

rbuels Sep 24, 2018

Collaborator

i ended up needing low-level control of unzipping in tabix-js, to stop unzipping a set of data when we come to the end of the chunk. if we unzip block by block and keep track of the block lengths during unzipping, we can unzip exactly the right amount

this actually led to a large speedup in the UI as well, which I was not expecting

@cmdcolin i think this fixed the gff3 bug you were seeing, could you look again?

Collaborator

rbuels commented Sep 24, 2018

i ended up needing low-level control of unzipping in tabix-js, to stop unzipping a set of data when we come to the end of the chunk. if we unzip block by block and keep track of the block lengths during unzipping, we can unzip exactly the right amount

this actually led to a large speedup in the UI as well, which I was not expecting

@cmdcolin i think this fixed the gff3 bug you were seeing, could you look again?

@cmdcolin

This comment has been minimized.

Show comment
Hide comment
@cmdcolin

cmdcolin Sep 24, 2018

Contributor

Looks like it's pretty good. The feature labels aren't being displayed on the genes though?

Contributor

cmdcolin commented Sep 24, 2018

Looks like it's pretty good. The feature labels aren't being displayed on the genes though?

@cmdcolin

This comment has been minimized.

Show comment
Hide comment
@cmdcolin

cmdcolin Sep 24, 2018

Contributor

Maybe that's because it's determining it's too dense to display the feature labels?

Contributor

cmdcolin commented Sep 24, 2018

Maybe that's because it's determining it's too dense to display the feature labels?

@cmdcolin

This comment has been minimized.

Show comment
Hide comment
@cmdcolin

cmdcolin Sep 24, 2018

Contributor

Also sometimes the mouseover shadow does not appear on the feature you are hovering over? I'm just scrolling around that ncbi human data

Contributor

cmdcolin commented Sep 24, 2018

Also sometimes the mouseover shadow does not appear on the feature you are hovering over? I'm just scrolling around that ncbi human data

@cmdcolin

This comment has been minimized.

Show comment
Hide comment
@cmdcolin

cmdcolin Sep 24, 2018

Contributor

At this zoom level NCF2 is not mouseoverable but zooming in it seems able to be mouseovered just fine NC_000001.11:183538351..183601700 (63.35 Kb)

Contributor

cmdcolin commented Sep 24, 2018

At this zoom level NCF2 is not mouseoverable but zooming in it seems able to be mouseovered just fine NC_000001.11:183538351..183601700 (63.35 Kb)

@rbuels

This comment has been minimized.

Show comment
Hide comment
@rbuels

rbuels Sep 24, 2018

Collaborator

@cmdcolin set topLevelFeaturesPercent = 10 on the track conf and see if that makes it behave more as you expect. added that config switch so people can fix the indexedstats line count-based stats. it counts a whole bunch of lines, but they pretty much collapse into only about 10% as many top-level features

Collaborator

rbuels commented Sep 24, 2018

@cmdcolin set topLevelFeaturesPercent = 10 on the track conf and see if that makes it behave more as you expect. added that config switch so people can fix the indexedstats line count-based stats. it counts a whole bunch of lines, but they pretty much collapse into only about 10% as many top-level features

@cmdcolin

This comment has been minimized.

Show comment
Hide comment
@cmdcolin

cmdcolin Sep 25, 2018

Contributor

This is looking very good. The topLevelFeaturesPercent seems like a reasonable correction factor for stats estimation. The mouseover issue I mentioned above could have been a fluke, I think things are looking good now!

Contributor

cmdcolin commented Sep 25, 2018

This is looking very good. The topLevelFeaturesPercent seems like a reasonable correction factor for stats estimation. The mouseover issue I mentioned above could have been a fluke, I think things are looking good now!

@rbuels rbuels merged commit 589f901 into dev Oct 2, 2018

@wafflebot wafflebot bot removed the in progress label Oct 2, 2018

@cmdcolin cmdcolin deleted the 1147_npm_tabix branch Oct 4, 2018

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment