Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Lazy-load assemblies on demand instead of all at app startup #3287

Merged
merged 17 commits into from
Oct 26, 2022

Conversation

cmdcolin
Copy link
Collaborator

@cmdcolin cmdcolin commented Oct 24, 2022

The current code for jbrowse-web/jbrowse-desktop loads ALL configured assemblies and their refnames on startup e.g. fetches their "fai" file or similar on startup. While these fai files are not often too large (unless assembly is fragmented, then can be megabytes, or if it is twobit, which has very poor access pattern with a range request for each refname in the file), but even for small ones, it is a bit of a network stampede. A user requested loading 1000 assemblies in gitter. This PR helps by lazily loading assemblies as the data is requested, either via assemblyManager.get('asm') or assemblyManager.waitForAssembly('asm')

The get() is a little funky since it is a "view" that kicks off an async action to start the loading even though it returns synchronously, but code that uses assemblyManager.get should know about this type of reactivity

One change for this is that LGV.navToLocString is now async. This is needed to handle locstrings that contain an assembly name which we test for, e.g. {volvox2}:ctgA:1-100

Some tests are updated accordingly to not immediately check the result of navToLocString

Fix #1735

@github-actions github-actions bot added the needs label triage Needs a label to show in changelog (breaking, enhancement, bug, documentation, or internal) label Oct 24, 2022
@cmdcolin cmdcolin added enhancement New feature or request and removed needs label triage Needs a label to show in changelog (breaking, enhancement, bug, documentation, or internal) labels Oct 24, 2022
@codecov
Copy link

codecov bot commented Oct 25, 2022

Codecov Report

Merging #3287 (2e3bc07) into main (fd08109) will increase coverage by 0.29%.
The diff coverage is 80.92%.

@@            Coverage Diff             @@
##             main    #3287      +/-   ##
==========================================
+ Coverage   59.40%   59.70%   +0.29%     
==========================================
  Files         679      675       -4     
  Lines       28868    28735     -133     
  Branches     7047     6981      -66     
==========================================
+ Hits        17150    17157       +7     
+ Misses      11441    11334     -107     
+ Partials      277      244      -33     
Impacted Files Coverage Δ
packages/core/rpc/WebWorkerRpcDriver.ts 0.00% <0.00%> (ø)
products/jbrowse-img/src/renderRegion.js 0.00% <0.00%> (ø)
...se-react-linear-genome-view/src/createViewState.ts 54.16% <10.00%> (+2.31%) ⬆️
packages/core/assemblyManager/assemblyManager.ts 72.30% <66.66%> (+0.51%) ⬆️
.../linear-genome-view/src/LinearGenomeView/index.tsx 83.42% <70.83%> (+0.25%) ⬆️
...-react-circular-genome-view/src/createViewState.ts 86.36% <87.50%> (-4.12%) ⬇️
packages/core/rpc/coreRpcMethods.ts 83.33% <95.23%> (+2.88%) ⬆️
packages/core/assemblyManager/assembly.ts 90.50% <97.87%> (+0.37%) ⬆️
packages/core/rpc/MainThreadRpcDriver.ts 66.66% <100.00%> (ø)
plugins/sequence/src/IndexedFastaAdapter/index.ts 75.00% <0.00%> (-25.00%) ⬇️
... and 19 more

📣 We’re building smart automated test selection to slash your CI/CD build times. Learn more

@cmdcolin cmdcolin force-pushed the lazy_assembly branch 2 times, most recently from c1c0d67 to c540ad3 Compare October 25, 2022 18:05
@cmdcolin cmdcolin force-pushed the lazy_assembly branch 5 times, most recently from 07d648f to 5a888b0 Compare October 26, 2022 02:09
@cmdcolin
Copy link
Collaborator Author

managed to pass all tests now, should be ready to go :)

@cmdcolin cmdcolin merged commit 821c85f into main Oct 26, 2022
@cmdcolin cmdcolin deleted the lazy_assembly branch October 26, 2022 18:29
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Test having 1000 assemblies
1 participant