New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Explore caching index for gems #1009
Comments
Could a Bundler plugin be an alternative? There's an |
We would depend on every gem maintainer to use the bundler plugin, which wouldn't really scale. |
It could be distributed as a gem which Ruby LSP installs in the |
Would bundler pick up the plugin automatically? Even from a different Gemfile? |
Overview of ProgressI'm wrapping up my work on this issue, so I'm leaving some context here for whoever decides to take it on. There were 2 main components to this task:
The progress for both tasks is available on the Serializing EntriesWe opted to use custom JSON to serialize entries instead of something like Marshal, for performance reasons. When using JSON vs. Marshal, we found a 15x improvement in serialization and deserialization time.
Next StepsThe serialization logic is ready to ship. Implementing CachingThe relevant code here is in In Next Steps
TL;DRProgress has been made on serializing entries and implementing caching for gems, with the latter showing a 43% reduction in indexing time. The serialization logic is ready to ship, and the remaining tasks involve addressing potential edge cases when caching, integrating caching into the indexing workflow, and adding tests for the caching. Code is available on the |
Caching the index for gems is easier than caching the entire index because we know the files will only change if the version of the gem has changed.
I think we can get a considerable performance improvement on indexing if we cache the entries for gems. Something like
Where each gem cache is basically a Marshal dump of the index. This may require defining a way to merge different indices.
For the long term, it would be amazing if rubygems could generate the index cache during packaging, so that all gems are exported with an index by default. By doing the work ahead of time, we would be guaranteed to always have cached indices for all gems, significantly speeding up indexing.
Tasks
RubyIndexer::Entry
objects #1919The text was updated successfully, but these errors were encountered: