Parallelize RBI parsing to speed up the check-shims command
#1056
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Signed-off-by: Alexandre Terrasa <alexandre.terrasa@shopify.com>
Morriar
force-pushed
the
at-check-shims-parallel
branch
from
55e7f39
to
0138278
Compare
ChrisBr
reviewed
Jul 12, 2022
| index_rbis(index, "shim", @shim_rbi_dir, number_of_workers: @number_of_workers) | ||
| index_rbis(index, "gem", @gem_rbi_dir, number_of_workers: @number_of_workers) | ||
| index_rbis(index, "dsl", @dsl_rbi_dir, number_of_workers: @number_of_workers) | ||
| index_rbis(index, "annotation", @annotations_rbi_dir, number_of_workers: @number_of_workers) | ||
|
|
||
| duplicates = duplicated_nodes_from_index(index, shim_rbi_dir: @shim_rbi_dir, todo_rbi_file: @todo_rbi_file) |
There was a problem hiding this comment.
Just wondering for a future improvement, could the duplicates for each file be stored in a cache similar to what RuboCop does?
There was a problem hiding this comment.
Maybe.
A fast and elegant way would be to ask Sorbet through a custom LSP request.
vinistock
reviewed
Jul 12, 2022
| @@ -250,5 +248,14 @@ def update_gem_rbis_strictnesses(errors, gem_dir) | |||
| def gem_name_from_rbi_path(path) | |||
| T.must(File.basename(path, ".rbi").split("@").first) | |||
| end | |||
|
|
|||
| sig { params(block: T.proc.void).returns(Numeric) } | |||
| def with_timer(&block) | |||
There was a problem hiding this comment.
Could we use Benchmark instead? I think it does exactly the same. Also, do we actually need to measure this?
There was a problem hiding this comment.
Could we use Benchmark instead?
Good idea!
Also, do we actually need to measure this?
I was actually thinking to extend the concept to other commands so we can easily see where Tapioca is slow or slower.
Morriar
force-pushed
the
at-check-shims-parallel
branch
from
0138278
to
9cda925
Compare
vinistock
approved these changes
Jul 12, 2022
Signed-off-by: Alexandre Terrasa <alexandre.terrasa@shopify.com>
Signed-off-by: Alexandre Terrasa <alexandre.terrasa@shopify.com>
Morriar
force-pushed
the
at-check-shims-parallel
branch
from
9cda925
to
1f349b0
Compare
KaanOzkan
approved these changes
Jul 14, 2022
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Motivation
Running
tapioca check-shimscan take up to 7min in Core (~13 when running on CI). The culprit is parsing the RBI files.This PR proposes to speed up the process by parallelizing RBI parsing.
Implementation
I reused
Executorthat we already use to parallelize RBI generation.Tests
No behavior change.
I tested the speed-up on Tapioca itself:
Before:
After:
So a > 50% improvement here.
I also tested on Core:
Before:
After:
So a > 50% improvement here too.