Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Parallelize the pipeline #2749

Closed
janmasrovira opened this issue Apr 22, 2024 · 1 comment
Closed

Parallelize the pipeline #2749

janmasrovira opened this issue Apr 22, 2024 · 1 comment
Assignees
Labels
enhancement New feature or request performance

Comments

@janmasrovira
Copy link
Collaborator

As part of the effort to improve the efficiency of the compiler and improve tooling responsiveness (#2685), we should parallelize the pipeline.

The idea is to build an import tree where the vertices are juvix (top) modules and the edges represent import dependencies.
Then, we create a compile queue from which worker threads pull pending modules and compile them in parallel.
A toy experiment that simulates this strategy has been implemented successfully in https://github.com/anoma/juvix/blob/parallel/app/Parallel/Experiment.hs

How much of the pipeline will be parallelized is yet to be decided. My initial goal is to parallelize (at least) the deserialization of the .jvo files. After that is done, we can extend it further.

Naturally, the efficiency gains from this will depend on the relation between the height and the size of the import tree. My rought expectation is that it will be at least 3x faster on average.

@janmasrovira
Copy link
Collaborator Author

Closed by #2779

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request performance
Projects
None yet
Development

No branches or pull requests

1 participant