juvix html has high memory usage
#2744
Comments
This was referenced Apr 19, 2024
paulcadman
added a commit
that referenced
this issue
Apr 22, 2024
The judoc examples feature is currently unused. This feature was added in #1442 Keeping support for this feature adds a cost to HTML generation. We are removing this to improve the performance of `juvix html`. To just render the HTML documentation we only require the scoper result from the pipeline. To support the examples we need the type checking result. The cost is significant in larger projects as the pipeline is run for each import. Part of #2744
paulcadman
added a commit
that referenced
this issue
Apr 25, 2024
This PR delays running of the pipeline `MCache` (renamed to `ModuleInfoCache`) to allow this cache to be shared between pipeline runs in `processRecursiveUpToTyped`. `processRecursiveUpToTyped` is used by the `juvix html` command to recursively build HTML for modules in a project. * Closes #2744 ## Performance The docs build is now much faster and takes much less memory: Before: ``` $ /usr/bin/time -lh juvix html docs/index.juvix.md 1m17.41s real 35.39s user 11.42s sys 5918703616 maximum resident set size 0 average shared memory size 0 average unshared data size 0 average unshared stack size 3665213 page reclaims 697 page faults 0 swaps 0 block input operations 0 block output operations 0 messages sent 0 messages received 0 signals received 114533 voluntary context switches 81450 involuntary context switches 595152097097 instructions retired 143688878963 cycles elapsed 19323983744 peak memory footprint ``` After: ``` $ /usr/bin/time -lh juvix html docs/index.juvix.md 8.35s real 5.76s user 0.62s sys 2992160768 maximum resident set size 0 average shared memory size 0 average unshared data size 0 average unshared stack size 221870 page reclaims 719 page faults 0 swaps 0 block input operations 0 block output operations 0 messages sent 0 messages received 0 signals received 1965 voluntary context switches 1962 involuntary context switches 93909891240 instructions retired 19317129226 cycles elapsed 2963053632 peak memory footprint ``` ## Notes * `MCache` is renamed to `ModuleInfoCache` * `ModuleInfoCache` must be defined in a separate module instead of being defined in `Compiler.Pipeline.Driver` to avoid a cyclic dependency. `ModuleInfoCache` is now used used in `Juvix.Compiler.Pipeline` (in `PipelineEff`) and this module is imported by `Juvix.Compiler.Pipeline.Driver`).
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
juvix htmlconsumes too much memory. This means that the docs build is slow and/or gets OOM killed:https://github.com/anoma/juvix-docs/actions/runs/8742259831/job/23990093219
To reproduce run the following in https://github.com/anoma/juvix-docs:
Maximum resident set size is about 6 GB, peak memory footprint is about 19 GB.
The text was updated successfully, but these errors were encountered: