Improve performance and clean up the preprocessor #353
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
Testing on larger inputs seem to show that the preprocessor is handling about 14MB/s. (For comparison, the largest |
|
Closes #342 |
Adopt a systematic and uniform format for the code of the preprocessor Document that given style Add a dune rule so that `dune fmt` reformats the preprocessor accordingly (when `ex` is available)
Use the name `print_directive`, to be consistent with the rest of the printing functions
OCaml needs directives to recover the real positions of any preprocessed source, so preprocessors must output a directive at the very beginning of their outputs
Change the interface of the preprocessor so that it outputs its result into a formatter rather than a string Propagate the change all the way down, so that: - the whole output is generated in a single buffer when the target is a `string`, or directly on `stdout` in the case of `gospel pps`, - `print` becomes tail recursive.
Ensure that `collapse_spaces` is tail recursive so that it does not overflow the stack when running on large inputs That requires an extra list reversal, unfortunately
|
Updated so that all |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This PR reworks the preprocessor by:
In a bit more details, the performance gains are obtained by changing the interface of the preprocessor so that it outputs its result into a formatter rather than a string and propagating the change all the way down, so that:
string, or directly onstdoutin the case ofgospel pps,printbecomes tail recursive.Using the example mentioned in #336 (test run in CI, so the performance may vary depending on load...), we go from:
to: