Improve file handling with lazy reads #363
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Lazy reads are slightly customized to better handle named pipes. The lazy file reads are combined with an early close immediately after the instance loader gets data from a file. As a result, check-jsonschema should only keep one instance file open at a time.
The previous implementation tried to repeatedly read data for a schema input, even across multiple instances. As a result, if the schema source was ephemeral (e.g. a memfd), the subsequent reads would fail.
Create a directory of 2000 instance files, then iterate through them and verify that the number of open files never exceeds 2000 at any point during iteration.
In order for pyjson5 to be satisfied, it must be passed the real file descriptor (or something satisfying `io.IOBase`). For simplicity, pass it the underlying open() result. Also improve the file close behavior using a `finally` block.
sirosen
force-pushed
the
improve-file-handling
branch
from
71accf7
to
2196b87
Compare
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Primarily this changeset introduces lazy file reading to resolve #352.
Additionally, it introduces a better caching mechanism to the schema file
readers so that schemas read from ephemeral sources (like pipes) can properly
be applied to multiple instances.
See also: pallets/click#2645