perf: cached os.pathconf() call in _record_path() #1920
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
roshnipatel
changed the title
|
Kudos, SonarCloud Quality Gate passed!
|
johanneskoester
pushed a commit
that referenced
this pull request
Oct 27, 2022
🤖 I have created a release *beep* *boop* --- ## [7.17.0](v7.16.2...v7.17.0) (2022-10-27) ### Features * allow to define the cache mode per rule (this enables to exclude software envs from the caching hash value, which can be handy e.g. for download rules where the software version does not affect the result) ([#1933](#1933)) ([715e618](715e618)) ### Performance Improvements * cached os.pathconf() call in _record_path() ([#1920](#1920)) ([551badb](551badb)) --- This PR was generated with [Release Please](https://github.com/googleapis/release-please). See [documentation](https://github.com/googleapis/release-please#release-please). Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
jakevc
pushed a commit
to jakevc/snakemake
that referenced
this pull request
Nov 3, 2022
### Description The value of `os.pathconf(self._incomplete_path, "PC_NAME_MAX")` is cached in an attribute `Persistence._max_len` -- this avoids repeatedly querying the operating system for the same configuration information every time `marked_incomplete()` is called on a file/job output. (Note: this will dramatically speed up Snakemake DAG building on compute clusters where `os.pathconf()` is slow to return and the number of files checked for incompleteness is very large.) ### QC <!-- Make sure that you can tick the boxes below. --> * [X] The PR contains a test case for the changes or the changes are already covered by an existing test case. * [X] The documentation (`docs/`) is updated to reflect the changes or this is not necessary (e.g. if the change does neither modify the language nor the behavior or functionalities of Snakemake).
jakevc
pushed a commit
to jakevc/snakemake
that referenced
this pull request
Nov 3, 2022
🤖 I have created a release *beep* *boop* --- ## [7.17.0](snakemake/snakemake@v7.16.2...v7.17.0) (2022-10-27) ### Features * allow to define the cache mode per rule (this enables to exclude software envs from the caching hash value, which can be handy e.g. for download rules where the software version does not affect the result) ([snakemake#1933](snakemake#1933)) ([715e618](snakemake@715e618)) ### Performance Improvements * cached os.pathconf() call in _record_path() ([snakemake#1920](snakemake#1920)) ([551badb](snakemake@551badb)) --- This PR was generated with [Release Please](https://github.com/googleapis/release-please). See [documentation](https://github.com/googleapis/release-please#release-please). Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description
The value of
os.pathconf(self._incomplete_path, "PC_NAME_MAX")is cached in an attributePersistence._max_len-- this avoids repeatedly querying the operating system for the same configuration information every timemarked_incomplete()is called on a file/job output. (Note: this will dramatically speed up Snakemake DAG building on compute clusters whereos.pathconf()is slow to return and the number of files checked for incompleteness is very large.)QC
docs/) is updated to reflect the changes or this is not necessary (e.g. if the change does neither modify the language nor the behavior or functionalities of Snakemake).