New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Improve memoization of interpreter constraints, Python parsing, and request classes #16141
Conversation
…ging for a single input. # Rust tests and lints will be skipped. Delete if not intended. [ci skip-rust] # Building wheels and fs_util will be skipped. Delete if not intended. [ci skip-build-wheels]
# Rust tests and lints will be skipped. Delete if not intended. [ci skip-rust] # Building wheels and fs_util will be skipped. Delete if not intended. [ci skip-build-wheels]
# Rust tests and lints will be skipped. Delete if not intended. [ci skip-rust] # Building wheels and fs_util will be skipped. Delete if not intended. [ci skip-build-wheels]
Commits are useful to review independently. |
Is this specifically meant to address #16122? As in, we don't need to investigate the code paths in that issue more? |
As commented there, I wasn't able to reproduce a difference on the reported scale, so I'm going to leave that one open until we can get a repro. |
…equest classes (pantsbuild#16141) Profiling showed that: 1. `Requirement.parse` and `InterpreterConstraint.merge_*` were taking up a significant fraction of time even with singular requirements (~2%). 2. `AllAssetTargetsRequest` did not have stable equality due to not being marked as a `@dataclass`, and so `find_all_assets` was not being memoized (~11%). 3. Repeatedly loading and creating a `Digest` for the `dependency_parser.py` script took a noticeable amount of time (~6%). Fixing these represents a ~19% performance improvement for `./pants --no-pantsd --changed-diffspec=fcaac98402..2a30000 --changed-dependees=transitive list`, which brings `2.12.x`/`2.13.x` roughly back in line with `2.11.x`. [ci skip-rust] [ci skip-build-wheels]
…equest classes (pantsbuild#16141) Profiling showed that: 1. `Requirement.parse` and `InterpreterConstraint.merge_*` were taking up a significant fraction of time even with singular requirements (~2%). 2. `AllAssetTargetsRequest` did not have stable equality due to not being marked as a `@dataclass`, and so `find_all_assets` was not being memoized (~11%). 3. Repeatedly loading and creating a `Digest` for the `dependency_parser.py` script took a noticeable amount of time (~6%). Fixing these represents a ~19% performance improvement for `./pants --no-pantsd --changed-diffspec=fcaac98402..2a30000 --changed-dependees=transitive list`, which brings `2.12.x`/`2.13.x` roughly back in line with `2.11.x`. # Rust tests and lints will be skipped. Delete if not intended. [ci skip-rust] # Building wheels and fs_util will be skipped. Delete if not intended. [ci skip-build-wheels]
…equest classes (Cherry-pick of #16141) (#16153) Profiling showed that: 1. `Requirement.parse` and `InterpreterConstraint.merge_*` were taking up a significant fraction of time even with singular requirements (~2%). 2. `AllAssetTargetsRequest` did not have stable equality due to not being marked as a `@dataclass`, and so `find_all_assets` was not being memoized (~11%). 3. Repeatedly loading and creating a `Digest` for the `dependency_parser.py` script took a noticeable amount of time (~6%). Fixing these represents a ~19% performance improvement for `./pants --no-pantsd --changed-diffspec=fcaac98402..2a30000 --changed-dependees=transitive list`, which brings `2.12.x`/`2.13.x` roughly back in line with `2.11.x`. [ci skip-rust] [ci skip-build-wheels]
…equest classes (Cherry-pick of #16141) (#16154) Profiling showed that: 1. `Requirement.parse` and `InterpreterConstraint.merge_*` were taking up a significant fraction of time even with singular requirements (~2%). 2. `AllAssetTargetsRequest` did not have stable equality due to not being marked as a `@dataclass`, and so `find_all_assets` was not being memoized (~11%). 3. Repeatedly loading and creating a `Digest` for the `dependency_parser.py` script took a noticeable amount of time (~6%). Fixing these represents a ~19% performance improvement for `./pants --no-pantsd --changed-diffspec=fcaac98402..2a30000 --changed-dependees=transitive list`, which brings `2.12.x`/`2.13.x` roughly back in line with `2.11.x`. [ci skip-rust] [ci skip-build-wheels]
Profiling showed that:
Requirement.parse
andInterpreterConstraint.merge_*
were taking up a significant fraction of time even with singular requirements (~2%).AllAssetTargetsRequest
did not have stable equality due to not being marked as a@dataclass
, and sofind_all_assets
was not being memoized (~11%).Digest
for thedependency_parser.py
script took a noticeable amount of time (~6%).Fixing these represents a ~19% performance improvement for
./pants --no-pantsd --changed-diffspec=fcaac98402..2a300002f0 --changed-dependees=transitive list
, which brings2.12.x
/2.13.x
roughly back in line with2.11.x
.