Changed indirect access to more direct access methods to improve performance #2149
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Update managing-environments.md
* export: fix exporting extras sub-dependencies (python-poetry#1294) * Support POETRY_HOME for install (python-poetry#794) Allow the `POETRY_HOME` environment variable to be passed during installation to change the default installation directory of `~/.poetry`: ``` POETRY_HOME=/etc/poetry python get-poetry.py ``` * * check if relative filename is in excluded file list (python-poetry#1459) * * check if relative filename is in excluded file list * removed find_excluded_files() method from wheel.py * added test for excluding files in wheels * creating an own test data folder, for testing excluding files by pyproject.toml * use as_posix() to respect windows file path delimiters * Exclude nested items (python-poetry#784) (python-poetry#1464) * This PR impliments the feature request python-poetry#784. When a folder is explicit defined in `pyproject.toml` as excluded, all nested data, including subfolder, are excluded. It is no longer neccessary to use the glob `folder/**/*` * use `Path` instead of `os.path.join` to create string for globbing * try to fix linting error * create glob pattern string by concatenating and not using Path * using `os.path.isdir()`` for checking of explicit excluded name is a folder, because pathlib's `is_dir()` raises in exception under windows of name contains globing characters * Remove nested data when wildcards where used. Steps to do this are: 1. expand any wildcard used 2. if expanded path is a folder append **/* and expand again * fix linting * only glob a second time if path is dir * implement @sdispater 's suggestion for better readability * fix glob for windows? * On Windows, testing if a path with a glob is a directory will raise an OSError * pathlibs glob function doesn't return the correct case (https://bugs.python.org/issue26655). So switching back to glob.glob() * removing obsolete imports * Update dependencies * Deprecate allows-prereleases in favor of allow-prereleases for consistency * Fix tests for Python 2.7 * Fix linting * Fix linting * Fix linting * Fix typing import * Correct a couple typos in get-poetry.py (python-poetry#573) * Docs: `self:update` changed to `self update` (python-poetry#1588) * Fix GitHub actions cache issues on develop (python-poetry#1918) * Fix Github actions cache issues * Fix Github Actions cache issues (python-poetry#1928) * Add --source option to "poetry add" (python-poetry#1912) * Add --source option to 'poetry add' * Add tests for 'poetry add --source' * Merge master into develop (python-poetry#2070) * Fix Github actions cache issues (python-poetry#1908) * Fix case of `-f` flag * Make it clearer what options to pass to `--format` * fix (masonry.api): `get_requires_for_build_wheel` must return additional list of requirements for building a package, not listed in `pyproject.toml` and not dependencies for the package itself (python-poetry#1875) fix (tests): adopted tests * Lazy Keyring intialization for PasswordManager (python-poetry#1892) * Fix Github Actions cache issues (python-poetry#1928) * Avoid nested quantifiers with overlapping character space on git url parsing (python-poetry#1902 (python-poetry#1913) * fix (git): match for `\w` instead of `.` for getting user * change (vcs.git): hold pattern of the regex parts in a dictionary to be consistent over all regexs * new (vcs.git): test for `parse_url` and some fixes for the regex pattern * new (vcs.git): test for `parse_url` with string that should fail * fix (test.vcs.git): make flake8 happy * fix: correct parsing of wheel version with regex. (python-poetry#1932) The previous regexp was only taking the first integer of the version number, this presented problems when the major version number reached double digits. Poetry would determine that the version of the dependency is '1', rather than, ie: '14'. This caused failures to solve versions. * Fix errors when using the --help option (python-poetry#1910) * Fix how repository credentials are retrieved from env vars (python-poetry#1909) # Conflicts: # poetry/utils/password_manager.py * Fix downloading packages from Simplepypi (python-poetry#1851) * fix downloading packages from simplepypi * unused code removed * remove unused imports * Upgrade dependencies for the 1.0.3 release (python-poetry#1965) * Bump version to 1.0.3 (python-poetry#1966) * Fix non-compliant Git URL matching RFC 3986 § 2.3 permits more characters in a URL than were matched. This corrects that, though there may be other deficiencies. This was a regression from v1.0.2, where at least “.” was matched without error. * Update README.md "Updating Poetry" Currently the note in "Updating Poetry" is different from the one below in "Enable tab completion for Bash, Fish, or Zsh". This MR is to make them more consistent. * init: change dev dependency prompt * Fix CI issues (python-poetry#2069) Co-authored-by: brandonaut <brandon@hubermx.com> Co-authored-by: finswimmer <finswimmer77@gmail.com> Co-authored-by: Yannick PÉROUX <yannick.peroux@gmail.com> Co-authored-by: Edward George <edwardgeorge@gmail.com> Co-authored-by: Jan Škoda <skoda@jskoda.cz> Co-authored-by: Andrew Marshall <andrew@johnandrewmarshall.com> Co-authored-by: Andrew Selzer <andrewfselzer@gmail.com> Co-authored-by: Andrii Maletskyi <andrii.maletskyi@gmail.com> * pre-commit: replace isort mirror with isort upstream (python-poetry#2118) The isort pre-commit mirror has been deprecated. This change updates configuration to use the upstream package repository instead of the mirror. * Add cache list command (python-poetry#1187) * Add poetry.locations.REPOSITORY_CACHE_DIR The repository cache directory is used in multiple places in the codebase. This change ensures that the value is reused. * Add cache list command This introduces a new cache sub-command that lists all available caches. Relates-to: python-poetry#1162 Co-authored-by: Tom Milligan <tommilligan@users.noreply.github.com> Co-authored-by: David Cramer <dcramer@users.noreply.github.com> Co-authored-by: finswimmer <finswimmer77@gmail.com> Co-authored-by: Kyle Altendorf <sda@fstab.net> Co-authored-by: Justin Mayer <entroP@gmail.com> Co-authored-by: Yannick PÉROUX <yannick.peroux@gmail.com> Co-authored-by: brandonaut <brandon@hubermx.com> Co-authored-by: Edward George <edwardgeorge@gmail.com> Co-authored-by: Jan Škoda <skoda@jskoda.cz> Co-authored-by: Andrew Marshall <andrew@johnandrewmarshall.com> Co-authored-by: Andrew Selzer <andrewfselzer@gmail.com> Co-authored-by: Andrii Maletskyi <andrii.maletskyi@gmail.com> Co-authored-by: Arun Babu Neelicattu <arun.neelicattu@gmail.com>
…ge section and also in the CLI. Resolves python-poetry#1848.
Document --no-root
Configuring the template chooser to include discord community server.
If multiple distributions have `tests` as a top-level package, they'll conflict whenever both are installed. (Examples [here]((https://github.com/python-poetry/poetry/issues/1905) and [here](NixOS/nixpkgs#81482). Two common alternative strategies are: 1. not distributing tests (as [here](https://github.com/pypa/sampleproject)), or 2. placing tests in a subdirectory of the main package, rather than adjacent (as [here](http://blog.habnab.it/blog/2013/07/21/python-packages-and-you/) and [here](http://as.ynchrono.us/2007/12/filesystem-structure-of-python-project_21.html)). Each of these strategies will avoid this issue. Users may fall into this trap because of the [package documentation](https://python-poetry.org/docs/pyproject/#packages) page, which gives an example: ``` packages = [ { include = "my_package" }, { include = "tests", format = "sdist" }, ] ``` Having two top-level packages in a distribution is relatively unusual, but does have some use cases. Using `"tests"` as a top-level package name in the example is likely to lead to conflicts, however. The alternate package in the documentation example could have a unique name like `"my_other_package"`, which would reduce the likelihood of this kind of overlap.
* Add fish shell tab completion for mac * Remove macOS ref as brew should work linux as well
…poetry#1653) Co-authored-by: finswimmer77@gmail.com <GiWahnsinn07!>
When testing the behavior for git packages with a setup.py file, the Provider class would create a temporary virtual environment and execute python setup.py egg_info. Both actions are costly and take time. By mocking them, we can reduce the tests time by a factor of 4 or 5.
'An' should come before a vowel sound. Reference: https://english.stackexchange.com/a/164/198117
Keep empty in-project venv dir when recreating venv to allow for docker volumes
These small changes allow `poetry shell` to properly activate a python virtual environment when the user's shell is `tcsh`.
Link with 'www' was being redirected wrong to 'https://spdx.org//licenses/'
Adds ways to specify a package to poetry add --help section.
finswimmer
added
the
kind/enhancement
…ions Provide hints when an invalid license id is input
|
@finswimmer @abn is it possible to get this change green-lit? Some of our builds are now taking in excess of 20 minutes. A 40% improvement on that is now significant time. Especially when compared to pipenv as mentioned by @jacques- in #2094. Would be great if we could get a couple of these performance boosters in! |
|
@alextriaca sorry this one fell through the gaps, we might need to port this to I'll get back to you this week. Feel free to poke at me if I do not. |
…thor_name Add error handling for malfoemed author
Ensure path requirement is converted to posix string. Relates-to: python-poetry#2398
|
Hey @abn did you possibly make any progress with this last week? If not here's that poke you asked for 😉 |
|
@alextriaca have not forgotten :) Thanks for the poke. We might release a new For this PR; please rebase on current master. |
…dcat/poetry into performance-improvements
|
@abn rebased and ready to go. |
2 tasks
2 tasks
|
@glitchedcat hi, sorry it fell through the gaps. Are you still interested in bringing this to Poetry? |
|
Closing in favor of #5805 |
|
This pull request has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs. |
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
When resolving libraries with large dependancy chains such as the "aws_cdk" the
solvemethod becomes more and more expensive due to the recursive dependancy resolution. Most of this cost comes from indirect variable access (getattr()and@property). See the breakdown image below which showsgetattr()accounting for 30% of the runtime.The change to
poetry/packages/dependency_package.pyresults in a 30% performance improvement and the changes topoetry/packages/package.pyresult in a further 10% improvement (workings below).Method
time poetry add aws_cdk.aws_codestar(see original timings)time poetry remove aws_cdk.aws_codestar(see original timings)@propertyaccessors inpoetry/packages/dependency_package.pynamevariable inpoetry/packages/package.pyTimings
*Timings in seconds
Initial installed dependancies