Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

New and faster installer implementation #2595

Merged
merged 12 commits into from Jul 10, 2020
Merged

New and faster installer implementation #2595

merged 12 commits into from Jul 10, 2020

Conversation

sdispater
Copy link
Member

@sdispater sdispater commented Jun 26, 2020

Pull Request Check List

  • Added tests for changed code.
  • Updated documentation for changed code.

This PR introduces a new installation experience with a brand new installer that supports parallel operations. This was made possible by splitting the different parts of the installation process into several new classes:

  • The Executor class is now the main point of entry and will be used by the existing Installer class if opted in via the use_executor method (note that the new installer will be opt-out by default since I want to gather as much feedback as possible). The Executor is responsible for executing a batch of operations (install, uninstall , update), in parallel by default. Note that not all operations will be executed in parallel but only those with the same depth in the dependency tree.
  • The Chooser class is responsible for picking the appropriate distributions (sdist or wheel) for a given package and environment.
  • The Chef class does not do much at the moment, it only manages the distributions cache, but will eventually be responsible for building wheels before package installation, this will be implemented in the 1.2 release.
  • The Authenticator class is reponsible for handling credentials when downloading distributions from custom indices. Most of the ideas have been taken from the cache system of pip.

This is the first step towards the removal of calls to pip for installation of packages. We now use it exclusively to install distributions available locally.

The installer can be opted out via the experimental.new-installer setting:

poetry config experimental.new-installer false

Here is an example of the new installer output:

Screen Recording 2020-05-29 at 17 39 38

@sdispater sdispater added area/installer Related to the dependency installer kind/feature Feature requests/implementations labels Jun 26, 2020
@sdispater sdispater added this to the 1.1 milestone Jun 26, 2020
@sdispater sdispater requested a review from a team Jun 26, 2020
@sdispater sdispater added this to In progress in 1.1 via automation Jun 26, 2020
Copy link
Member

@abn abn left a comment

A few recommendations. Haven't gone through the test cases yet and have largely skimmed the UI part of the executor.

poetry/console/commands/mixins/with_installer.py Outdated Show resolved Hide resolved
installer = Installer(
self.io, self.env, self.poetry.package, self.poetry.locker, self.poetry.pool
self._installer.use_executor(
self.poetry.config.get("experimental.new-installer", False)
Copy link
Member

@abn abn Jun 28, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should we consider moving this into a configuration constants section?

poetry/console/config/application_config.py Outdated Show resolved Hide resolved
poetry/installation/authenticator.py Show resolved Hide resolved
poetry/installation/authenticator.py Show resolved Hide resolved
poetry/utils/_compat.py Outdated Show resolved Hide resolved
.github/workflows/main.yml Show resolved Hide resolved
poetry/installation/chooser.py Outdated Show resolved Hide resolved
poetry/installation/chooser.py Show resolved Hide resolved
poetry/installation/executor.py Outdated Show resolved Hide resolved
1.1 automation moved this from In progress to Review in progress Jun 28, 2020
Copy link
Contributor

@pradyunsg pradyunsg left a comment

Some, not 100% related to this PR, comments. Lemme know if this is less than useful. :)

poetry/installation/chooser.py Show resolved Hide resolved
poetry/installation/executor.py Outdated Show resolved Hide resolved
@davidszotten
Copy link
Contributor

davidszotten commented Jul 1, 2020

hi. excited to try this out!

when i tried it i get a KeyError: interpreter_name in poetry/installation/chef.py:97 (key_parts["interpreter_name"] = self._env.marker_env["interpreter_name"])

i wonder if it's because we have virtualenvs.create false and self._env == SystemEnv("/usr/local")

@sdispater
Copy link
Member Author

sdispater commented Jul 3, 2020

@davidszotten Thanks for reporting this!

There were actually missing elements for SystemEnv. It should be fixed now.

poetry/installation/executor.py Outdated Show resolved Hide resolved
for task in as_completed(tasks):
task.result()
except KeyboardInterrupt:
self._shutdown = True
Copy link
Contributor

@davidszotten davidszotten Jul 9, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

could we not just (cancel tasks and) shutdown the executor and just return here? if i'm reading correctly nothing else happens if shutdown becomes True anyway

Copy link
Member Author

@sdispater sdispater Jul 9, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It's just to have the same code path whether we have a KeyboardInterrupt or an actual error, since an error will also set _shutdown.

Copy link
Contributor

@davidszotten davidszotten Jul 9, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ah, missed that the tasks can also set _shutdown. thanks replied to the wrong comment

@davidszotten
Copy link
Contributor

davidszotten commented Jul 9, 2020

hm, something else has gone wrong in 35225a7. before, the status message (Pending, Downloading, Installing) would disappear after each package finished installing. Now it remains next to each package (so i can no longer tell what is still installing from what is done)

@sdispater
Copy link
Member Author

sdispater commented Jul 9, 2020

@davidszotten Yes, I noticed that as well but it should be fixed now.

@@ -304,6 +304,13 @@ def package(self, name, version, extras=None): # type: (...) -> Package

return package

def find_links_for_package(self, package):
Copy link
Contributor

@MichaelAquilina MichaelAquilina Jul 9, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

minor, but you missed the type annotation here :) have you considered turning on disallow_untyped_defs?

@sdispater
Copy link
Member Author

sdispater commented Jul 10, 2020

Thanks to everyone who reviewed this and especially @davidszotten for testing it thoroughly.

I am pretty confident that this is in a stable enough state to be released in the next alpha release of the 1.1.0 version so I'll go ahead and merge it.

If further bugs and issues are discovered we will address them in the next alpha (which should be the last) and beta releases.

@sdispater sdispater merged commit c7055be into develop Jul 10, 2020
1.1 automation moved this from Review in progress to Done Jul 10, 2020
@sdispater sdispater deleted the new-installer branch Jul 10, 2020
@davidszotten
Copy link
Contributor

davidszotten commented Jul 10, 2020

🎉
thanks so much for your work on poetry!

@danihodovic
Copy link

danihodovic commented Sep 26, 2020

@sdispater is it possible to track what makes an installation slower?

I'd like to debug the below install time to see if any package is causing problems.

$ poetry add watchdog@latest -D
Using version ^0.10.3 for watchdog
Updating dependencies
Resolving dependencies... (959.6s)

@abn
Copy link
Member

abn commented Sep 26, 2020

@danihodovic I am unable to reproduce that in a new project.

$ poetry@1.1.0rc1 add watchdog 
Using version ^0.10.3 for watchdog

Updating dependencies
Resolving dependencies... (0.1s)

Writing lock file

Package operations: 2 installs, 0 updates, 0 removals

  • Installing pathtools (0.1.2)
  • Installing watchdog (0.10.3)

Just to make sure we are not missing anything, what version of poetry are you using? If you are using 1.0.10 and the lock file was generated by a 1.1 release, resolving will not complete. Alternatively, it could be a reslult of some other dependencies in your pyproject.toml file. If the issue persists, please raise an issue, this PR is closed.

@danihodovic
Copy link

danihodovic commented Sep 27, 2020

I'm not interested in opening a new bug on the slow dependency resolution as there are plenty open already. I want to know how to debug it when the dependency resolution takes an unreasonable amount of time.

@abn
Copy link
Member

abn commented Sep 27, 2020

Use poetry add -D -vvv watchdog@latest.

@Vozf
Copy link

Vozf commented Oct 6, 2020

I've got some problems in gitlab ci using new poetry parallel installation.
the poetry install fails with

  • Installing holoviews (1.13.4)
  • Installing comet-ml (3.2.3)
Retrying HTTP request in 0.5 seconds.
Retrying HTTP request in 0.5 seconds.

  ConnectionError

  ('Connection aborted.', ConnectionResetError(104, 'Connection reset by peer'))

  at ~/.poetry/lib/poetry/_vendor/py3.8/requests/adapters.py:498 in send
      494│                     low_conn.close()
      495│                     raise
      496│ 
      497│         except (ProtocolError, socket.error) as err:
    → 498│             raise ConnectionError(err, request=request)
      499│ 
      500│         except MaxRetryError as e:
      501│             if isinstance(e.reason, ConnectTimeoutError):
      502│                 # TODO: Remove this in 3.0.0: see #2811


  ConnectionError

  ('Connection aborted.', ConnectionResetError(104, 'Connection reset by peer'))

  at ~/.poetry/lib/poetry/_vendor/py3.8/requests/adapters.py:498 in send
      494│                     low_conn.close()
      495│                     raise
      496│ 
      497│         except (ProtocolError, socket.error) as err:
    → 498│             raise ConnectionError(err, request=request)
      499│ 
      500│         except MaxRetryError as e:
      501│             if isinstance(e.reason, ConnectTimeoutError):
      502│                 # TODO: Remove this in 3.0.0: see #2811

I don't even know which package causes this. It is likely not related to comet_ml as it is different package everytime. but it can be fixed with poetry config experimental.new-installer false

@earshinov
Copy link

earshinov commented Oct 7, 2020

We also face occasional errors when using new version of Poetry:

Skipping virtualenv creation, as specified in config file.
Installing dependencies from lock file
Package operations: 119 installs, 13 updates, 0 removals
  • Installing colorama (0.4.1)
  • Installing markupsafe (1.1.1)
  • Updating six (1.11.0 /usr/lib/python3/dist-packages -> 1.11.0)
  EnvCommandError
  Command ['/usr/bin/python3.8', '-m', 'pip', 'install', '--no-deps', '/root/.cache/pypoetry/artifacts/cf/46/78/4a92e28748cc243c738fb10b6dcaae48ea8ddc2684dfff839e0ce9f6c4/MarkupSafe-1.1.1-cp38-cp38-manylinux1_x86_64.whl'] errored with the following return code 1, and output: 
  Processing /root/.cache/pypoetry/artifacts/cf/46/78/4a92e28748cc243c738fb10b6dcaae48ea8ddc2684dfff839e0ce9f6c4/MarkupSafe-1.1.1-cp38-cp38-manylinux1_x86_64.whl
  Installing collected packages: MarkupSafe
  ERROR: Could not install packages due to an EnvironmentError: [Errno 2] No such file or directory: '/usr/local/lib/python3.8/dist-packages/~ix-1.15.0.dist-info'
  
  
  at /usr/local/lib/python3.8/dist-packages/poetry/utils/env.py:948 in _run
       944│                 output = subprocess.check_output(
       945│                     cmd, stderr=subprocess.STDOUT, **kwargs
       946│                 )
       947│         except CalledProcessError as e:
    →  948│             raise EnvCommandError(e, input=input_)
       949│ 
       950│         return decode(output)
       951│ 
       952│     def execute(self, bin, *args, **kwargs):

For now we decided to run poetry install in CI in a loop like this:

RUN poetry_succeeded=;                 \
    for i in $(seq 3);                 \
    do                                 \
        if poetry install --no-root;   \
        then                           \
            poetry_succeeded=true;     \
            break;                     \
        fi;                            \
    done;                              \
    if [ -z "$poetry_succeeded" ];     \
    then                               \
        exit 1;                        \
    fi;                                \

@abn
Copy link
Member

abn commented Oct 7, 2020

@earshinov you can disable the new installer using poetry config experimental.new-installer false for now.

@python-poetry python-poetry locked as too heated and limited conversation to collaborators Oct 7, 2020
@python-poetry python-poetry unlocked this conversation Oct 7, 2020
@python-poetry python-poetry locked as off-topic and limited conversation to collaborators Oct 7, 2020
@abn
Copy link
Member

abn commented Oct 7, 2020

Locking this thread as issues reported here might be missed and get lost.

If there are issues with the installer, please first search existing issues. If an issue does not exist, please create one with details on how to reproduce it, -vvv logs and if possible a pyproject.toml file to use. This helps us in identifying the root cause of he issue and improving the installer in upcoming versions.

For issues related to the new installer, if a fix has not already been made for your issue, you may want to disable the new installer like so until the issue has been resolved.

poetry config experimental.new-installer false

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
area/installer Related to the dependency installer kind/feature Feature requests/implementations
Projects
No open projects
1.1
  
Done
Development

Successfully merging this pull request may close these issues.

None yet

10 participants