Add dramatically improved queued installation #3209

techalchemy · 2018-11-11T23:00:24Z

Add dramatically improved queued installation

Use queues and parallelized installation
Better UI/UX -- progress bar moves gradually as items are moved
on and off the queue
Queue is handled by item instead of in massive batches
TODO: Call out when task is done from the install function?

Signed-off-by: Dan Ryan <dan@danryan.co>

- Implement improvements and bugfixes in codebase - Remote archives will now resolve properly Signed-off-by: Dan Ryan <dan@danryan.co>

- Specific construct for isolationg operations Signed-off-by: Dan Ryan <dan@danryan.co>

Signed-off-by: Dan Ryan <dan@danryan.co>

- Use queues and parallelized installation - Better UI/UX -- progress bar moves gradually as items are moved on and off the queue - Queue is handled by item instead of in massive batches - TODO: Call out when task is done from the install function? Signed-off-by: Dan Ryan <dan@danryan.co>

…ndor

Signed-off-by: Dan Ryan <dan@danryan.co>

pipenv/core.py

pipenv/vendor/vistir/spin.py

jxltom · 2018-11-12T02:36:18Z

I just found that when using dummyspinner, sp.text will show in the final output.

For example,

pipenv/pipenv/utils.py

Line 606 in d03cfff

sp.text = to_native_string("Locking VCS Dependencies...")

will outout followings.

➜  ✗ pipenv lock
Locking [dev-packages] dependencies...
Locking [packages] dependencies...
Locking VCS Dependencies...Updated Pipfile.lock (637151)!

techalchemy · 2018-11-12T02:58:35Z

That's because the dummy spinner is not a real spinner, it only writes things to stderr

jxltom · 2018-11-12T03:02:32Z

Is it better to just disable output things when use dummy spinner?

The above output looks a bit redundant when users disable spinner by PIPENV_NOSPIN.

pipenv/project.py

Signed-off-by: Dan Ryan <dan@danryan.co>

techalchemy · 2018-11-12T17:35:17Z

Thanks for the reviews, I cleaned up everything you guys caught and a bunch of other things also. Just cut releases of pythonfinder, requirementslib, and vistir.

Now if this builds I can start merging for the last release

Signed-off-by: Dan Ryan <dan@danryan.co>

- Fix virtualenv - Update pythonfinder Signed-off-by: Dan Ryan <dan@danryan.co>

Signed-off-by: Dan Ryan <dan@danryan.co>

jxltom · 2018-11-13T05:24:02Z

pipenv/vendor/pythonfinder/models/path.py

+        new_order = []
+        target = os.path.normcase(os.path.normpath(os.path.abspath(path)))
+        path_map = {
+            os.path.normcase(os.path.normpath(os.path.abspath(pth))): pth


The pth may be a PosixPath type, then abspath will raise AttributeError: 'PosixPath' object has no attribute 'startswith'

Just saw this. Pushed a fix on the other branch. Thanks for keeping track, I’m planning to annotate these non-pipenv codebases

frostming · 2018-11-13T06:05:55Z

pipenv/vendor/vendor.txt

    attrs==18.2.0
    distlib==0.2.8
    packaging==18.0
    pyparsing==2.2.2
    pytoml==0.1.19
    plette==0.2.2
-        tomlkit==0.4.6
+        tomlkit==0.5.2


Don't update this line or otherwise vendoring build will fail.

I undid this on the other branch since I need that to pass first anyway

…tenance/merge-3191-3196-3209 - Merge #3191, #3196, and #3209 - Closes #3191 - Closes #3196 - Closes #3214 - Closes #3209 Signed-off-by: Dan Ryan <dan@danryan.co>

Merge #3191, #3196 and #3209

techalchemy added 13 commits November 8, 2018 04:47

Grab updates from latest vendored changes

36f054d

Signed-off-by: Dan Ryan <dan@danryan.co>

Fix broken requirementslib updates

5b49670

Signed-off-by: Dan Ryan <dan@danryan.co>

Merge branch 'master' into update-vendor

f3003b6

Revendor requirementslib

2b90c89

- Implement improvements and bugfixes in codebase - Remote archives will now resolve properly Signed-off-by: Dan Ryan <dan@danryan.co>

Introduce pipenv.environments.Environment

dec7be5

- Specific construct for isolationg operations Signed-off-by: Dan Ryan <dan@danryan.co>

Update vistir and requirementslib

642b6f9

Signed-off-by: Dan Ryan <dan@danryan.co>

Fix stdout and stderr wrappers

aedb41c

Signed-off-by: Dan Ryan <dan@danryan.co>

Fix stdout and stderr wrappers

45100b8

Signed-off-by: Dan Ryan <dan@danryan.co>

Merge branch 'update-vendor' of github.com:pypa/pipenv into update-ve…

5602952

…ndor

Merge branch 'update-vendor' into feature/improved-async-installer

690d1e8

Merge branch 'master' into update-vendor

8644611

Merge branch 'update-vendor' into feature/improved-async-installer

ac96370

techalchemy added Type: Enhancement 💡 This is a feature or enhancement request. Category: Performance Issue relates to performance labels Nov 11, 2018

techalchemy requested a review from uranusjr November 11, 2018 23:00

techalchemy added 3 commits November 11, 2018 18:05

Fix configparser import

8643a73

Signed-off-by: Dan Ryan <dan@danryan.co>

Fix resource errors

fe9d996

Signed-off-by: Dan Ryan <dan@danryan.co>

Fix python 2.7 installations

e5be2ac

Signed-off-by: Dan Ryan <dan@danryan.co>

frostming reviewed Nov 12, 2018

View reviewed changes

pipenv/core.py Show resolved Hide resolved

frostming mentioned this pull request Nov 12, 2018

Fix unexpected error when installing editable packages #3207

Closed

2 tasks

jxltom reviewed Nov 12, 2018

View reviewed changes

pipenv/vendor/vistir/spin.py Show resolved Hide resolved

jxltom reviewed Nov 12, 2018

View reviewed changes

pipenv/project.py Outdated Show resolved Hide resolved

techalchemy added 3 commits November 11, 2018 23:38

Fix various bugs with python 2.7 and vendored deps

3984632

Signed-off-by: Dan Ryan <dan@danryan.co>

Support python 2 parsing

32a6dd3

Signed-off-by: Dan Ryan <dan@danryan.co>

Fix environment site import

4009198

Signed-off-by: Dan Ryan <dan@danryan.co>

techalchemy added 2 commits November 12, 2018 10:06

Fix import errors on setup parsing

7e139ad

Signed-off-by: Dan Ryan <dan@danryan.co>

Revendor

8502ac9

Signed-off-by: Dan Ryan <dan@danryan.co>

techalchemy added 5 commits November 12, 2018 13:40

Fix prefix comparison for py2

0c7f287

Signed-off-by: Dan Ryan <dan@danryan.co>

Merge branch 'master' into feature/improved-async-installer

eed0f0f

Signed-off-by: Dan Ryan <dan@danryan.co>

no samefile for windows python2.7

d7d50ef

Signed-off-by: Dan Ryan <dan@danryan.co>

Fix bugs in environment implementation

9296f56

- Fix virtualenv - Update pythonfinder Signed-off-by: Dan Ryan <dan@danryan.co>

Fix syntax

e62b800

Signed-off-by: Dan Ryan <dan@danryan.co>

jxltom reviewed Nov 13, 2018

View reviewed changes

frostming reviewed Nov 13, 2018

View reviewed changes

techalchemy mentioned this pull request Nov 13, 2018

Merge #3191, #3196 and #3209 #3217

Merged

techalchemy closed this in #3217 Nov 14, 2018

techalchemy added a commit that referenced this pull request Nov 14, 2018

Merge pull request #3217 from pypa/maintenance/merge-3191-3196-3209

d9ebad3

Merge #3191, #3196 and #3209

frostming deleted the feature/improved-async-installer branch November 5, 2020 10:04

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add dramatically improved queued installation #3209

Add dramatically improved queued installation #3209

techalchemy commented Nov 11, 2018

jxltom commented Nov 12, 2018

techalchemy commented Nov 12, 2018

jxltom commented Nov 12, 2018 •

edited

Loading

techalchemy commented Nov 12, 2018

jxltom Nov 13, 2018 •

edited

Loading

techalchemy Nov 13, 2018

jxltom Nov 13, 2018

frostming Nov 13, 2018

techalchemy Nov 13, 2018

Add dramatically improved queued installation #3209

Add dramatically improved queued installation #3209

Conversation

techalchemy commented Nov 11, 2018

jxltom commented Nov 12, 2018

techalchemy commented Nov 12, 2018

jxltom commented Nov 12, 2018 • edited Loading

techalchemy commented Nov 12, 2018

jxltom Nov 13, 2018 • edited Loading

Choose a reason for hiding this comment

techalchemy Nov 13, 2018

Choose a reason for hiding this comment

jxltom Nov 13, 2018

Choose a reason for hiding this comment

frostming Nov 13, 2018

Choose a reason for hiding this comment

techalchemy Nov 13, 2018

Choose a reason for hiding this comment

jxltom commented Nov 12, 2018 •

edited

Loading

jxltom Nov 13, 2018 •

edited

Loading