PEP 518: enable source installs for build dependencies #5336

benoit-pierre · 2018-04-26T09:45:35Z

Re-enable support for source installs, and detect fork-bombs.

Constraints:

there could be multiple concurrent runs of pip (e.g. installing to different target directories with -t)
a fork-bomb can happen during a chain of prepare requirement -> install build dependencies -> [...] -> try to prepare the original requirement again, so we can't identify requirements based on a NAME-VERSION key because those are not always known (or can be dependend upon) before and during preparation

Solution:

identify requirements based on their source link (sans fragments)
each pip run will use a temporary directory to track preparation of requirements
sharing the directory is done using a simple environment variable to forward the information to child processess
there's no need for a lock because there's always only one active process during a call to pip download/install/wheel (during the install build dependencies phase, the parent process is waiting for the pip sub-process completion)

Note: this PR is based on and includes the changes in #5286 (see here for a comparison between the 2).

Fixes #5229.

benoit-pierre · 2018-04-26T09:48:32Z

TODO:

add a news entry
fix py_module declaration for pep518_forkbomb test project
drop packages4 test data
check documentation is up-to-date
add another delayed fork-bomb test?

pradyunsg · 2018-05-10T12:15:55Z

You'll want to modify the documentation paragraph about the limitations, changing it to say in pip 10.

benoit-pierre · 2018-05-10T14:20:05Z

I updated the check-list above. Will amend after rebasing on master once #5286 is merged.

BrownTruck · 2018-05-18T14:00:05Z

Hello!

I am an automated bot and I have noticed that this pull request is not currently able to be merged. If you are able to either merge the master branch into this pull request or rebase this pull request against master then it will eligible for code review and hopefully merging!

benoit-pierre · 2018-05-19T10:52:56Z

Rebased on master + updated the documentation.

pradyunsg

I don't have the bandwidth to do a full review right now.

pradyunsg · 2018-05-19T10:54:46Z

docs/reference/pip.rst

-  unbounded recursion involved was not considered acceptable, and so
-  installation of build dependencies from source has been disabled until a safe
-  resolution of this issue is found.
+* ``pip<18.0`` only support installing build requirements from wheels.


merge it with the next point?

pradyunsg

Code looks good to me. Inline notes on tests.

pradyunsg · 2018-06-11T15:34:10Z

news/5229.feature

@@ -0,0 +1 @@
+Add support for installing PEP 518 build dependencies from source.


Could you remove the period?

I said that for consistency across news files -- I usually commit them without periods. Looking at the directory currently, it's inconsistent already so I guess this doesn't matter all that much.

Yeah, I'd definitely say this doesn't matter. FWIW personally I prefer complete sentences (i.e., periods).

pradyunsg · 2018-06-11T15:36:20Z

tests/functional/test_install.py

+        path_to_url(data.packages.join('pep518_forkbomb-235.tar.gz'))
+    ) in result.stdout, result.stdout
+
+


A test for fork-bomb due to a cycle in the build-requirements (instead of self-depending) would be nice.

pradyunsg · 2018-06-11T16:13:18Z

tests/functional/test_install.py

        data.src.join("pep518_with_extra_and_markers-1.0"),
        use_module=True,
    )


+@pytest.mark.parametrize('command', ('install', 'wheel'))
+def test_pep518_forkbomb(script, data, common_wheels, command):


Is it possible to limit this test so that it doesn't actually fork-bomb the system is we mess up the logic being tested?

I'll mark the test with @pytest.mark.timeout(60).

pradyunsg · 2018-06-11T16:15:57Z

tests/lib/__init__.py

@@ -100,10 +100,6 @@ def packages2(self):
    def packages3(self):
        return self.root.join("packages3")

-    @property
-    def packages4(self):


bye bye packages4! 👋

pradyunsg

Code looks good to me. Inline notes on tests.

benoit-pierre · 2018-06-17T17:19:54Z

Rebased with additional tests.

pradyunsg

LGTM.

pradyunsg · 2018-06-21T17:32:37Z

A gentle ping for other @pypa/pip-committers. Would appreciate more 👀 through this.

Happy to merge as is though. :)

BrownTruck · 2018-06-25T21:30:03Z

Hello!

I am an automated bot and I have noticed that this pull request is not currently able to be merged. If you are able to either merge the master branch into this pull request or rebase this pull request against master then it will eligible for code review and hopefully merging!

No need for a wheel of `simple` now that build dependencies support source installs.

pradyunsg · 2018-07-02T17:54:27Z

src/pip/_internal/req/req_tracker.py

+        self.cleanup()
+
+    def _entry_path(self, link):
+        hashed = hashlib.sha224(link.url_without_fragment.encode()).hexdigest()


Noting here that I did briefly think that we should split the folder name, to be like pip's cache.

wheels [...] ├── bf │ └── c9 │ └── a3 │ └── c538d90ef17cf7823fa51fc701a7a7a910a80f6a405bf15b1a │ └── future-0.16.0-cp37-none-any.whl [...]

I understand the cache might be this way to prevent far too many files in a single directory. I doubt that build-trees would get all that large though, considering that we only have a tracker for what's being built right now.

pradyunsg · 2018-07-11T06:36:50Z

@pypa/pip-committers Does anyone else have bandwidth to review this?

auvipy

nothing outstanding remain imho.

This was reverted because pip didn't support installing build deps from sdists. But this looks fixed since pypa/pip#5336 pip now builds and installs pycairo before building pygobject. This reverts commit ad1bbfa.

lock · 2019-06-01T16:13:54Z

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.

benoit-pierre force-pushed the src_build_reqs_in_pep518 branch from a70ef1e to 6bcfb60 Compare April 27, 2018 09:45

pradyunsg added the S: needs triage Issues/PRs that need to be triaged label May 11, 2018

benoit-pierre mentioned this pull request May 11, 2018

improve handling of PEP 518 build requirements #5286

Merged

pradyunsg mentioned this pull request May 16, 2018

PEP-517 implementation for pip #5407

Closed

BrownTruck added the needs rebase or merge PR has conflicts with current master label May 18, 2018

benoit-pierre force-pushed the src_build_reqs_in_pep518 branch from 6bcfb60 to 6e1ef58 Compare May 19, 2018 10:50

pypa-bot removed the needs rebase or merge PR has conflicts with current master label May 19, 2018

pradyunsg reviewed May 19, 2018

View reviewed changes

pradyunsg added type: enhancement Improvements to functionality PEP implementation Involves some PEP and removed S: needs triage Issues/PRs that need to be triaged labels May 19, 2018

benoit-pierre closed this May 21, 2018

benoit-pierre reopened this May 21, 2018

pradyunsg added this to the 18.0 milestone Jun 6, 2018

pradyunsg reviewed Jun 11, 2018

View reviewed changes

benoit-pierre mentioned this pull request Jun 12, 2018

Load build backend in isolated build environment with --no-index given #5402

Closed

benoit-pierre force-pushed the src_build_reqs_in_pep518 branch from e3b2efb to 4dd5b27 Compare June 17, 2018 17:19

pradyunsg approved these changes Jun 17, 2018

View reviewed changes

BrownTruck added the needs rebase or merge PR has conflicts with current master label Jun 25, 2018

benoit-pierre force-pushed the src_build_reqs_in_pep518 branch from 4dd5b27 to ec64ca9 Compare June 26, 2018 08:35

pypa-bot removed the needs rebase or merge PR has conflicts with current master label Jun 26, 2018

benoit-pierre force-pushed the src_build_reqs_in_pep518 branch from ec64ca9 to 06fb565 Compare June 26, 2018 08:39

benoit-pierre added 3 commits June 26, 2018 10:40

enable source installs for build dependencies

3ae5219

detect fork-bombs during build dependencies installs

43b8ed4

tests: drop package4 test data

06fb565

No need for a wheel of `simple` now that build dependencies support source installs.

pradyunsg reviewed Jul 2, 2018

View reviewed changes

pradyunsg added the !release blocker Hold a release until this is resolved label Jul 2, 2018

auvipy reviewed Jul 15, 2018

View reviewed changes

pradyunsg merged commit 0bd01d9 into pypa:master Jul 20, 2018

pradyunsg removed the !release blocker Hold a release until this is resolved label Jul 22, 2018

segevfiner mentioned this pull request Jul 23, 2018

Use PEP 518 pyproject.toml to require cffi pyca/pynacl#453

Closed

jamadden mentioned this pull request Aug 9, 2018

python 3.7 binary wheels. zopefoundation/BTrees#92

Merged

benoit-pierre mentioned this pull request Aug 22, 2018

Intentional reuse of deleted root dir in RequirementTracker #5725

Closed

benoit-pierre deleted the src_build_reqs_in_pep518 branch September 23, 2018 17:44

pv mentioned this pull request Sep 27, 2018

Enable pip build isolation scipy/scipy#8734

Closed

2 tasks

lock bot added the auto-locked Outdated issues that have been locked by automation label Jun 1, 2019

lock bot locked as resolved and limited conversation to collaborators Jun 1, 2019

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

PEP 518: enable source installs for build dependencies #5336

PEP 518: enable source installs for build dependencies #5336

benoit-pierre commented Apr 26, 2018

benoit-pierre commented Apr 26, 2018 •

edited by pradyunsg

Loading

pradyunsg commented May 10, 2018

benoit-pierre commented May 10, 2018

BrownTruck commented May 18, 2018

benoit-pierre commented May 19, 2018

pradyunsg left a comment

pradyunsg May 19, 2018

benoit-pierre May 19, 2018

pradyunsg left a comment

pradyunsg Jun 11, 2018

benoit-pierre Jun 12, 2018

pradyunsg Jun 12, 2018

pfmoore Jun 12, 2018

benoit-pierre Jun 12, 2018

pradyunsg Jun 11, 2018

pradyunsg Jun 11, 2018

benoit-pierre Jun 12, 2018

pradyunsg Jun 11, 2018

pradyunsg left a comment

benoit-pierre commented Jun 17, 2018

pradyunsg left a comment

pradyunsg commented Jun 21, 2018

BrownTruck commented Jun 25, 2018

pradyunsg Jul 2, 2018

pradyunsg commented Jul 11, 2018

auvipy left a comment

lock bot commented Jun 1, 2019

		@@ -0,0 +1 @@
		Add support for installing PEP 518 build dependencies from source.

		path_to_url(data.packages.join('pep518_forkbomb-235.tar.gz'))
		) in result.stdout, result.stdout

PEP 518: enable source installs for build dependencies #5336

PEP 518: enable source installs for build dependencies #5336

Conversation

benoit-pierre commented Apr 26, 2018

benoit-pierre commented Apr 26, 2018 • edited by pradyunsg Loading

pradyunsg commented May 10, 2018

benoit-pierre commented May 10, 2018

BrownTruck commented May 18, 2018

benoit-pierre commented May 19, 2018

pradyunsg left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

pradyunsg left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

pradyunsg left a comment

Choose a reason for hiding this comment

benoit-pierre commented Jun 17, 2018

pradyunsg left a comment

Choose a reason for hiding this comment

pradyunsg commented Jun 21, 2018

BrownTruck commented Jun 25, 2018

Choose a reason for hiding this comment

pradyunsg commented Jul 11, 2018

auvipy left a comment

Choose a reason for hiding this comment

lock bot commented Jun 1, 2019

benoit-pierre commented Apr 26, 2018 •

edited by pradyunsg

Loading