New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

pip wheel . does not follow symlinks whereas python setup.py bdist_wheel does #3500

Open
sbidoul opened this Issue Feb 21, 2016 · 15 comments

Comments

Projects
None yet
7 participants
@sbidoul
Contributor

sbidoul commented Feb 21, 2016

When the package source code contains symlinks, python setup.py bdist_wheel follows them and store their content in the resulting wheel.

pip wheel . ignores them.

@sbidoul sbidoul changed the title from pip wheel does not follow symlinks whereas python setup.py bdist_wheel does to pip wheel . does not follow symlinks whereas python setup.py bdist_wheel does Feb 21, 2016

@sbidoul

This comment has been minimized.

Show comment
Hide comment
@sbidoul

sbidoul Feb 22, 2016

Contributor

I was apparently using a wrong branch. Sorry for the noise.

Contributor

sbidoul commented Feb 22, 2016

I was apparently using a wrong branch. Sorry for the noise.

@sbidoul sbidoul closed this Feb 22, 2016

@sbidoul sbidoul reopened this Feb 23, 2016

@sbidoul

This comment has been minimized.

Show comment
Hide comment
@sbidoul

sbidoul Feb 23, 2016

Contributor

After all I could reproduce. The directory structure must be as follow

  • linked
  • setup
  • setup/setup.py
  • setup/package -> ../linked

In the setup directory, when runnning setup.py bdist_wheel, the package is properly included, while running pip wheel . does not include it.

The root cause seems to be that bdist_wheel builds in place, while pip wheel copies the directory to another temporary place where it builds, and it does that without following the symlinks.

So my first question is why pip wheel does this copy when the source is a local directory?

Contributor

sbidoul commented Feb 23, 2016

After all I could reproduce. The directory structure must be as follow

  • linked
  • setup
  • setup/setup.py
  • setup/package -> ../linked

In the setup directory, when runnning setup.py bdist_wheel, the package is properly included, while running pip wheel . does not include it.

The root cause seems to be that bdist_wheel builds in place, while pip wheel copies the directory to another temporary place where it builds, and it does that without following the symlinks.

So my first question is why pip wheel does this copy when the source is a local directory?

@dstufft

This comment has been minimized.

Show comment
Hide comment
@dstufft

dstufft Mar 30, 2017

Member

This is going to effectively be as intended. We copy the directory so as not to modify the source directory and to contain all of our modifications to a throw away temporary directory.

Member

dstufft commented Mar 30, 2017

This is going to effectively be as intended. We copy the directory so as not to modify the source directory and to contain all of our modifications to a throw away temporary directory.

@dstufft dstufft closed this Mar 30, 2017

@sbidoul

This comment has been minimized.

Show comment
Hide comment
@sbidoul

sbidoul Mar 30, 2017

Contributor

@dstufft would it make sense to follow symlinks when copying?

Contributor

sbidoul commented Mar 30, 2017

@dstufft would it make sense to follow symlinks when copying?

@dstufft

This comment has been minimized.

Show comment
Hide comment
@dstufft

dstufft Mar 30, 2017

Member

@sbidoul Without thinking too hard about it I could see allowing that, but I'd need to make sure we weren't doing that for a reason.

Member

dstufft commented Mar 30, 2017

@sbidoul Without thinking too hard about it I could see allowing that, but I'd need to make sure we weren't doing that for a reason.

@RonnyPfannschmidt

This comment has been minimized.

Show comment
Hide comment
@RonnyPfannschmidt

RonnyPfannschmidt Mar 30, 2017

Contributor

@sbidoul that would present as massive security issue in many cases (symlink attacks are a pain)

Contributor

RonnyPfannschmidt commented Mar 30, 2017

@sbidoul that would present as massive security issue in many cases (symlink attacks are a pain)

@dstufft

This comment has been minimized.

Show comment
Hide comment
@dstufft

dstufft Mar 30, 2017

Member

@RonnyPfannschmidt I'm not sure that's true, since we're going to be executing a setup.py anyways after we copy it, I doubt there's anything you can do with symlinks you couldn't just do in the Python file itself.

Member

dstufft commented Mar 30, 2017

@RonnyPfannschmidt I'm not sure that's true, since we're going to be executing a setup.py anyways after we copy it, I doubt there's anything you can do with symlinks you couldn't just do in the Python file itself.

@RonnyPfannschmidt

This comment has been minimized.

Show comment
Hide comment
@RonnyPfannschmidt

RonnyPfannschmidt Mar 30, 2017

Contributor

@dstufft a paranoid possibility is using a symlink attack to turn a package that ships web served data into a package that ships web served data and some private details of the package builder, but building wheels in isolation should elevate that

Contributor

RonnyPfannschmidt commented Mar 30, 2017

@dstufft a paranoid possibility is using a symlink attack to turn a package that ships web served data into a package that ships web served data and some private details of the package builder, but building wheels in isolation should elevate that

@sbidoul

This comment has been minimized.

Show comment
Hide comment
@sbidoul

sbidoul Mar 30, 2017

Contributor

@RonnyPfannschmidt @dstufft my point is that setup.py bdist_wheel does follow symlinks whereas pip wheel . does not. In my understanding the resulting wheel must be identical. So if there is any security implication (which I doubt) it is already present in setuptools.

Contributor

sbidoul commented Mar 30, 2017

@RonnyPfannschmidt @dstufft my point is that setup.py bdist_wheel does follow symlinks whereas pip wheel . does not. In my understanding the resulting wheel must be identical. So if there is any security implication (which I doubt) it is already present in setuptools.

@sbidoul

This comment has been minimized.

Show comment
Hide comment
@sbidoul

sbidoul Mar 30, 2017

Contributor

@dstufft @xavfernandez does my last comment warrant reopening this?

Contributor

sbidoul commented Mar 30, 2017

@dstufft @xavfernandez does my last comment warrant reopening this?

@r-barnes

This comment has been minimized.

Show comment
Hide comment
@r-barnes

r-barnes Jan 1, 2018

I've just been burned by this symlink issue.

At the very least, wheel should provide a warning that it hasn't followed symlinks so the user knows what's wrong.

r-barnes commented Jan 1, 2018

I've just been burned by this symlink issue.

At the very least, wheel should provide a warning that it hasn't followed symlinks so the user knows what's wrong.

@pradyunsg

This comment has been minimized.

Show comment
Hide comment
@pradyunsg

pradyunsg Mar 4, 2018

Member

I think we can add a small documentation note to pip wheel as a short term workaround.

@pypa/pip-committers What do we want to do here? Following symlinks while copying seems fine to me, if that's what setuptools is doing.

Member

pradyunsg commented Mar 4, 2018

I think we can add a small documentation note to pip wheel as a short term workaround.

@pypa/pip-committers What do we want to do here? Following symlinks while copying seems fine to me, if that's what setuptools is doing.

@RonnyPfannschmidt

This comment has been minimized.

Show comment
Hide comment
@RonnyPfannschmidt

RonnyPfannschmidt Mar 4, 2018

Contributor

setuptools doesnt make a copy of the source folder to begin with

Contributor

RonnyPfannschmidt commented Mar 4, 2018

setuptools doesnt make a copy of the source folder to begin with

@realjo

This comment has been minimized.

Show comment
Hide comment
@realjo

realjo Apr 4, 2018

sbidoul commented on 30 Mar 2017:
... would it make sense to follow symlinks when copying?

It would be sufficient to create a symlink in the copy pointing to the absolute path of the target of the symlink in the source which would reduce the attack risk IMO.

realjo commented Apr 4, 2018

sbidoul commented on 30 Mar 2017:
... would it make sense to follow symlinks when copying?

It would be sufficient to create a symlink in the copy pointing to the absolute path of the target of the symlink in the source which would reduce the attack risk IMO.

@pradyunsg pradyunsg removed their assignment May 11, 2018

@pradyunsg

This comment has been minimized.

Show comment
Hide comment
@pradyunsg

pradyunsg May 11, 2018

Member

I think moving forward, pip is going to offload this work to PEP 517 backends and that should offload these tasks to the backends.

That said, I'll happily accept a PR that I some form improves this situation without breaking things.

Member

pradyunsg commented May 11, 2018

I think moving forward, pip is going to offload this work to PEP 517 backends and that should offload these tasks to the backends.

That said, I'll happily accept a PR that I some form improves this situation without breaking things.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment