Fix #725 and #729. /tmp/pip-build issues

[d1b]

#725: /tmp/pip-build not secure
#729: /tmp/pip-build can't be shared by several linux users

Signed-off-by: David db@d1b.org

[pnasrat]

Maybe use logger.fatal

[d1b]

logger isn't used in that module at all at present. Also there is already a case where sys.exit() is called in the file.

[qwcode]

woops, sorry, clicked the wrong button : )

[d1b]

@qwcode ha! so does that mean you meant to press merge ;) ?

[qwcode]

help me out, why the quick return for windows?

[d1b]

os.getuid() and other calls used later in the function do not work on windows.

[qwcode]

windows temp dirs are user isolated/protected anyway, atleast on some versions of windows.
on windows 7, you get something like this:

tempfile.gettempfir()
c:\users\me\appdata\local\temp

how about a comment mentioning that.

[d1b]

Okay. I don't use windows so I had no idea.

[qwcode]

as for tests, unit tests of _get_build_prefix would be sufficient I think (with mocking for fstat/getuid)

[d1b]

@qwcode if you point me where I should add a test for _get_build_prefix I will write one. As an aside I am not that familiar with the pip code base. I intend to help out with a few security related issues that are present in pip.

[d1b]

@qwcode also is there an existing appropriate exception I could raise instead of doing sys.exit(1) ?

[qwcode]

you could create a new tests/test_locations.py for your tests.
there's no unit tests yet for anything in pip.locations.
fyi, there's no class decorators in py2.5, if you end up trying that for your mock patching.
see tests/test_util.py for a rather obsessive mock/unittest example for a function in pip.util
btw, most of our test coverage is functional using scripttest and virtualenv.

our exceptions are in pip.exceptions. the closest one for this is "InstallationError" I guess, although it could occur on any pip command at import, but the code that's running relates to installation prep.

[d1b]

@qwcode I might get around to writing tests one day. For now I do not think one is required. If you think one is - you write it :-)

[qwcode]

this code is being added to prevent a certain security flaw.
whether you end up writing it or not, we need a test case that covers that scenario.

[qwcode]

@guettli, since you opened the issue this relates to, just curious if you have any comments on this fix.

[guettli]

@qwcode, your does not work if /tmp/pip-build-username is a symbolic link. I changed the code to this:

            file_uid=os.lstat(path).st_uid

Look at this example. User "evil" is bad and he wants to modify some files of tguettler, if tguettler uses pip:
The user "evil" created a symbolic link to a directory of tguettler

===> ls -dl /tmp/pip-build-tguettler
lrwxrwxrwx 1 evil evil  27 2012-11-26 09:47 /tmp/pip-build-tguettler -> /home/tguettler/tmp/hidden/

===> ls -dl /tmp/pip-build-tguettler/
drwxr-xr-x 2 tguettler tguettler 6 2012-11-26 09:46 /tmp/pip-build-tguettler/

That's why you need to use lstat().

[qwcode]

@guettli , thanks.
btw, this is @d1b 's pull, just trying to rally feedback

[d1b]

@guettli the change you have suggest using os.lstat(path) is incorrect - as it would be vulnerable to a time of check vs time of use vulnerability. However, you are correct that I am checking the target and not the link's uid, as I forgot to include the os.O_NOFOLLOW flag in the os.open() call.

[guettli]

@d1b I am not a security expert. Can you please explain why your solution is not vulnerable to a time of check vs time of use vulnerability?

[d1b]

@guettli I have updated the code so that the os.open() call includes the os.O_NOFOLLOW flag.

Responding to to your latest question regarding why this code is not vulnerable to a time of check vs time of use vulnerability: Using file descriptors returned by the operating system is a safe way to check the property of a file. Specifying and checking against a path in general not a safe approach as there could be a 'race' between the path check(with lstat) and later using the checked path.

[guettli]

@d1b thank you for your explanation. But I think that the current pull request has the same problem like os.lstat(): there could be a 'race' between the path check(with os.open(...)) and later using the checked path. But I don't care about this race condition. The initial bug I found can be closed.

[d1b]

@guettli there is no such race - unless a daemon or user rm's their temporary build directory :-)

[guettli]

The current changes look good. What can I do to get this into the main line?

[qwcode]

1. see my comment above about adding a test(s)
2. I'd personally prefer the check/creation of this dir not happening at import. i.e. what the path should be, should live in pip.locations, but the creation should happen when needed. the security logic could be factored out into pip.utils function. but that's me, maybe other maintainers wont care.
3. and lastly, this needs one or more other maintainers to look at it I think.

[guettli]

Is it possible to create a file with a different user-id in the unittest? On linux you need root access to change the user-id of a file. I guess that's not possible, but maybe I am wrong.

You are right: it is not good to create the build_prefix during import time. But this was done before (at least the file name was created at import time). If creating the build_prefix should be refactored, then it can be done in a different pull request.

[qwcode]

@guettli

we'd use the "mock" library to mock the fstat and getuid calls, i.e. we wouldn't actually need files owned by other users. we have a few pip tests using mock

you can certainly open another pull requests that resuses his commits up to this point, and then add to them.
i.e. 1) add his repo as a remote 2) check out his branch, 3) then branch off that, 4) then make your changes, and tests, 5) submit a pull from your branch.

[d1b]

@qwcode do you want me to rebase on this branch ?

[qwcode]

it's not auto-mergable right now thru github, so the pt of you rebasing on the latest "develop" is to resolve whatever the conflict is.

[pnasrat]

@d1b you should just go ahead and make it mergeble. Start by pulling develop and fixing any conflicts locally then repush to github.

[d1b]

@pnasrat Sure I'll update it now, but if it becomes un-mergable before it is merged I'll wait until it is about to be merged to fix it again.

[pnasrat]

If you do this I or Marcus will merge
On Jan 25, 2013 7:01 AM, "David" notifications@github.com wrote:

@pnasrat https://github.com/pnasrat I'd rather do that when it is about
to be merged so that I don't have to continually keep it "mergable".

—
Reply to this email directly or view it on GitHubhttps://github.com//pull/734#issuecomment-12698161.

[d1b]

@pnasrat I have re-based the code against develop :-)

[qwcode]

I wasn't looking to merge until after cutting a 1.3 branch and beta (which is soon.)
all my focus right now is on test fixes in and handling any blockers.

[pnasrat]

This is a security fix, which I feel probably should go in to 1.3, but you're doing most of the work on the release so I'll defer to your judgement.

[qwcode]

ok, I hear you, will look at the new tests later tonight .

[qwcode]

ok, the 2 tests make sense to me.
going to go confirm tests work on windows now before merging...

a few thoughts that I won't hold up the merge for (but may fix later):

• there could be a 3rd test for when the dir exists and it's owned by the current user
• our suite for the most part uses nose.tools.assert_raises instead of the try/except pattern in your 2nd test
• since there's just 2 tests, I'm inclined to just use the @patch decorator instead of the manual patching
  (I used manual patching in test_util because we can't use class decorators in py25, and there were so many tests
  with the same patch)

[qwcode]

tests fail on windows. http://jenkins.qwcode.com/job/pip_win_33/10/console
some of the patching needs to be guarded and not run on windows.
I can fix it up later tonight or tomorrow. gotta run for the moment.

[qwcode]

the message from github here is a little misleading.
it almost appears like merged using this pull, but I didn't, I merged pull #779.
I guess because my branch fully subsumes this branch it appears like this.

[d1b]

@qwcode thank you for you help and work in getting my fix for #725 and #729 merged :-)

[qwcode]

sure, np. thanks for being persistent and working on the tests.