os.path.exists returns false negatives in MAC environments.

[sbennett]

BPO	1117601
Nosy	@terryjreedy, @josiahcarlson, @pitrou

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2017-03-24.16:34:15.837>
created_at = <Date 2005-02-07.00:57:06.000>
labels = ['type-bug', 'library']
title = 'os.path.exists returns false negatives in MAC environments.'
updated_at = <Date 2017-03-24.16:34:15.836>
user = 'https://bugs.python.org/sbennett'

bugs.python.org fields:

activity = <Date 2017-03-24.16:34:15.836>
actor = 'serhiy.storchaka'
assignee = 'none'
closed = True
closed_date = <Date 2017-03-24.16:34:15.837>
closer = 'serhiy.storchaka'
components = ['Library (Lib)']
creation = <Date 2005-02-07.00:57:06.000>
creator = 'sbennett'
dependencies = []
files = []
hgrepos = []
issue_num = 1117601
keywords = []
message_count = 11.0
messages = ['60645', '60646', '60647', '60648', '60649', '70891', '70893', '70895', '91909', '114406', '253564']
nosy_count = 7.0
nosy_names = ['terry.reedy', 'josiahcarlson', 'pitrou', 'sbennett', 'vdupras', 'cda32', 'stefano-m']
pr_nums = []
priority = 'normal'
resolution = 'wont fix'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue1117601'
versions = ['Python 3.1', 'Python 2.7', 'Python 3.2']

[sbennett]

In Mandatory Access Control environments (such as
SELinux), it's quite possible for stat to fail with
permission denied. In this case, os.path.exists will
return False incorrectly. The simple(ish) fix is to
check for an access denied error (which would indicate
present, but not readable) when using stat to check for
existence of files.

[terryjreedy]

Logged In: YES
user_id=593130

Does 'access denied' always mean 'present but not readable'
in every environment that gives such messages? I ask
because I have vague memories of wasting time trying to
get access to something that did not exist, because access
denied (or something like that) meant that I was denied
access even to info about whether it existed or not.

In any case, a reproducible example would help someone to
verify, fix, and write a test case for this if it is deemed to be
a fixable bug.

[sbennett]

Logged In: YES
user_id=817465

As far as I know (at least for SELinux), permission denied
on stat() always means that the file exists, but getattr
isn't allowed. As for a reproducible test case, probably the
simplest example is a vanilla Fedora Core 3 system with
SELinux enabled and strict policy. From a regular user
account, call os.path.exists("/etc/shadow"). It will return
False even though the file exists. For comparison, an ls -l /etc/shadow from the command line will simply print
'Permission Denied'.

[josiahcarlson]

Logged In: YES
user_id=341410

I believe Terry was curious about something like
os.path.exists("/etc/shadow/abc123") vs ls -l /etc/shadow/abc123. If not, I know I am curious, and I
believe it may help with a corner case.

[sbennett]

Logged In: YES
user_id=817465

In the case of /etc/shadow/abc123, the stat will fail with
"Not a directory". However, attempting to stat /root/abc123
as a limited account will return permission denied unless
you have the search permission to the parent directory.
However, this is an issue with regular Unix permissions too
-- try to stat() a file that's inside a directory with 000
permissions.

One possible way around this is to attempt to get a listing
of the parent dir if the stat fails with permission denied
-- if it succeeds then the file exists but can't be statted;
if it fails then (at least for the purposes of the library
functions) if doesn't.

[vdupras]

hsoft-dev:~ hsoft$ mkdir foobar
hsoft-dev:~ hsoft$ echo "baz" > foobar/baz
hsoft-dev:~ hsoft$ chmod 000 foobar/baz
hsoft-dev:~ hsoft$ python
Python 2.5.2 (r252:60911, Feb 22 2008, 07:57:53) 
[GCC 4.0.1 (Apple Computer, Inc. build 5363)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import os.path
>>> os.path.exists('foobar/baz')
True
>>> 
hsoft-dev:~ hsoft$ chmod 000 foobar
hsoft-dev:~ hsoft$ python
Python 2.5.2 (r252:60911, Feb 22 2008, 07:57:53) 
[GCC 4.0.1 (Apple Computer, Inc. build 5363)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import os.path
>>> os.path.exists('foobar/baz')
False
>>> os.path.exists('foobar')
True
>>> 

This seems like the correct behavior to me.

[pitrou]

The only sane alternative to the current behaviour would be to raise an
Exception from os.path.exists rather than returning False. But it would
also break a lot of code, and complexify code using os.path.exists which
currently doesn't need to check for exceptions. Returning True sounds
completely incorrect on the other hand.

If there is no other straightforward method than stat() to know if a
path exists, I suggest closing this bug as wontfix.

[vdupras]

Antoine, I think that when this bug was filed, the first
"os.path.exists('foobar/baz')" in my example would return False.

So it's not that this bug shouldn't be fixed, but that it is already
fixed.

[cda32]

I also hit upon this issue and IMHO returning False in a "permission
denied" scenario is less than obvious behaviour.

It also means I have no way to catch this edge case in my own code
immediately, I have to implement far larger logic to check now if False
means the file doesn't exist, or if the user doesn't have access to the
file.

os.path.exists should absolutely raise a Permission exception.

[BreamoreBoy]

Previous comments suggest there is no agreement as to whether or not this is an oustanding bug.

[stefano-m]

FWIW, I have just been experiencing this on CentOS 6.5 with Python 2.7.5 (sorry).

I could make the Python code happy by running chcon[1] with the correct context type, otherwise os.path.exists would happily return False for a file that actually existed.

[1] https://fedoraproject.org/wiki/SELinux/chcon