incorrect path in subprocess.Popen() FileNotFoundError message

[cjerdonek]

BPO	16114
Nosy	@gpshead, @asvetlov, @cjerdonek
Dependencies	bpo-16115: test the executable arg to Popen() and improve related docs
Files	issue-16114-1-tests-default.patch

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = 'https://github.com/gpshead'
closed_at = <Date 2013-06-26.00:47:09.461>
created_at = <Date 2012-10-03.04:16:47.169>
labels = ['type-bug', 'library']
title = 'incorrect path in subprocess.Popen() FileNotFoundError message'
updated_at = <Date 2013-06-26.00:47:09.453>
user = 'https://github.com/cjerdonek'

bugs.python.org fields:

activity = <Date 2013-06-26.00:47:09.453>
actor = 'gregory.p.smith'
assignee = 'gregory.p.smith'
closed = True
closed_date = <Date 2013-06-26.00:47:09.461>
closer = 'gregory.p.smith'
components = ['Library (Lib)']
creation = <Date 2012-10-03.04:16:47.169>
creator = 'chris.jerdonek'
dependencies = ['16115']
files = ['27396']
hgrepos = []
issue_num = 16114
keywords = ['patch']
message_count = 19.0
messages = ['171850', '171851', '171852', '171857', '171864', '172125', '172128', '172129', '172135', '172456', '172556', '172561', '172562', '172563', '172564', '172567', '172570', '175323', '191893']
nosy_count = 5.0
nosy_names = ['gregory.p.smith', 'Arfrever', 'asvetlov', 'chris.jerdonek', 'python-dev']
pr_nums = []
priority = 'normal'
resolution = 'fixed'
stage = 'commit review'
status = 'closed'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue16114'
versions = ['Python 3.2', 'Python 3.3', 'Python 3.4']

[cjerdonek]

The error message in the FileNotFoundError error raised by subprocess.Popen() displays the wrong path when the bad path is due to the executable argument rather than args. The message gives the path for args[0] rather than for the executable argument.

For example, this--

import subprocess, sys
python_path = sys.executable
p = subprocess.Popen([python_path, "-c", "import sys; sys.exit(1)"])
p.wait()
p = subprocess.Popen([python_path, "-c", "import sys; sys.exit(1)"],
                     executable="foo")
p.wait()

gives--

Traceback (most recent call last):
  File "test-subprocess.py", line 6, in <module>
    executable="foo")
  File ".../Lib/subprocess.py", line 818, in __init__
    restore_signals, start_new_session)
  File ".../Lib/subprocess.py", line 1416, in _execute_child
    raise child_exception_type(errno_num, err_msg)
FileNotFoundError: [Errno 2] No such file or directory: '.../python.exe'

The path in the last line should read "foo" since '.../python.exe' is obviously found as evidenced from the previous Popen() invocation.

[cjerdonek]

2.7 is not affected because 2.7 makes no attempt to display the path:

OSError: [Errno 2] No such file or directory

[cjerdonek]

It looks like the error is here:

if issubclass(child_exception_type, OSError) and hex_errno:
    errno_num = int(hex_errno, 16)
    if errno_num != 0:
        err_msg = os.strerror(errno_num)
        if errno_num == errno.ENOENT:
            err_msg += ': ' + repr(args[0])
    raise child_exception_type(errno_num, err_msg)
raise child_exception_type(err_msg)

http://hg.python.org/cpython/file/b40025e37bcd/Lib/subprocess.py#l1415

The "args[0]" should be "executable" when appropriate.

[cjerdonek]

Adding bpo-16115 as a dependency so that the more general case can be settled and committed before dealing with the current issue's more specific (and platform-specific) case.

[cjerdonek]

Attaching proposed tests.

[asvetlov]

Maybe better to fix Windows behavior for unifying FileNotFoundError?

[cjerdonek]

I will be opening a separate issue to have the same behavior in a future version after this issue is closed. For existing releases, we don't want to break working code that could be relying on the difference.

[cjerdonek]

Sorry, I was confusing this issue with bpo-15533. Yes, I support adding the file path to the error message in the Windows implementation, though my preference would be for that to be addressed as part of a separate issue.

[asvetlov]

I'm ok with your patch if it will be applied to 3.2 etc and after that new issue will fix Windows problem.
BTW, the patch fails for 3.2 and 3.3 but works for 3.4

[cjerdonek]

BTW, the patch fails for 3.2 and 3.3 but works for 3.4

By "patch" do you mean "test"? And by "works", do you mean fails or succeeds? :)

I haven't prepared a patch yet, but I just started working on it. On my machine, I found that the test fails as is on 3.3 and 3.4, but on 3.2 it fails for a different reason: the FileNotFoundError needs to be switched to OSError (which was to be expected).

[gpshead]

replacing repr(args[0]) with repr(executable) in the identified python should be sufficient for this bug as originally reported.

BUT it goes deeper: I just ran into this error in a different case. It also happens when cwd is passed and the chdir(cwd) fails in the child process. args[0] is reported as not found rather than cwd in the error message.

I have a test and fix for that. I'll take care of both.

[cjerdonek]

replacing repr(args[0]) with repr(executable) in the identified python should be sufficient for this bug as originally reported.

Yes, almost. The executable variable is a bytes object so it needs to be fsdecoded first.

I have a test and fix for that. I'll take care of both.

Let me at least upload what I already prepared. You can add to it or modify it as you see fit.

[python-dev]

New changeset e938fa6be509 by Gregory P. Smith in branch '3.2':
Fixes Issue bpo-16114: The subprocess module no longer provides a
http://hg.python.org/cpython/rev/e938fa6be509

New changeset ee30d7ef70be by Gregory P. Smith in branch '3.3':
Fixes Issue bpo-16114: The subprocess module no longer provides a
http://hg.python.org/cpython/rev/ee30d7ef70be

[python-dev]

New changeset 543bb0e0afb9 by Gregory P. Smith in branch 'default':
Fixes Issue bpo-16114: The subprocess module no longer provides a
http://hg.python.org/cpython/rev/543bb0e0afb9

[gpshead]

I'm keeping this open until I backport this to subprocess32 for use on Python 2.

[cjerdonek]

I made some comments on the changes as committed here:

http://mail.python.org/pipermail/python-dev/2012-October/122125.html

"""
It would be cleaner to use the self.assertRaises() pattern here and
also probably better to share code across the three test methods which
are nearly identical to one another (there is a fourth scenario I
would also add of shell=True).

I would also check for FileNotFoundError instead of OSError in the 3.3
and later versions.
"""

[cjerdonek]

Maybe better to fix Windows behavior for unifying FileNotFoundError?

I created bpo-16185 to include the path in the messages of the corresponding errors on Windows.

[gpshead]

fyi - I agree with your comments about the test and assertRaises. This code is old, there's a lot that could be improved in there. I chose to maintain a style equivalent to the existing surrounding code.

Feel free to clean that up.

[gpshead]

I backported the fix to this in the subprocess32 3.2.5rc1 release I made a week or two ago.