-
-
Notifications
You must be signed in to change notification settings - Fork 31.7k
run() - unified high-level interface for subprocess #67531
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Comments
This follows on from the python-ideas thread starting here: https://mail.python.org/pipermail/python-ideas/2015-January/031479.html subprocess gains:
Things I'm not sure about:
|
Another question: With this patch, CalledProcessError and TimeoutExceeded exceptions now have attributes called output and stderr. It would seem less surprising for output to be called stdout, but we can't break existing code that relies on the output attribute. Using properties, either stdout or output could be made an alias for the other, so both names work. Is this desirable? |
A 1) Opting not to capture by default is good. Let people explicitly request that. A 2) "check" seems like a reasonable parameter name for the "should i raise if rc != 0" bool. I don't have any other good bikeshed name suggestions. A 3) Calling it args the same way Popen does is consistent. That the attribute on the exceptions is 'cmd' is a bit of an old wart but seems reasonable. Neither the name 'args' or 'cmd' is actually good for any use in subprocess as it is already an unfortunately multi-typed parameter. It can either be a string or it can be a sequence of strings. The documentation is not clear about what type(s) 'cmd' may be. A Another) Now that they gain a stderr attribute, having a corresponding stdout one would make sense. Implement it as a property and document it with a versionadded 3.5 as usual. |
I haven't checked the code, but does check_output and friends combine stdout and stderr when ouput=PIPE? |
Updated patch following Gregory's suggestions:
Ethan: to combine stdout and stderr in check_output, you need to pass stderr=subprocess.STDOUT - it doesn't assume you want that. I did consider having a simplified interface so you could pass e.g. capture='combine', or capture='stdout', but I don't think the brevity is worth the loss of flexibility. |
Ethan: check_output combines them when stdout=subprocess.STDOUT is passed ( the documentation tells people not to do this. i don't recall why we On Wed Jan 28 2015 at 3:30:59 PM Ethan Furman <report@bugs.python.org>
|
Maybe you don’t want to touch the implementation of the “older high-level API” for fear of subtly breaking something, but for clarification, and perhaps documentation, would the old functions now be equivalent to this? def call(***):
# Verify PIPE not in (stdout, stderr) if needed
return run(***).returncode
def check_call(***):
# Verify PIPE not in (stdout, stderr) if needed
run(***, check=True)
def check_output(***):
# Verify stderr != PIPE if needed
return run(***, check=True, stdout=PIPE) If they are largely equivalent, perhaps simplify the documentation of them in terms of run(), and move them closer to the run() documentation. Is it worth making the CalledProcessError exception a subclass of CompletedProcess? They seem to be basically storing the same information. |
Yep, they are pretty much equivalent to those, except:
I'll work on documenting the trio in those terms. If people want, some/all of the trio could also be implemented on top of run(). check_output() would be the most likely candidate for this, since I copied that code to create run(). I'd probably leave call and check_call as separate implementations to avoid subtle bugs, though. Sharing inheritance between CalledProcessError and CompletedProcess: That would mean that either CompletedProcess is an exception class, even though it's not used as such, or CalledProcessError uses multiple inheritance. I think duplicating a few attributes is preferable to having to think about multiple inheritance, especially since the names aren't all the same (cmd vs args, output vs stdout). |
It’s okay to leave them as independent classes, if you don’t want multiple inheritance. I was just putting the idea out there. It is a similar pattern to the HTTPError exception and HTTPResponse return value for urlopen(). |
Third version of the patch (subprocess_run3):
I didn't reimplement call or check_call - as previously discussed, they are more different from the code in run(), so subtly breaking things is more possible. They are also simpler. |
Would anyone like to do further review of this - or commit it ;-) ? I don't think anyone has objected to the concept since I brought it up on python-ideas, but if anyone is -1, please say so. |
Have you seen the code review comments on the Rietveld, <https://bugs.python.org/review/23342\>? (Maybe check spam emails.) Many of the comments from the earlier patches still stand. In particular, I would like to see the “input” default value addressed, at least for the new run() function, if not the old check_output() function. |
Aha, I hadn't seen any of those. They had indeed been caught by the spam filter. I'll look over them now. |
Fourth version of patch, responding to review comments on Rietveld. The major changes are:
I also made various minor fixes - thanks to everyone who found them. |
A few observations in passing. I beg your pardon for not commenting after a more in depth study of the issue, but as someone that's written and managed several subprocess module front-ends, my general observations seem applicable. subprocess needs easier and more robust ways of managing input and output streams subprocess should have easier ways of managing input: file streams are fine, but plain strings would also be nice for string commands, shell should always be true. for list/Tupperware commands, shell should be false. in fact you'll get an error if you don't ensure this. instead, just have what is passed key execution (for windows, I have no idea. I'm lucky enough not to write windows software these days) subprocess should always terminate processes on program exit robustly (unless asked not too). I always have a hard time figuring out how to get processes to terminate, and how to have them not to. I realize POSIX is black magic, to some degree. I'm attaching a far from perfect front end that I currently use for reference |
Jeff: This makes it somewhat easier to handle input and output as strings instead of streams. Most of the functionality was already there, but this makes it more broadly useful. It doesn't especially address your other points, but I'm not aiming to completely overhaul subprocess.
I wondered why this is not the case before, but on Windows a subprocess is actually launched by a string, not a list. And on POSIX, a string without shell=True is interpreted like a one-element list, so you can do e.g. Popen('ls') instead of Popen(['ls']). Changing that would probably break backwards compatibility in unexpected ways. |
string vs list: see bpo-6760 for some background. Yes, I think it is an API bug, but there is no consensus for fixing it (it would require a deprecation period). Jeff: in general your points to do not seem to be apropos to this particular proposed enhancement, but are instead addressing other aspects of subprocess and should be dealt with in other targeted issues. |
Can I interest any of you in further review? I think I have responded to all comments so far. Thanks! |
Is there anything further I should be doing for this? |
One thing that just popped into my mind that I don’t think has been discussed: The patch adds the new run() function to subprocess.__all__, but the CompletedProcess class is still missing. Was that an oversight or a conscious decision? |
Thanks, that was an oversight. Patch 5 adds CompletedProcess to __all__. |
I am still keen for this to move forwards. I am at PyCon if anyone wants to discuss it in person. |
I'm at pycon as well, we can get this taken care of here. :) |
Great! I'm free after my IPython tutorial this afternoon, all of tomorrow, and I'm around for the sprints. |
6a following in-person review with Gregory:
|
New changeset f0a00ee094ff by Gregory P. Smith in branch 'default': |
thanks! i'll close this later after some buildbot runs and any post-commit reviews. |
I expect this can be closed now, unless there's some post-commit review somewhere that needs addressing? |
This change has made the subprocess docs intimidating and unapproachable again - this is a *LOWER* level swiss-army knife API than the 3 high level convenience functions. I've filed http://bugs.python.org/issue27050 to suggest changing the way this is documented to position run() as a mid-tier API that's more flexible than the high level API, but still more convenient than accessing subprocess.Popen directly. |
Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.
Show more details
GitHub fields:
bugs.python.org fields:
The text was updated successfully, but these errors were encountered: