stdout error at interpreter shutdown fails to return OS error status

[mkc]

BPO	5319
Nosy	@loewis, @birkenfeld, @terryjreedy, @pitrou, @rbtcollins, @vadmium, @zooba, @corona10
PRs	bpo-39828: Fix json.tool to catch BrokenPipeError. #18779
Files	devfull.patch finalize-error.patch finalize-error.v2.patch: Py_Finalize() returns -1 Py_FinalizeEx.patch: New API Py_FinalizeEx-120.patch: Exit status 120

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2015-11-30.04:12:14.400>
created_at = <Date 2009-02-19.17:38:00.077>
labels = ['interpreter-core', 'type-bug']
title = 'stdout error at interpreter shutdown fails to return OS error status'
updated_at = <Date 2020-03-04.22:11:56.278>
user = 'https://bugs.python.org/mkc'

bugs.python.org fields:

activity = <Date 2020-03-04.22:11:56.278>
actor = 'corona10'
assignee = 'none'
closed = True
closed_date = <Date 2015-11-30.04:12:14.400>
closer = 'martin.panter'
components = ['Interpreter Core']
creation = <Date 2009-02-19.17:38:00.077>
creator = 'mkc'
dependencies = []
files = ['18395', '37513', '39592', '39636', '40306']
hgrepos = []
issue_num = 5319
keywords = ['patch']
message_count = 26.0
messages = ['82482', '82484', '112342', '112919', '112923', '113330', '148423', '232967', '244647', '244662', '244666', '244812', '244817', '244891', '248628', '249183', '249187', '249191', '249192', '249356', '249370', '249377', '249378', '255585', '255605', '255606']
nosy_count = 11.0
nosy_names = ['loewis', 'georg.brandl', 'terry.reedy', 'mkc', 'pitrou', 'rbcollins', 'Arfrever', 'python-dev', 'martin.panter', 'steve.dower', 'corona10']
pr_nums = ['18779']
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue5319'
versions = ['Python 3.6']

[mkc]

$ python2.5 -c 'print((1, 2, 3))' > /dev/full || echo error status
close failed: [Errno 28] No space left on device
$

The above sequence should also output 'error status' before the next
prompt. That is, python should not be returning EXIT_SUCCESS in this
situation.

[mkc]

Oops, I should have used the old python print syntax. Nonetheless, the
behavior is the same:

$ python2.5 -c 'print 1, 2, 3' > /dev/full || echo error status
close failed: [Errno 28] No space left on device
$

[birkenfeld]

Python 2.6, from bpo-5320:

$ python2.6 -c 'print 1, 2, 3' > /dev/full || echo error status
close failed in file object destructor:
Error in sys.excepthook:

Original exception was:
$

Python 3.1 doesn't output anything and also doesn't set an error status.

[terryjreedy]

Georg, are you saying that there is or is not a problem with 2.6 (which is beyond non-critical bug fixes) and/or 3.1?

[pitrou]

Here is a possible patch for 3.x.
It doesn't change the return status of the process, though.

[pitrou]

I've committed the py3k fix in r83854 (3.1 in r83855).
As for 2.6/2.7, I've finally diagnosed the problem: 2.6 and upwards use PyErr_Print() in the file destructor, but the sys.stderr and sys.excepthook objects are already None at that time (because of interpreter shutdown). Therefore the error can't be printed out.
2.5 and earlier used a direct fprintf() of the errno code to stderr.

I propose to ignore the 2.6/2.7 issue of the error not being printed properly, since it only happens at shutdown with a misbehaving stdout, which is not very common.

As for the issue of the stdout error not being reported in the process return code, this would be possible in 3.x, although it would involve changing the signature of Py_Finalize() to include a return status (it currently returns void).

[python-dev]

New changeset 150e096095e5 by Antoine Pitrou in branch '3.2':
Issue bpo-13444: When stdout has been closed explicitly, we should not attempt to flush it at shutdown and print an error.
http://hg.python.org/cpython/rev/150e096095e5

New changeset 37300a1df7d7 by Antoine Pitrou in branch 'default':
Issue bpo-13444: When stdout has been closed explicitly, we should not attempt to flush it at shutdown and print an error.
http://hg.python.org/cpython/rev/37300a1df7d7

[vadmium]

Here is a patch that changes Py_Finalize() to return -1 on error, and then sets the exit status to 1. It did not introduce any failures in the test suite for me. However I suspect it deserves more consideration about backwards compatibility etc, which is beyond my knowledge.

[vadmium]

Updated patch resolving minor merge conflicts with current code, a coding style fix, and some tweaks and syntax fixes to the documentation.

[pitrou]

I don't know how acceptable it is to add a return value to Py_Finalize(). Can it break the stable ABI?

[zooba]

Going from void to int is fine for when the return value is ignored. However, those who check the return value will see garbage on earlier versions.

As always, safest to add a new function here. Or make the stable version continue to be void - it won't affect the exports.

[vadmium]

I guess this would involve:

• Making a new API called Py_Finalize2() or something that returns the status
• Redefine the existing Py_Finalize() to call Py_Finalize2() and ignore the return value

[pitrou]

We would probably call it Py_FinalizeEx(), but yes.

[vadmium]

Here is a patch adding a new Py_FinalizeEx() API. Please let me know if anything else needs to be added. Perhaps some sort of Py_LIMITED_API condition, adding to PC/python3.def, etc. I am not really familiar with these things.

I also changed the version notices to 3.6, assuming it is too late for 3.5.

[rbtcollins]

I think you need to make the following changes:

• Py_Exit with a non-zero status code should probably preserve the code rather than replacing it with 1.
• Ditto in Py_Main.
• Add a defs entry for Py_FinalizeEx - should be a copy of the Py_Finalize entry with the return type modified.

[vadmium]

Thanks for the hints Robert. I will definitely add that defs entry.

I’m a bit unsure about preserving an existing non-zero exit status. It seems a slightly more complicated and unexpected way of do it to me. But I cannot remember why I was originally interested in this issue. Do you have a use case where keeping the exit status > 1 would be good?

Also need to add some entries to What’s New.

[rbtcollins]

def main():
   try:
       thing()
   except SomethingChanged:
       exit(1)
   except BinaryDiff:
       exit(2)
   except:
       exit(3)

This is the sort of pattern e.g. 'diff' has.

Should exit(3) there become exit(1) ? exit(1) means 'succeeded and there were changes'. So no, 3,for this app, means 'broke entirely'.

Exiting with '1' is also poor/confusing for this app. So perhaps we should pick something rarely used - like 255. But anyhow the same logic applies: if an app signals 'error' with a different code, we shouldn't change the apps 'error' signal to be something else when we detect an error.

But OTOH preserving a non-zero not-error code when an error occurs is also wrong.

So I think, on reflection:

1. pick an unusual code. Not 1. Not 77 (automake test skip code). Maybe 255? https://www.freebsd.org/cgi/man.cgi?query=sysexits&sektion=3&apropos=0&manpath=FreeBSD+10.2-RELEASE defines some codes we should avoid as well.
2. We document that python itself may cause this exit code to be used and that apps written in python should never use that code
3. we use your patch as it is - on shutdown error force it to this chosen value.

What do you think?

[vadmium]

I guess I like setting the status to 1 because it matches the status already used by the default exception handling, the value when you raise SystemExit("error message"), and the value used for most internal errors. However I did discover that the exit status is 2 if you give Python wrong CLI arguments or there is an OS error opening a script file.

I’m still rather unsure, only weakly in favour of my original proposal. Maybe see if anyone else has an opinion?

[rbtcollins]

Right. So I think one could also propose that the default exception handling should also change to this new number, but it is a different situation - programs don't conflict with that - they allow it to bubble up, but this situation is outside its control.

CLI handling of Python itself is clearly outside both situations and there's no need to reevaluate it now.

[vadmium]

Maybe 255 is a reasonable choice. Another option is some value slightly less than 126, since Posix shells have special meanings for 126, 127, and > 128 values.

My patches already document the exit status used under the Py_Exit() function: “If :c:func:`Py_FinalizeEx` indicates an error, the exit status is set to 1 [or 255].” Do you think it should be documented somewhere else as well, perhaps for the interpreter as a whole?

[rbtcollins]

It seems to me that how, and when, Python exits the process is important to folk who will never ever touch the C API.

[vadmium]

So far I haven’t noticed much documentation on the exit statuses generated by Python itself (e.g. for unhandled exceptions). I did find some discussion in <https://docs.python.org/3/library/sys.html#sys.exit\>, so perhaps I will add to that.

Since it implies values beyond 127 may not work on some systems, I now propose to use the arbitrary value of 120 instead of 255. (Keeping clear of 126 and 127, which have a special meaning in Posix shells.) Although, that statement about 0–127 was added in 1998 (bbe92932a005), so may not be relevant these days.

[vadmium]

Here is a new patch setting the exit status to 120. I also added a What’s New entry, and some more documentation under sys.exit().

Robert, I added an entry to /PC/python3.def, following the lead of your patch in bpo-2786. You mentioned modifying the return value, but in that file I don’t see any return values. Should I ignore that bit, or am I looking in the wrong place?

[rbtcollins]

@martin I was wrong re: the defs - they only cover function vs data, not return codes. So it looks fine to me.

[python-dev]

New changeset 6b08429a3932 by Martin Panter in branch 'default':
Issue bpo-5319: New Py_FinalizeEx() API to exit with status 120 on failure
https://hg.python.org/cpython/rev/6b08429a3932

[vadmium]

Thanks everyone for helping to get this right.