Broken pipe when using head
to display only the first lines of history
#160
Comments
Michael Bayer (@zzzeek) wrote: somehow mine's not doing that (on OSX) - i can use head, tail, less, i don't get a stack trace. Maybe this has something to do with how the "head" command is behaving on your OS ? |
Janne Vanhala (@jpvanhal) wrote: I'm on OSX (10.9.1) as well and I'm using |
Derek Harland (@donkopotamus) wrote: I've observed this issue before in other python projects. You can generally replicate it yourself directly with something like: $ python -c 'for i in range(100000): print(i)' | head
0
1
2
3
4
5
6
7
8
9
Traceback (most recent call last):
File "<string>", line 1, in <module>
BrokenPipeError: [Errno 32] Broken pipe
Exception BrokenPipeError: BrokenPipeError(32, 'Broken pipe') in <_io.TextIOWrapper name='<stdout>' mode='w' encoding='UTF-8'> ignored NOTE: In Python 3.3 you'll get a To cause it you basically need:
It happens because pythons default signal handling is setup to handle a Options for removing this annoyance:
import signal
signal.signal(signal.SIGPIPE, signal.SIG_DFL) eg $ python -c 'import signal; signal.signal(signal.SIGPIPE, signal.SIG_DFL)
...> for i in range(100000): print(i)' | head
0
1
2
3
4
5
6
7
8
9 |
Michael Bayer (@zzzeek) wrote: ill just catch the IOError. we just put that around the stdout writing routine and we're done, right ? |
Derek Harland (@donkopotamus) wrote: yep |
Changes by Michael Bayer (@zzzeek):
|
Changes by Michael Bayer (@zzzeek):
|
Michael Bayer (@zzzeek) wrote: yup, easy to replicate with a command added like this:
|
Michael Bayer (@zzzeek) wrote: it doesnt seem to be preventable in Python 3, it comes out as a warning in _io.TextIOWrapper. |
Michael Bayer (@zzzeek) wrote: unless we use the signal approach. But, that means, when "head" finishes reading output, the program terminates. Which I don't think we can do. If someone runs a long migration through "head" we don't except the program to silently not complete. I've verified this is the effect. |
Michael Bayer (@zzzeek) wrote:
→ f889070 |
Changes by Michael Bayer (@zzzeek):
|
Florent Xicluna (@flox) wrote: I tried to reproduce the issue on Python3 with the sample code on one of the previous comment. No success. On Debian with Python 3.2.5, 3.3.5 and 3.4.0 $ python3.2 -c 'for i in range(100000): print(i)' | head -2
0
1
Traceback (most recent call last):
File "<string>", line 1, in <module>
IOError: [Errno 32] Broken pipe
$ python3.3 -c 'for i in range(100000): print(i)' | head -2
0
1
Traceback (most recent call last):
File "<string>", line 1, in <module>
BrokenPipeError: [Errno 32] Broken pipe
$ python3.4 -c 'for i in range(100000): print(i)' | head -2
0
1
Traceback (most recent call last):
File "<string>", line 1, in <module>
BrokenPipeError: [Errno 32] Broken pipe
$ I never got the uncatchable message
|
Michael Bayer (@zzzeek) wrote: script:
Py2K:
Py3K:
|
Florent Xicluna (@flox) wrote: I tried exactly the above, and i had the same output for Python 3.2.5, 3.3.5 and 3.4.0 : no message "Exception ignored" at the end. So I guess this is something specific to OS X. |
Michael Bayer (@zzzeek) wrote: seems like |
Changes by Michael Bayer (@zzzeek):
|
Derek Harland (@donkopotamus) wrote: To suppress the BrokenPipe warning on OSX, what about the following:
With python 3.3 on OSX this will produce
|
Frazer McLean (@RazerM) wrote: Cam
|
Frazer McLean (@RazerM) wrote: Yes, it prevents the warning. I'll provide a PR |
Michael Bayer (@zzzeek) wrote: Ok. I'm going to try to work all the ugly into a context manager or something from your PR |
Michael Bayer (@zzzeek) wrote: @RazerM can I please see the OS in use, Python version, stack trace, and commands to reproduce? The current approach of a simple IOError catch resolves the issue for me, and also note that rcollins' snippet on the Python bug is not an "official" workaround, it's conjectural for my specific case (but I cannot reproduce as long as IOError catch is there). script:
running with both Python3.5, python3.6, piping into "head", "echo", etc., no error:
If I remove the existing IOError catch, error is immediate:
Also tried with keyboard interrupts, no issue there either. |
Frazer McLean (@RazerM) wrote: macOS 10.12.6 Using Python 3 installed by homebrew or pyenv I see this:
The reason is that the flush is being deferred until a |
Michael Bayer (@zzzeek) wrote: so the main thing I want to figure out is if I can just put stdout.flush() in write_outstream and have it be local to each time we write things. can you try this patch?
|
Frazer McLean (@RazerM) wrote: The patch doesn't fix it. |
Michael Bayer (@zzzeek) wrote: but the stream is flushed every time and there's a catch around it. how? |
Derek Harland (@donkopotamus) wrote: Just to repeat something from earlier in this thread. Does this not work?
With python 3.5 on OSX this produces:
|
Michael Bayer (@zzzeek) wrote: I'm mostly looking to make the workarounds here local to the inside of the write_outstream() method. putting a block around the "main()" of the program I find bothersome. Will do it if we really have to. |
Migrated issue, originally created by Janne Vanhala (@jpvanhal)
I have hundreds of migrations in a project. I would like to list only the most recent ones. If I use
head
like this:Only the first 10 lines of
alembic history
are correctly printed, but then Alembic crashes with the following exception:The text was updated successfully, but these errors were encountered: