Broken pipe when using `head` to display only the first lines of history #160

sqlalchemy-bot · 2013-12-18T12:18:43Z

Migrated issue, originally created by Janne Vanhala (@jpvanhal)

I have hundreds of migrations in a project. I would like to list only the most recent ones. If I use head like this:

alembic history | head

Only the first 10 lines of alembic history are correctly printed, but then Alembic crashes with the following exception:

Traceback (most recent call last):
  File "/Users/janne/.virtualenvs/project/bin/alembic", line 9, in <module>
    load_entry_point('alembic==0.6.1', 'console_scripts', 'alembic')()
  File "/Users/janne/.virtualenvs/project/lib/python2.7/site-packages/alembic/config.py", line 298, in main
    CommandLine(prog=prog).main(argv=argv)
  File "/Users/janne/.virtualenvs/project/lib/python2.7/site-packages/alembic/config.py", line 293, in main
    self.run_cmd(cfg, options)
  File "/Users/janne/.virtualenvs/project/lib/python2.7/site-packages/alembic/config.py", line 279, in run_cmd
    **dict((k, getattr(options, k)) for k in kwarg)
  File "/Users/janne/.virtualenvs/project/lib/python2.7/site-packages/alembic/command.py", line 193, in history
    _display_history(config, script, base, head)
  File "/Users/janne/.virtualenvs/project/lib/python2.7/site-packages/alembic/command.py", line 171, in _display_history
    config.print_stdout(sc.log_entry)
  File "/Users/janne/.virtualenvs/project/lib/python2.7/site-packages/alembic/config.py", line 95, in print_stdout
    "\n"
  File "/Users/janne/.virtualenvs/project/lib/python2.7/site-packages/alembic/util.py", line 139, in write_outstream
    stream.write(t)
IOError: [Errno 32] Broken pipe

The text was updated successfully, but these errors were encountered:

sqlalchemy-bot · 2013-12-18T16:37:42Z

Michael Bayer (@zzzeek) wrote:

somehow mine's not doing that (on OSX) - i can use head, tail, less, i don't get a stack trace. Maybe this has something to do with how the "head" command is behaving on your OS ?

sqlalchemy-bot · 2014-01-18T21:47:39Z

Janne Vanhala (@jpvanhal) wrote:

I'm on OSX (10.9.1) as well and I'm using head that comes with the OS. This doesn't happen with tail or less, only head.

sqlalchemy-bot · 2014-02-04T00:34:32Z

Derek Harland (@donkopotamus) wrote:

I've observed this issue before in other python projects. You can generally replicate it yourself directly with something like:

$ python -c 'for i in range(100000): print(i)' | head
0
1
2
3
4
5
6
7
8
9
Traceback (most recent call last):
  File "<string>", line 1, in <module>
BrokenPipeError: [Errno 32] Broken pipe
Exception BrokenPipeError: BrokenPipeError(32, 'Broken pipe') in <_io.TextIOWrapper name='<stdout>' mode='w' encoding='UTF-8'> ignored

NOTE: In Python 3.3 you'll get a BrokenPipeError, in versions prior its just a plain IOError

To cause it you basically need:

python to write a large block of data to (a piped) stdout. It has to be enough to overflow the write buffer.
the pipe reader to quit/die. As long as the writer has overflown the buffer on the block the reader was reading, then the writer will die with the above exception. (e.g. it won't die if you reduce that loop size).

It happens because pythons default signal handling is setup to handle a SIGPIPE by raising an exception (similar to the way it handles a SIGINT by raising KeyboardInterrupt).

Options for removing this annoyance:

just wrap your output routines in a try: … catch IOError:
flush stdout regularly … yuck … bad design
alter the SIGPIPE handler

import signal
signal.signal(signal.SIGPIPE, signal.SIG_DFL)

eg

$ python -c 'import signal; signal.signal(signal.SIGPIPE, signal.SIG_DFL)
...> for i in range(100000): print(i)' | head
0
1
2
3
4
5
6
7
8
9

sqlalchemy-bot · 2014-02-04T00:57:05Z

Michael Bayer (@zzzeek) wrote:

ill just catch the IOError. we just put that around the stdout writing routine and we're done, right ?

sqlalchemy-bot · 2014-02-04T01:38:55Z

Derek Harland (@donkopotamus) wrote:

yep

sqlalchemy-bot · 2014-03-10T04:53:22Z

Changes by Michael Bayer (@zzzeek):

added labels: command interface

sqlalchemy-bot · 2014-03-10T04:53:22Z

Changes by Michael Bayer (@zzzeek):

set milestone to "tier 1"

sqlalchemy-bot · 2014-03-10T23:26:11Z

Michael Bayer (@zzzeek) wrote:

yup, easy to replicate with a command added like this:

diff --git a/alembic/command.py b/alembic/command.py
index f1c5962..a5386ff 100644
--- a/alembic/command.py
+++ b/alembic/command.py
@@ -264,3 +264,8 @@ def splice(config, parent, child):
 
     """
     raise NotImplementedError()
+
+
+def simulate_crap(config):
+    for i in range(10000):
+        print i
\ No newline at end of file

#!

$ python -m alembic.config simulate_crap | head
0
1
2
3
4
5
6
7
8
9
Traceback (most recent call last):
  File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/runpy.py", line 162, in _run_module_as_main
    "__main__", fname, loader, pkg_name)
  File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/runpy.py", line 72, in _run_code
    exec code in run_globals
  File "/Users/classic/dev/alembic/alembic/config.py", line 301, in <module>
    main()
  File "/Users/classic/dev/alembic/alembic/config.py", line 298, in main
    CommandLine(prog=prog).main(argv=argv)
  File "/Users/classic/dev/alembic/alembic/config.py", line 293, in main
    self.run_cmd(cfg, options)
  File "/Users/classic/dev/alembic/alembic/config.py", line 279, in run_cmd
    **dict((k, getattr(options, k)) for k in kwarg)
  File "alembic/command.py", line 271, in simulate_crap
    print i
IOError: [Errno 32] Broken pipe

sqlalchemy-bot · 2014-03-10T23:40:42Z

Michael Bayer (@zzzeek) wrote:

it doesnt seem to be preventable in Python 3, it comes out as a warning in _io.TextIOWrapper.

sqlalchemy-bot · 2014-03-11T00:09:53Z

Michael Bayer (@zzzeek) wrote:

unless we use the signal approach. But, that means, when "head" finishes reading output, the program terminates. Which I don't think we can do. If someone runs a long migration through "head" we don't except the program to silently not complete. I've verified this is the effect.

sqlalchemy-bot · 2014-03-11T00:31:06Z

Michael Bayer (@zzzeek) wrote:

Suppressed IOErrors which can raise when program output pipe
is closed under a program like head; however this only
works on Python 2. On Python 3, there is not yet a known way to
suppress the BrokenPipeError warnings without prematurely terminating
the program via signals. fixes Broken pipe when using head to display only the first lines of history #160.
Added comments to http://bugs.python.org/issue11380 to see what the
status is on py3k.

→ f889070

sqlalchemy-bot · 2014-03-11T00:31:06Z

Changes by Michael Bayer (@zzzeek):

changed status to closed

sqlalchemy-bot · 2014-04-18T14:47:46Z

Florent Xicluna (@flox) wrote:

I tried to reproduce the issue on Python3 with the sample code on one of the previous comment. No success.
https://bitbucket.org/zzzeek/alembic/issue/160/broken-pipe-when-using-head-to-display#comment-8068721

On Debian with Python 3.2.5, 3.3.5 and 3.4.0

$ python3.2 -c 'for i in range(100000): print(i)' | head -2
0
1
Traceback (most recent call last):
  File "<string>", line 1, in <module>
IOError: [Errno 32] Broken pipe

$ python3.3 -c 'for i in range(100000): print(i)' | head -2                                           
0
1
Traceback (most recent call last):
  File "<string>", line 1, in <module>
BrokenPipeError: [Errno 32] Broken pipe

$ python3.4 -c 'for i in range(100000): print(i)' | head -2
0
1
Traceback (most recent call last):
  File "<string>", line 1, in <module>
BrokenPipeError: [Errno 32] Broken pipe

$

I never got the uncatchable message

Exception BrokenPipeError: BrokenPipeError(32, 'Broken pipe') in <_io.TextIOWrapper name='<stdout>' mode='w' encoding='UTF-8'> ignored

sqlalchemy-bot · 2014-04-18T14:56:45Z

Michael Bayer (@zzzeek) wrote:

script:

import sys

def simulate_crap(stream):
    for i in range(10000):
        stream.write("%d\n" % i)

try:
    simulate_crap(sys.stdout)
except IOError:
    pass

Py2K:

#!

classic$ python test.py | head
0
1
2
3
4
5
6
7
8
9
classics-MacBook-Pro-2:alembic classic$

Py3K:

#!

$ python3 test.py | head
0
1
2
3
4
5
6
7
8
9
Exception ignored in: <_io.TextIOWrapper name='<stdout>' mode='w' encoding='US-ASCII'>
BrokenPipeError: [Errno 32] Broken pipe

sqlalchemy-bot · 2014-04-18T15:08:17Z

Florent Xicluna (@flox) wrote:

I tried exactly the above, and i had the same output for Python 3.2.5, 3.3.5 and 3.4.0 : no message "Exception ignored" at the end.

So I guess this is something specific to OS X.
(or the minor version differs)

sqlalchemy-bot · 2014-04-18T17:02:57Z

Michael Bayer (@zzzeek) wrote:

seems like

sqlalchemy-bot · 2014-05-03T22:49:54Z

Changes by Michael Bayer (@zzzeek):

removed milestone (was: "tier 1")

sqlalchemy-bot · 2014-05-06T00:51:41Z

Derek Harland (@donkopotamus) wrote:

To suppress the BrokenPipe warning on OSX, what about the following:

import sys

def simulate_crap(stream):
    for i in range(10000):
        stream.write("%d\n" % i)

try:
    simulate_crap(sys.stdout)
except IOError:
    # assume that stdout has been closed at the other end rendering it unusable
    sys.stdout = None

With python 3.3 on OSX this will produce

$ python blah.py | head
0
1
2
3
4
5
6
7
8
9

sqlalchemy-bot · 2017-08-19T19:04:40Z

Frazer McLean (@RazerM) wrote:

Cam config.py be changed to the following, so that it is also fixed on Python 3? (from http://bugs.python.org/issue11380,#msg248579)

if __name__ == '__main__':
    try:
        main()
    finally:
        try:
            sys.stdout.flush()
        finally:
            try:
                sys.stdout.close()
            finally:
                try:
                    sys.stderr.flush()
                finally:
                    sys.stderr.close()

sqlalchemy-bot · 2017-08-20T23:00:17Z

Michael Bayer (@zzzeek) wrote:

@RazerM that workaround looks insane. does that prevent the warning we're getting in py3k? how is python itself not fixing that, it's awful? provide a pull request please where that would go? thanks

sqlalchemy-bot · 2017-08-26T17:38:21Z

Frazer McLean (@RazerM) wrote:

Yes, it prevents the warning. I'll provide a PR

sqlalchemy-bot · 2017-08-26T18:27:58Z

Michael Bayer (@zzzeek) wrote:

Ok. I'm going to try to work all the ugly into a context manager or something from your PR

sqlalchemy-bot · 2017-09-14T21:11:58Z

Michael Bayer (@zzzeek) wrote:

@RazerM can I please see the OS in use, Python version, stack trace, and commands to reproduce? The current approach of a simple IOError catch resolves the issue for me, and also note that rcollins' snippet on the Python bug is not an "official" workaround, it's conjectural for my specific case (but I cannot reproduce as long as IOError catch is there).

script:

import sys
from alembic.util.messaging import write_outstream


def simulate_crap(stream):
    for i in range(10000):
        write_outstream(stream, "%d\n" % i)

simulate_crap(sys.stdout)

running with both Python3.5, python3.6, piping into "head", "echo", etc., no error:

#!

[classic@photon2 alembic]$ .tox/py36/bin/python test.py  | head
0
1
2
3
4
5
6
7
8
9
[classic@photon2 alembic]$ .tox/py36/bin/python test.py  | head -20
0
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
[classic@photon2 alembic]$ .tox/py35/bin/python test.py  | echo

[classic@photon2 alembic]$ .tox/py35/bin/python test.py  | head
0
1
2
3
4
5
6
7
8
9

If I remove the existing IOError catch, error is immediate:

[classic@photon2 alembic]$ .tox/py35/bin/python test.py  | echo

Traceback (most recent call last):
  File "test.py", line 9, in <module>
    simulate_crap(sys.stdout)
  File "test.py", line 7, in simulate_crap
    write_outstream(stream, "%d\n" % i)
  File "/home/classic/dev/alembic/alembic/util/messaging.py", line 36, in write_outstream
    stream.write(t)
BrokenPipeError: [Errno 32] Broken pipe

Also tried with keyboard interrupts, no issue there either.

sqlalchemy-bot · 2017-09-14T22:59:26Z

Frazer McLean (@RazerM) wrote:

macOS 10.12.6

Using Python 3 installed by homebrew or pyenv I see this:

$ python3 test.py | echo

Exception ignored in: <_io.TextIOWrapper name='<stdout>' mode='w' encoding='UTF-8'>
BrokenPipeError: [Errno 32] Broken pipe
frazer@Atreus› ~/D/S/Python $

The reason is that the flush is being deferred until a __del__, and that's what prints the exception.

sqlalchemy-bot · 2017-09-15T14:12:24Z

Michael Bayer (@zzzeek) wrote:

so the main thing I want to figure out is if I can just put stdout.flush() in write_outstream and have it be local to each time we write things. can you try this patch?

diff --git a/alembic/util/messaging.py b/alembic/util/messaging.py
index c202e96..a944617 100644
--- a/alembic/util/messaging.py
+++ b/alembic/util/messaging.py
@@ -28,17 +28,20 @@ except (ImportError, IOError):
 
 def write_outstream(stream, *text):
     encoding = getattr(stream, 'encoding', 'ascii') or 'ascii'
-    for t in text:
-        if not isinstance(t, binary_type):
-            t = t.encode(encoding, 'replace')
-        t = t.decode(encoding)
+    try:
+        for t in text:
+            if not isinstance(t, binary_type):
+                t = t.encode(encoding, 'replace')
+            t = t.decode(encoding)
+            try:
+                stream.write(t)
+            except IOError:
+                break
+    finally:
         try:
-            stream.write(t)
+            stream.flush()
         except IOError:
-            # suppress "broken pipe" errors.
-            # no known way to handle this on Python 3 however
-            # as the exception is "ignored" (noisily) in TextIOWrapper.
-            break
+            pass
 
 
 def status(_statmsg, fn, *arg, **kw):

sqlalchemy-bot · 2017-09-15T21:27:02Z

Frazer McLean (@RazerM) wrote:

The patch doesn't fix it.

sqlalchemy-bot · 2017-09-15T22:51:43Z

Michael Bayer (@zzzeek) wrote:

but the stream is flushed every time and there's a catch around it. how?

sqlalchemy-bot · 2017-09-17T22:30:32Z

Derek Harland (@donkopotamus) wrote:

Just to repeat something from earlier in this thread. Does this not work?

import sys

def simulate_crap(stream):
    for i in range(10000):
        stream.write("%d\n" % i)

try:
    simulate_crap(sys.stdout)
except IOError:
    # assume that stdout has been closed at the other end rendering it unusable
    sys.stdout = None

With python 3.5 on OSX this produces:

$ python blah.py | head -2
0
1

sqlalchemy-bot · 2017-09-17T23:21:31Z

Michael Bayer (@zzzeek) wrote:

I'm mostly looking to make the workarounds here local to the inside of the write_outstream() method. putting a block around the "main()" of the program I find bothersome. Will do it if we really have to.

sqlalchemy-bot closed this as completed Mar 11, 2014

sqlalchemy-bot added command interface bug Something isn't working labels Nov 27, 2018

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Broken pipe when using `head` to display only the first lines of history #160

Broken pipe when using `head` to display only the first lines of history #160

sqlalchemy-bot commented Dec 18, 2013

sqlalchemy-bot commented Dec 18, 2013

sqlalchemy-bot commented Jan 18, 2014

sqlalchemy-bot commented Feb 4, 2014

sqlalchemy-bot commented Feb 4, 2014

sqlalchemy-bot commented Feb 4, 2014

sqlalchemy-bot commented Mar 10, 2014

sqlalchemy-bot commented Mar 10, 2014

sqlalchemy-bot commented Mar 10, 2014

sqlalchemy-bot commented Mar 10, 2014

sqlalchemy-bot commented Mar 11, 2014

sqlalchemy-bot commented Mar 11, 2014

sqlalchemy-bot commented Mar 11, 2014

sqlalchemy-bot commented Apr 18, 2014

sqlalchemy-bot commented Apr 18, 2014

sqlalchemy-bot commented Apr 18, 2014

sqlalchemy-bot commented Apr 18, 2014

sqlalchemy-bot commented May 3, 2014

sqlalchemy-bot commented May 6, 2014

sqlalchemy-bot commented Aug 19, 2017

sqlalchemy-bot commented Aug 20, 2017

sqlalchemy-bot commented Aug 26, 2017

sqlalchemy-bot commented Aug 26, 2017

sqlalchemy-bot commented Sep 14, 2017

sqlalchemy-bot commented Sep 14, 2017

sqlalchemy-bot commented Sep 15, 2017

sqlalchemy-bot commented Sep 15, 2017

sqlalchemy-bot commented Sep 15, 2017

sqlalchemy-bot commented Sep 17, 2017

sqlalchemy-bot commented Sep 17, 2017

Broken pipe when using head to display only the first lines of history #160

Broken pipe when using head to display only the first lines of history #160

Comments

sqlalchemy-bot commented Dec 18, 2013

sqlalchemy-bot commented Dec 18, 2013

sqlalchemy-bot commented Jan 18, 2014

sqlalchemy-bot commented Feb 4, 2014

sqlalchemy-bot commented Feb 4, 2014

sqlalchemy-bot commented Feb 4, 2014

sqlalchemy-bot commented Mar 10, 2014

sqlalchemy-bot commented Mar 10, 2014

sqlalchemy-bot commented Mar 10, 2014

sqlalchemy-bot commented Mar 10, 2014

sqlalchemy-bot commented Mar 11, 2014

sqlalchemy-bot commented Mar 11, 2014

sqlalchemy-bot commented Mar 11, 2014

sqlalchemy-bot commented Apr 18, 2014

sqlalchemy-bot commented Apr 18, 2014

sqlalchemy-bot commented Apr 18, 2014

sqlalchemy-bot commented Apr 18, 2014

sqlalchemy-bot commented May 3, 2014

sqlalchemy-bot commented May 6, 2014

sqlalchemy-bot commented Aug 19, 2017

sqlalchemy-bot commented Aug 20, 2017

sqlalchemy-bot commented Aug 26, 2017

sqlalchemy-bot commented Aug 26, 2017

sqlalchemy-bot commented Sep 14, 2017

sqlalchemy-bot commented Sep 14, 2017

sqlalchemy-bot commented Sep 15, 2017

sqlalchemy-bot commented Sep 15, 2017

sqlalchemy-bot commented Sep 15, 2017

sqlalchemy-bot commented Sep 17, 2017

sqlalchemy-bot commented Sep 17, 2017

Broken pipe when using `head` to display only the first lines of history #160

Broken pipe when using `head` to display only the first lines of history #160