io.open('/dev/stdout', 'a') raises OSError with errno=ESPIPE

[hathawsh]

BPO	27805
Nosy	@smontanaro, @hathawsh, @vstinner, @benjaminp, @bitdancer, @vadmium, @ztane, @ZackerySpytz, @javabrett
PRs	bpo-27805: Allow opening /dev/std{out,err} in append mode #13254 closes bpo-27805: Ignore ESPIPE in initializing seek of append-mode files. #17112 [3.8] closes bpo-27805: Ignore ESPIPE in initializing seek of append-mode files. (GH-17112) #17136 [3.7] closes bpo-27805: Ignore ESPIPE in initializing seek of append-mode files. (GH-17112) #17137

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2019-11-12.22:51:45.122>
created_at = <Date 2016-08-19.20:11:12.895>
labels = ['3.8', 'type-bug', '3.7', 'expert-IO']
title = "io.open('/dev/stdout', 'a') raises OSError with errno=ESPIPE"
updated_at = <Date 2019-11-12.23:54:27.193>
user = 'https://github.com/hathawsh'

bugs.python.org fields:

activity = <Date 2019-11-12.23:54:27.193>
actor = 'benjamin.peterson'
assignee = 'none'
closed = True
closed_date = <Date 2019-11-12.22:51:45.122>
closer = 'benjamin.peterson'
components = ['IO']
creation = <Date 2016-08-19.20:11:12.895>
creator = 'hathawsh'
dependencies = []
files = []
hgrepos = []
issue_num = 27805
keywords = ['patch']
message_count = 15.0
messages = ['273158', '273169', '273171', '273172', '273204', '273207', '273223', '273225', '273262', '273274', '315005', '353343', '356503', '356506', '356508']
nosy_count = 9.0
nosy_names = ['skip.montanaro', 'hathawsh', 'vstinner', 'benjamin.peterson', 'r.david.murray', 'martin.panter', 'ztane', 'ZackerySpytz', 'javabrett']
pr_nums = ['13254', '17112', '17136', '17137']
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue27805'
versions = ['Python 2.7', 'Python 3.7', 'Python 3.8']

[hathawsh]

With Python 2, the following call worked:

open('/dev/stdout', 'a')

Users of Supervisor have been depending on that for a long time.

With Python 3.5, this is what happens:

>>> open('/dev/stdout', 'a')
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
OSError: [Errno 29] Illegal seek

Presumably, this happens because Python 3 opens the file in 'w' mode and seeks to the end, while Python 2 passed the 'a' flag directly to the underlying C library; the underlying library is apparently smart enough to treat 'a' and 'w' identically when opening character device files and FIFOs.

It's a nasty little surprise for those upgrading from Python 2.

[vadmium]

The origin of this seems to be r68835 and bpo-5008. Victor mentioned imitating the Gnu C library. Maybe there is a better way that also supports non-seekable files better, perhaps handle ESPIPE without failing.

This also affects Python 2, if you consider io.open() or io.FileIO directly.

[vstinner]

Syscalls made by open("/dev/stdout", "a") in Python 2:

open("/dev/stdout", O_WRONLY|O_CREAT|O_APPEND, 0666) = 3
lseek(3, 0, SEEK_END) = -1 ESPIPE (Illegal seek)
fstat(3, {st_mode=S_IFCHR|0620, st_rdev=makedev(136, 0), ...}) = 0

I used this comand:

  $ strace -o trace python2 -c 'import os; os.uname();os.uname();os.uname(); f=open("/dev/stdout", "a"); os.uname(); f.close()'

It looks like the C library simply ignores ESPIPE on lseek(fd, 0, SEEK_END) when opening a file in append mode:
https://sourceware.org/git/?p=glibc.git;a=blob;f=libio/fileops.c;h=13157354272ff9ab1832d4a619a81f05898fcd69;hb=HEAD#l242

  if ((read_write & _IO_IS_APPENDING) && (read_write & _IO_NO_READS))
    if (_IO_SEEKOFF (fp, (_IO_off64_t)0, _IO_seek_end, _IOS_INPUT|_IOS_OUTPUT)
	== _IO_pos_BAD && errno != ESPIPE)
      {
	close_not_cancel (fdesc);
	return NULL;
      }

from _IO_file_open() file operation.

[vstinner]

Users of Supervisor have been depending on that for a long time.

Hum, the workaround is very simple no? Just open /dev/stdout with mode "w", no?

[ztane]

Presumably the case was that a *named* log file is opened with 'a' mode, and one could pass '/dev/stdout' just like any other name of a file, and it did work, but not in Python 3.5.

[vstinner]

Antti Haapala added the comment:

Presumably the case was that a *named* log file is opened with 'a' mode, and one could pass '/dev/stdout' just like any other name of a file, and it did work, but not in Python 3.5.

Oh ok, in this case, you need something smarter like:

mode = 'a' if not stat.S_ISCHR(os.stat(filename).st_mode) else 'w'
fp = open(filename, mode)

or something like (emulate append mode, but catch ESPIPE):

fp = open(filename, 'w')
try:
  fp.seek(0, os.SEEK_END)
except OSError as exc:
  if exc.errno != errno.ESPIPE: raise

[hathawsh]

Thanks for the analysis. I have already started a pull request to fix this in Supervisor, but I also thought this change to Python might be gratuitous and accidental. It seems like open('/dev/stdout', 'a') ought to work the same as Python 2. If not, the Python documentation should warn people that using 'a' with character devices and FIFOs will cause an OSError.

[bitdancer]

And the fact that python deviates from posix in this regard seems like a bug to me, since the posix behavior is, as noted, very useful. Can't we handle ESPIPE like the C library does, instead of forcing users to learn that arcane incantation?

[vadmium]

Handling ESPIPE for append mode seems reasonable to me, even as a bug fix for existing versions.

But there is a similar problem with "r+" and "w+" modes and unseekable files (unless buffering=0). See bpo-20074. So we can’t say in general that Python 3 faithfully implements all aspects of Python 2’s / C’s / Posix’s file modes.

[ztane]

Yeah, it definitely is a bug in CPython. open(mode='a') should always append to the end of the given file.

If you're writing an append-only text log to some file-like object, that's the mode you use, not some version/platform/filesystem specific voodoo to find out what's the least incorrect way to work around Python implementation deficiencies.

[smontanaro]

I was bitten by this porting a system from Python 2.7 to 3.6. "/dev/stderr" is a very nice default for logfiles. Users will frequently override the default, so you really want to open the logfile in append mode. Having to jump through hoops to avoid blasting a user's logfile is kinda dumb, and as others have pointed out, error-prone.

os.open works just fine with O_WRONLY|O_APPEND as the flags. /dev/null can be opened in append mode, but not /dev/stderr or /dev/stdout.

[benjaminp]

If we were starting from zero, I would suggest omitting the lseek() after open. Sure .tell() might report 0 on newly opened files, but avoiding unnecessary io operations is nicer.

[benjaminp]

New changeset 74fa9f7 by Benjamin Peterson in branch 'master':
closes bpo-27805: Ignore ESPIPE in initializing seek of append-mode files. (GH-17112)
74fa9f7

[benjaminp]

New changeset b8b3e43 by Benjamin Peterson in branch '3.7':
[3.7] closes bpo-27805: Ignore ESPIPE in initializing seek of append-mode files. (GH-17137)
b8b3e43

[benjaminp]

New changeset 9788f97 by Benjamin Peterson in branch '3.8':
[3.8] closes bpo-27805: Ignore ESPIPE in initializing seek of append-mode files. (GH-17136)
9788f97