subprocess: Calling Popen.communicate() after Popen.stdout.read() returns an empty string

[FrostMing]

BPO	41406
Nosy	@gpshead, @vstinner, @gst, @frostming

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = None
created_at = <Date 2020-07-27.08:26:24.981>
labels = ['type-bug', 'library', '3.10']
title = 'subprocess: Calling Popen.communicate() after Popen.stdout.read() returns an empty string'
updated_at = <Date 2020-08-04.01:55:39.697>
user = 'https://github.com/frostming'

bugs.python.org fields:

activity = <Date 2020-08-04.01:55:39.697>
actor = 'gregory.p.smith'
assignee = 'none'
closed = False
closed_date = None
closer = None
components = ['Library (Lib)']
creation = <Date 2020-07-27.08:26:24.981>
creator = 'Frost Ming'
dependencies = []
files = []
hgrepos = []
issue_num = 41406
keywords = []
message_count = 4.0
messages = ['374366', '374436', '374769', '374783']
nosy_count = 4.0
nosy_names = ['gregory.p.smith', 'vstinner', 'gstarck', 'Frost Ming']
pr_nums = []
priority = 'normal'
resolution = None
stage = None
status = 'open'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue41406'
versions = ['Python 3.10']

[FrostMing]

The following snippet behaves differently between Windows and POSIX.

import subprocess
import time


p = subprocess.Popen("ls -l", shell=True, stdout=subprocess.PIPE, stderr=subprocess.PIPE)

print(p.stdout.read(1))   # read 1 byte
print(p.communicate())    # Returns empty output

It works fine on Windows and Python 2.x(communicate() returning the remaining output). So from the best guess it should be the expected behavior.

The reason behind this is that Popen.stdout is a BufferedReader. It stores all output in the buffer when calling read(). However, communicate() and the lower API _communicate() use a lower level method os.read() to get the output, which does not respect the underlying buffer. When an empty output is retrieved the file object is closed then.

First time to submit a bug report and pardon me if I am getting anything wrong.

[gst]

also affecting 3.6

[vstinner]

Calling proc.communicate() after proc.stdout.read() doesn't seem to be supported. What is your use case? Why not just calling communicate()? Why not only using stdout directly?