Document how to make pty.spawn not copy data

[RadicalZephyr]

BPO	22865
Nosy	@Yhg1s, @vstinner, @vadmium, @RadicalZephyr, @csabella
PRs	bpo-22865: Expand on documentation for the pty.spawn function #11980 [3.7] bpo-22865: Expand on documentation for the pty.spawn function (GH-11980) #13455
Files	pty.patch pty.patch: Updated patch pty.patch: Documentation update

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2019-05-21.09:38:34.948>
created_at = <Date 2014-11-13.18:01:26.047>
labels = ['3.8', 'type-feature', '3.7', 'docs']
title = 'Document how to make pty.spawn not copy data'
updated_at = <Date 2019-05-21.09:38:34.947>
user = 'https://github.com/RadicalZephyr'

bugs.python.org fields:

activity = <Date 2019-05-21.09:38:34.947>
actor = 'vstinner'
assignee = 'docs@python'
closed = True
closed_date = <Date 2019-05-21.09:38:34.948>
closer = 'vstinner'
components = ['Documentation']
creation = <Date 2014-11-13.18:01:26.047>
creator = 'RadicalZephyr'
dependencies = []
files = ['37189', '37620', '39313']
hgrepos = []
issue_num = 22865
keywords = ['patch']
message_count = 15.0
messages = ['231126', '233561', '242730', '244956', '252444', '252446', '252453', '336171', '336246', '336257', '336268', '342871', '342910', '342993', '342994']
nosy_count = 6.0
nosy_names = ['twouters', 'vstinner', 'docs@python', 'martin.panter', 'RadicalZephyr', 'cheryl.sabella']
pr_nums = ['11980', '13455']
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'enhancement'
url = 'https://bugs.python.org/issue22865'
versions = ['Python 3.7', 'Python 3.8']

[RadicalZephyr]

While using the pty.py module I had cause to want to be able to silently eat characters from the input stream. I discovered that because of the check for end of file being "if not data" that there was no way to do this.

Since the default function used by spawn and _copy is a thin wrapper on os.read, I figured that it would be useful to interpret None as a sentinel value meaning "this is not EOF, but there's nothing I actually want to copy over."

[RadicalZephyr]

I tweaked the patch a bit to not include the parentheses since that seems to be the style here.

[RadicalZephyr]

Okay, I just found another way to achieve the same effect of letting the _read function ignore data but not inadvertantly close the stream. It relies on the fact that terminals will ignore null bytes fed to them.

Now there are no code changes required, just an addition to the documentation.

[RadicalZephyr]

Hey, pinging this issue. Hoping someone has time to take a look at it.

[vadmium]

If you return null bytes, they will be written to the parent process’s stdout file descriptor, rather than being ignored. I do not think it is possible to ignore output with the spawn() API, unless perhaps you previously set up stdout write to /dev/null or similar.

Do you have a reference for your fact about terminals ignoring null input bytes? I’m skeptical because a basic serial terminal can send arbitrary data including nulls, but I am not so familiar with pseudo-terminals.

However the bit about interpreting an empty string could be useful. I would drop the “or any falsey value”; is that a typo? I suspect that the functions have to return byte strings (not text). It would be nice to say what the consequences of the EOF condition is, e.g. I can see for the child’s output it would stop copying, but not actually close the parent’s stdout to signal EOF. What about in the other direction? Can you signal multiple EOFs with data in between?

[RadicalZephyr]

Hmm, I spoke improperly. I think you are entirely correct in your statements. What I meant by "terminals ignore null bytes" is that returning a string consisting of only a null byte doesn't cause anything observable to happen, including anything to be written to the screen. But I have no idea why this is.

My basis for this is limited to empirical observation of how my terminal behaves (I use the iTerm2 terminal emulator on Mac OS X), and I don't recall whether I checked it with other terminals like the regular terminal on Mac and terminal emulators on Linux.

Re: "or any falsey value". In general, the read functions should return byte strings yes, but since the check for EOF in python is whether a stream returned an empty string, and the way that pty.spawn() currently checks for that is if not data then any falsey value will cause pty.spawn to think that the underlying fd was closed.

I agree that the documentation on this could use more fleshing out in general, especially regarding what happens when certain values are returned. I'm not very familiar with this code any more, but I might have time to play with it and try to add some more information in the coming weeks.

[vadmium]

I would avoid suggesting one can return non-byte-strings; treat that as an implementation detail. Maybe something like:

The functions *master_read* and *stdin_read* are passed a file descriptor which they should read from, and should return a byte string. The defaults try to read 1024 bytes each time they are called. Returning an empty string is be interpreted as an EOF condition, and the *_read* function will no longer be called.

[csabella]

@RadicalZephyr, are you interested in making a GitHub pull request with the recommended the documentation change?

[RadicalZephyr]

It is submitted @cheryl.sabella. Thanks for reviving this, I had totally lost track of it.

[csabella]

Thanks @RadicalZephyr!

@martin.panter, please review the PR when you get a chance. Thank you!

[vadmium]

I'm not sure it is wise for the Python documentation to suggest inserting null bytes in general. This seems more like an application-specific hack. There is nothing in Python that handles these null bytes specially, and I expect they will be seen if the child reads the terminal in raw mode, or if the parent's output is redirected to a file, sent over the network or a real serial link.

[RadicalZephyr]

@martin.panter I removed the mention of inserting null bytes and restricted the documentation updates to more fully documenting the current behaviour.

[vstinner]

New changeset 522ccef by Victor Stinner (Geoff Shannon) in branch 'master':
bpo-22865: Expand on documentation for the pty.spawn function (GH-11980)
522ccef

[vstinner]

New changeset cdb2dbf by Victor Stinner (Geoff Shannon) in branch '3.7':
[3.7] bpo-22865: Expand on documentation for the pty.spawn function (GH-11980) (GH-13455)
cdb2dbf

[vstinner]

Thanks Geoff Shannon for the doc enhancement!