Idle: use pipes instead of sockets to talk with user subprocess

[terryjreedy]

BPO	18823
Nosy	@terryjreedy, @ezio-melotti, @asvetlov, @serhiy-storchaka
Files	Carlson.py

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = None
created_at = <Date 2013-08-23.23:35:04.085>
labels = ['expert-IDLE', 'type-bug', '3.10']
title = 'Idle: use pipes instead of sockets to talk with user subprocess'
updated_at = <Date 2020-06-06.16:26:52.685>
user = 'https://github.com/terryjreedy'

bugs.python.org fields:

activity = <Date 2020-06-06.16:26:52.685>
actor = 'terry.reedy'
assignee = 'none'
closed = False
closed_date = None
closer = None
components = ['IDLE']
creation = <Date 2013-08-23.23:35:04.085>
creator = 'terry.reedy'
dependencies = []
files = ['34654']
hgrepos = []
issue_num = 18823
keywords = []
message_count = 6.0
messages = ['196055', '215062', '215069', '215123', '336154', '370832']
nosy_count = 5.0
nosy_names = ['terry.reedy', 'ezio.melotti', 'asvetlov', 'tshepang', 'serhiy.storchaka']
pr_nums = []
priority = 'normal'
resolution = None
stage = 'needs patch'
status = 'open'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue18823'
versions = ['Python 3.10']

[terryjreedy]

Idle once used os.spawnv to open the user subprocess and 2.7 still does. Sockets were then used to interact with the subprocess. Since rev71746, 2011-8-3, 3.2+ use subprocess.Popen but still use sockets. A rare but continual problem is failure to make the socket connection.

In bpo-16123, Amaury Forgeot d'Arc suggested "pipes?" The idea resurfaced again in current Idle-sig thread "Idle does not open on mac" (because socket connection timed out). Guido, who designed the current 2.x system, chimed in with "I would recommend trying to use the subprocess module."

A current python-list thread indicates that there can be problems with
"Running a command line program and reading the result as it runs"
http://mail.python.org/pipermail/python-list/2013-August/654265.html

In particular, "for line in p:" gets lines delayed (buffered) while
Peter Otten discovered that "for line in iter(p.stdout.readline, ''):" gets them as they are produced.
http://mail.python.org/pipermail/python-list/2013-August/654330.html

We first need to experiment running a simple echo server with python(w).exe. I have not succeeded yet on Windows.

[terryjreedy]

In the thread that started here
https://mail.python.org/pipermail/python-dev/2014-March/133492.html
in this message.
https://mail.python.org/pipermail/python-dev/2014-March/133671.html
Josiah Carlson posted a pastebin link about how to actually use the pipes. Guido recommended that the he post a revised version to a new issue, as a patch to subprocess, but I am uploading it here to make sure it does not get lost.

[terryjreedy]

https://mail.python.org/pipermail/python-dev/2014-March/133641.html
from Victor Stinner gives example code using asyncio

[terryjreedy]

Pydev, Re: Status of PEP-3145 - Asynchronous I/O for subprocess.popen; Antoine Pitrou:

Why don't you use multiprocessing or concurrent.futures? They have
everything you need for continuous conversation between processes.

[terryjreedy]

bpo-34313 reports problems with multiprocessing when using at least some versions of tkinter on some versions of macOS.

[terryjreedy]

Problems with multiprocessing and tkinter on Mac might have been due to using the default start method, which used to be fork and is now spawn. This should be retested sometime.

[arhadthedev]

Problems with multiprocessing and tkinter on Mac might have been due to using the default start method, which used to be fork and is now spawn.

• multiprocessing's default posix start method of 'fork' is broken: change to `'forkserver' || 'spawn' #84559.