allow running multiple instances of IDLE

[jshute]

BPO	1201569
Nosy	@kbkaiser, @taleinat
Superseder	bpo-1529142: Allowing multiple instances of IDLE with sub-processes
Files	multiport.py: patch

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = 'https://github.com/kbkaiser'
closed_at = <Date 2009-03-30.17:47:04.194>
created_at = <Date 2005-05-13.18:03:15.000>
labels = ['expert-IDLE', 'type-feature']
title = 'allow running multiple instances of IDLE'
updated_at = <Date 2009-03-30.17:47:04.193>
user = 'https://bugs.python.org/jshute'

bugs.python.org fields:

activity = <Date 2009-03-30.17:47:04.193>
actor = 'kbk'
assignee = 'kbk'
closed = True
closed_date = <Date 2009-03-30.17:47:04.194>
closer = 'kbk'
components = ['IDLE']
creation = <Date 2005-05-13.18:03:15.000>
creator = 'jshute'
dependencies = []
files = ['6661']
hgrepos = []
issue_num = 1201569
keywords = ['patch']
message_count = 12.0
messages = ['48342', '48343', '48344', '48345', '48346', '48347', '48348', '48349', '48350', '48351', '48352', '83737']
nosy_count = 3.0
nosy_names = ['kbk', 'jshute', 'taleinat']
pr_nums = []
priority = 'low'
resolution = 'later'
stage = 'test needed'
status = 'closed'
superseder = '1529142'
type = 'enhancement'
url = 'https://bugs.python.org/issue1201569'
versions = ['Python 3.1', 'Python 2.7']

[jshute]

This patch will allow running multiple instances of
IDLE in the default (mutli-process) mode by choosing
the first available port from 8833-8893 instead of
failing when another IDLE has already bound port 8833.

(Not tested on windows.)

[kbkaiser]

Logged In: YES
user_id=149084

Duplicate of IDLEfork 661363

Will not be considered until remaining problems with socket
binding on Windows have been resolved.

Note: You can start as many copies of IDLE as you want with
the -n switch. What is your use case?

Also note:
# spawning first avoids passing a listening socket to the subprocess

[jshute]

Logged In: YES
user_id=39615

My use case is just that I find it annoying that in the
default mode, you can't run multiple instances, and I kept
seeing the annoying dialog box and having to restart. I
normally use -n mode partly because of this, but sometimes I
forget. It seemed like a trivial fix.

Regarding the note, I didn't understand the comment because
it doesn't say why. I thought maybe it was done that way so
that it could be made to fail so the error messages could be
tested. It seemed like you avoid the sleep() and several
race conditions by starting the listener first.

[taleinat]

Logged In: YES
user_id=1330769

First of all, there is (theoretically) no reason to un IDLE
-without- a subprocess. The subprocess is an awesome feature
and I feel that it should be the (good, usable, reliable)
default.

IDLE's "Run Module" option is very handy but its behavior
depends on whether or not there is a subprocess. If there is
a subprocess it restarts the interpreter every time, which
is good! If there is no subprocess it runs the module the
first time, and later does nothing (because of Python's
import-once mechanism).

On windows the Python installer adds "Edit with IDLE" to
Explorer's right-click context menu, which is VERY
convenient. But this shortcut start IDLE with the -n
precisely because of this subprocess issue. So if I want to
be able to use Run Module and have the interpreter restart,
I have to manually open the file for editting. Yes, it's
something I can live with, but it's a serious turn-off for
new users.

I often teach Python using IDLE. Having IDLE sometimes run
with a subprocess and sometimes without confuses new users,
so I end up having to explain the subprocess issue very
early, and they still get bitten by it a few times. And they
also have to manually open files instead of using the "Edit
with IDLE" shortcut.

Allowing multiple instances, each with a subprocess
listening on a different port, seems like the cleanest
solution. Whatever socket issues there are, let's get them
cleared up!

P.S. I've been using this patch for some time, it works
pretty well but not 100%.

[kbkaiser]

Logged In: YES
user_id=149084

There are a couple of reasons: some systems don't
have networking installed, and running w/o the
subprocess causes the debugger to work slightly
differently: it is possible to debug IDLE itself. It
is surprising that it was possible to run IDLE in
either mode w/o mucking up the code too badly.

But there is one major problem: on Windows, it's
possible to start more than one listener on a port.
I haven't been able to track this down, it's somewhat
erratic. Given the problems on Windows with stuck
subprocesses, I'm loath to go to multiple instances
of IDLE w/o fixing the multiple listener problem with
its collisions.

I certainly agree with you that it would be very
desireable not to use the -n switch in the "edit with
IDLE" binding. And I also agree about the efficacy of
the subprocess, that's why we went to all the effort
on IDLEfork.

https://sourceforge.net/tracker/
index.php?func=detail&aid=661363&group_id=9579&atid=309579

Maybe we can get this fixed in 2.6. It's way too
risky to attempt it in 2.5 and it's a new feature.

Could you add your patch so I can look at it?

[taleinat]

Logged In: YES
user_id=1330769

I agree that IDLE must have a no-subprocess mode, since it
has its uses. Still, I feel these uses are relatively
isoteric, so while the option should exist as a command line
option, it shouldn't usually be used by non-developers.

On systems without networking trying to use sockets will
fail, and IDLE should detect this and fall-back to
no-subprocess mode automatically. (Another item on the TODO
list...)

Well, we could "dodge" the Windows problem for now by having
IDLE try a random port every time (like in the IDLEfork
patch) - thus collisions will be kept to a minimum. If we
choose among over 10,000 ports, most users will never
encounter a port collision. And even when a collision does
happen, it will probably be detected and properly dealt with

• collision non-detection errors are, as you say, erratic.

Also, if a collision happens the listening socket won't
recieve a connection, and accept() will timeout. After this
we can continue trying other ports.

I submitted my implementation as a separate patch.

[kbkaiser]

Logged In: YES
user_id=149084

On Windows 2000, at least, when a collision occurs
there is a significant (30%?) chance that the late
arrival will get a listening connection. Then both
sets of processes merrily communicate on the same
socket, apparently w/o mutual corruption. At least
that's the way it seemed to me last time I looked.
I can't put much faith in a situation like that.

However, your patch 1529142, though a hack, may be
practical, at least for now. 2.6.

[taleinat]

Logged In: YES
user_id=1330769

My patch is not so much a hack, it's actually a more optimal
implementation, with or without the Windows issue. It just
also happens to make port collision very rare ;) Aren't
coincidences wonderful?

</sleepy attempt at humor>

Anyways, I remember seeing something about "reuse address"
when binding sockets... ah, here it is. Check out
SocketServer.py:

if self.allow_reuse_address:
    self.socket.setsockopt(socket.SOL_SOCKET,
                           socket.SO_REUSEADDR, 1)
    self.socket.bind(self.server_address)

And this comment in BaseHTTPServer.py:

allow_reuse_address = 1 # Seems to make sense in testing
environment

Could socket.SO_REUSEADDR be the cause? What does it default
to? Maybe its implementation on Windows is broken? (Probably
not, but still worth checking.)

Maybe we should get the discussion up and running again in
Python-Dev or c.l.p...

[kbkaiser]

Logged In: YES
user_id=149084

I don't mean to be derogatory by calling it a
hack! Hacks can be very practical and useful, but
they are often not 100% reliable. I'm only referring
to the port selection part of the patch.

Regarding REUSEADDR, if it is not set True there is
a timeout before the listening connection can be set
up again on the same socket.

As I recollect, it's ten sec, and so very annoying
when restarting IDLE especially during development.

Guido clued me in on REUSEADDR, which solved the
problem. According to the below links, it's only
operative when the socket is in the TIME_WAIT state,
and that should not be the case in the W2K overlapping
socket situation.

http://www.unixguide.net/network/socketfaq/4.5.shtml
http://www.unixguide.net/network/socketfaq/2.7.shtml

IDLE discussion should be on IDLE-dev unless it's
something which affects the Python implementation or
changes the installation, i.e. how .py is bound on
Windows.

[kbkaiser]

Logged In: YES
user_id=149084

Timeout might have been a whole minute.

[ronaldoussoren]

Logged In: YES
user_id=580910

Timeout with REUSADDR tends to be several minutes.

<http://msdn.microsoft.com/library/default.asp?url=/library/en-us/
winsock/winsock/using_so_reuseaddr_and_so_exclusiveaddruse.asp> seems
to indicate that SO_REUSEADDR behaves unlike what unix does: two
applications can bind to the same port if one of them uses SO_REUSEADDR.

Picking the first available port and disabling SO_REUSEADDR should therefore
both enable multi-process mode and solve the delay issue kbk is having
during debugging.

[taleinat]

This should be dropped in favor of issue bpo-1529142, which proposes a
simpler and better solution.