"Restart Shell" command leaves pythonw.exe processes running

[PeterCaven]

BPO	12540
Nosy	@loewis, @birkenfeld, @terryjreedy, @kbkaiser, @tjguk, @ned-deily, @briancurtin
Files	issue12540.1.patch issue12540_rev2.patch unnamed

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = 'https://github.com/ned-deily'
closed_at = <Date 2011-08-05.06:43:12.172>
created_at = <Date 2011-07-12.09:21:40.622>
labels = ['expert-IDLE', 'performance']
title = '"Restart Shell" command leaves pythonw.exe processes running'
updated_at = <Date 2011-08-09.08:17:30.823>
user = 'https://bugs.python.org/PeterCaven'

bugs.python.org fields:

activity = <Date 2011-08-09.08:17:30.823>
actor = 'Aneesh'
assignee = 'ned.deily'
closed = True
closed_date = <Date 2011-08-05.06:43:12.172>
closer = 'ned.deily'
components = ['IDLE']
creation = <Date 2011-07-12.09:21:40.622>
creator = 'Peter.Caven'
dependencies = []
files = ['22799', '22830', '22837']
hgrepos = []
issue_num = 12540
keywords = ['patch', '3.2regression']
message_count = 36.0
messages = ['140179', '140474', '140486', '140487', '140555', '140773', '140775', '140786', '140872', '140896', '140900', '140933', '140938', '140948', '140954', '140965', '140968', '140970', '140971', '140973', '140974', '140975', '141010', '141037', '141060', '141330', '141344', '141420', '141422', '141595', '141611', '141647', '141648', '141656', '141672', '141810']
nosy_count = 12.0
nosy_names = ['loewis', 'georg.brandl', 'terry.reedy', 'kbk', 'tim.golden', 'ned.deily', 'eli.bendersky', 'brian.curtin', 'sunqiang', 'python-dev', 'Peter.Caven', 'Aneesh']
pr_nums = []
priority = 'high'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'resource usage'
url = 'https://bugs.python.org/issue12540'
versions = ['Python 3.2', 'Python 3.3']

[PeterCaven]

On Windows Vista (x64) the IDLE "Restart Shell" command leaves a "pythonw.exe" process running each time that the command is used.
Observed in Python 3.2.1 release and RC2.

[terryjreedy]

I just 'upgraded' to 3.2.1 on my XP machine and I see the same with F5-run, which restarts before running the saved file. This appears to be a nasty regression from 3.2.0 that should have been a release blocker if caught earlier. I believe it merits a quick re-release, once fixed, of either the Windows binary or Python itself depending on whether this is windows specific or not.

The normal behavior when starting IDLE is 2 pythonw.exe processes -- one to run IDLE itself and the other for the attached process that runs user code. Restart/Run starts a third process with a fresh interpreter for user code and the old, orphaned user process should and usually does disappear in 2-3 seconds (on my old, slow machine). (There are tracker issues about this not always happening when a runaway process is stopped with ^C, but is has always worked otherwise.) Each process appears to take about 10+Mb, so anyone doing rapid code/test iterations, as I sometimes do, could easily overfill physical memory.
Closing the IDLE windows does not stop the detached processes, so they have to be killed 1 at a time with 3 clicks each or by rebooting. Neither is pleasant.

Although I should have 3.2.1 loaded at least for reviewing issues, I plan to revert to 3.2.0.

Victor: do you know of any way to test for process extinction on Windows?

[loewis]

Terry, please don't overreact. Nobody has noticed it during the *long* rc period of 3.2.1, so it can't be that bad. Actually, I *did* notice, but didn't have the time to submit a bug report.

[loewis]

FWIW, it only happens with IDLE; python.exe seems to terminate fine when done.

[ned-deily]

This appears to be a Windows-only issue; I'm not able to readily reproduce it on either Linux or OS X. Taking a quick look at diffs between 3.2 and 3.2.1, there aren't a lot of changes in IDLE (Lib/idlelib) and nothing obviously related. There are a number of changes elsewhere in signal handling and process handling, though. The restart_subprocess code is in Lib/idlelib/PyShell.py. If there isn't anything obvious elsewhere, perhaps someone can try to hg bisect it on Windows.

[terryjreedy]

This nasty bug really can cause severe problems. If a zombie process ran a tkinter (tk) window, then attempting to logout/restart/shutdown eventually brings up a window I have never seen before: End Process -- EmbeddedMenuWindow. The message window shows a countdown timer that implies that it will shutdown the process automatically when the timer reaches 0. But it does not because the process does not respond. It will sit there indefinitely. One has to click a shutdown button to get rid of it. Then a few seconds later, another window for the next zombie appears. And so on. It there are 50 zombies, then one would have to repeat 50 times. Not acceptible to me.

Since IDLE is my Python workspace and since I plan to soon start running lots of tkinter tests and experiments soon, I am reverting to 3.2.0 until there is a fixed Windows binary.

Adding the two listed Windows experts.

[terryjreedy]

Restart is not required to create a zombie. Just start IDLE and quit, and the second, user process does not disappear.

Reverting completely does not seem possible. I first just ran the 3.2 installer and it did not complain, that I noticed, about replacing a newer version, but apparently only replaced .exe and .dll and a few other files but not Lib/*.py. Idle or command prompt window said it was still 3.2.1 and the bug remained.

So I uninstalled, deleted everything left except my .py files in site-packages and another subdirectory, and reinstalled 3.2. Now everything seems to predate 2/22/2011. BUT IDLE and command prompt window *still* report "Python 3.2.1 (default, Jul 10 2011, 21:51:15) [MSC v.1500 32 bit (Intel)] on win32". This is sys.version. Something somewhere (registry?) seems to not be deleted by uninstall. And the bug remains. Could a registry entry possibly affect this?

My system is 7 years old with updated win xp 32 bit.

I then checked the never updated 64 bit Py 3.2 on an 18 month old 64 bit Win 7 laptop and detached user processes *do* disappear as I remember on this machine. It did, however, take 8 sec over restart and 12 after closing, which is longer than I remember for my older and definitely slower machine.

[Aneesh]

We are also noticed same issue and reverted to Python 3.1. Whenever we run a script, two new pythonw.exe process is started and it is really irritating to see all in Windows Task Manager.

Last day I killed around 14 Pythonw.exe to clean up everything.

[elibendersky]

Is the problem happening only on 64-bit Windows, or 32-bit as well?

[sunqiang]

I can reproduce this in 32-bit Windows XP Pro. SP3

[terryjreedy]

Good question.
Peter, you said Vista x64. Are you running 32 or 64 bit Python?

My system with the apparently irreversible problem is 32 bit xp home.
I am reluctant to test on my daughter's 64 bit laptop as I do not know that I would be able to revert successfully and I want her to be able to write small programs for schoolwork without extra hassle.

[elibendersky]

OK, I can reproduce the problem on a clean virtual machine running a pristine XP home SP2. No Python was previously installed.

Steps to reproduce:

1. Install Python 3.2.1 MSI from http://www.python.org/download/releases/3.2.1/
2. Run IDLE from start menu
3. Close IDLE
4. A pythonw.exe process stays running after IDLE has exited

Running steps 2-3 N times leaves N pythonw.exe processes alive

[elibendersky]

Am I missing something, or is there no explicit command to kill the subprocess on Windows in PyShell.py

The kill_subprocess method (which does get invoked) of ModifiedInterpreter is:

    def kill_subprocess(self):
        try:
            self.rpcclt.close()
        except AttributeError:  # no socket
            pass
        self.unix_terminate()
        self.tkconsole.executing = False
        self.rpcclt = None

The subprocess is started with:

    self.rpcpid = os.spawnv(os.P_NOWAIT, sys.executable, args)

Could it be that in earlier versions this ensured the subprocess exited with its parent, but this somehow got modified?

Note that the same code existed in PyShell.py for ages, so it's unlikely that the culprit is there.