Document tkinter and threads

[terryjreedy]

BPO	33479
Nosy	@terryjreedy, @roseman, @ambv, @serhiy-storchaka, @native-api, @miss-islington, @richardsheridan
PRs	bpo-33479: Remove unqualified tkinter threadsafe claim. #6990 bpo-33479: Tkinter docs #7287 [3.10] bpo-33479: Remove unqualified tkinter threadsafe claim. (GH-6990) #27704 [3.9] bpo-33479: Remove unqualified tkinter threadsafe claim. (GH-6990) #27705 bpo-33479: add architecture and threading model sections to Tkinter module docs #27717 [3.10] bpo-33479: Add architecture and threading model sections to Tkinter module docs (GH-27717) #27730
Files	[Python-Dev] Tkinter threading model description and fix plan.eml

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2021-08-17.11:49:56.261>
created_at = <Date 2018-05-13.04:02:32.380>
labels = ['3.11', 'type-bug', '3.9', '3.10', 'docs']
title = 'Document tkinter and threads'
updated_at = <Date 2021-08-17.11:49:56.260>
user = 'https://github.com/terryjreedy'

bugs.python.org fields:

activity = <Date 2021-08-17.11:49:56.260>
actor = 'lukasz.langa'
assignee = 'docs@python'
closed = True
closed_date = <Date 2021-08-17.11:49:56.261>
closer = 'lukasz.langa'
components = ['Documentation']
creation = <Date 2018-05-13.04:02:32.380>
creator = 'terry.reedy'
dependencies = []
files = ['47586']
hgrepos = []
issue_num = 33479
keywords = ['patch']
message_count = 19.0
messages = ['316447', '316450', '316492', '316606', '316607', '316610', '316669', '316687', '317149', '318085', '318146', '318207', '318264', '399308', '399311', '399312', '399407', '399410', '399744']
nosy_count = 9.0
nosy_names = ['terry.reedy', 'markroseman', 'docs@python', '__Vano', 'lukasz.langa', 'serhiy.storchaka', 'Ivan.Pozdeev', 'miss-islington', 'Richard Sheridan']
pr_nums = ['6990', '7287', '27704', '27705', '27717', '27730']
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue33479'
versions = ['Python 3.9', 'Python 3.10', 'Python 3.11']

[terryjreedy]

(Proposed patch below)

Library Reference Chapter 25, Graphical User Interfaces with Tk, covers tinter, some of its subpackages, and IDLE. The introduction, https://docs.python.org/3/library/tk.html states "the internal module _tkinter provides a threadsafe mechanism which allows Python and Tcl to interact." Or as Martin Loewis claimed, "Tkinter is thread-safe." (bpo-11077, msg127979).

Unfortunately, the unqualified 'threadsafe' claim is not true. If Tcl is compiled with thread support (the default for 8.6) worker thread calls work (for the examples I have tested). If not (the default for 8.5 and before), they are unreliable and can fail in multiple ways. Known factors include the number of worker threads making call, the type of calls, and the number of calls.

The deceptive claim and the lack of knowledge about the crucial role of how tcl is compiled has negative effects. First, people who have non-thread tcl have suffered trying to deal with random failures. (See issues listed in bpo-11077, msg183774, and bpo-33257.) Second, non-coredev tkinter experts have spread the equally wrong claim that 'tkinter is not threadsafe". (See the same message.) So people who have do have thread tcl are discouraged from exploiting the fact.

I suggest

1. Remove 'threadsafe' from the intro sentence. After it, add "See <Tkinter and threads> for more details." <..> indicates a link to the proposed new section below.

2. The tkinter section itself, https://docs.python.org/3/library/tkinter.html, currently says nothing about threads. I propose to add a new subsection, location to be decided.
   """
   Tkinter and threads.

If you want to use both tkinter and threads, the safest method is to make all tkinter calls in the main thread. If worker threads generate data needed for tkinter calls, use a queue.Queue to send the data to the main thread. For a clean shutdown, add a method to wait for threads to stop and have it called when the window close button [X] is pressed.

If you are using Tcl/Tk compiled with thread support, you can skip the queue and call tkinter methods in the worker threads. To avoid deadlocks when shutting down, you may have to join working threads in a separate 'join' thread that does not make tkinter calls.

If you are using Tcl/Tk compiled without thread support, tkinter calls in worker threads may initially work, but may eventually fail somewhat randomly. Known factors include the number of threads, the type of call, and the number of calls.

To determine whether your tcl/tk has thread support, look in <install-dir>/Lib/DLLs for files such as, tcl86(t).dll and tk86(t).dll (The tcl/tk version would then be '8.6'.) The t for 'thread' suffix is present or absent as tcl/tk was compiled with or without thread support.
<Add info for Linux and MacOS.>

<Consider adding code examples.>
"""

[terryjreedy]

The information about the compile option comes from Ivan Pozdeev, bpo-33257. I meant to add somewhere that thread support became default only with tk 8.6.

[native-api]

I was composing a letter to python-dev with all I know of how tkinter works with regard to threads and Tcl, and the fixing plans so that we're all on the same page if you wish to participate.

I'm no longer sure if it belongs in the mailing list so see it in the attachment.

The plan I have for fixing the documentation is there towards the end.
It includes what you suggested but is more detailed. I'll recite it below:

---

• Document the current behaviour by rewriting https://docs.python.org/3/library/tkinter.html .

  Since I've recently learned Tkinter, I can say which critical information is missing and which existing one proved useless.

  Principles:

  • include fundamental information critical to understand the module's behaviour
  • concentrate on reference documentation for the module because it's more important to have than a user guide ( https://meta.serverfault.com/questions/8934/what-to-do-with-questions-when-the-answer-is-in-a-man-page#comment22241_8938 )
  • split off/drop anything unrelated to the above two

  Resulting scheme:

  0. Intro: +supported Tcl/Tk versions and the two flavors; the fact that Tkinter adds its own logic, and the Tkinter doc concentrates on that logic and refers to Tcl/Tk docs for details that are taken from there without changes.
  • Move external links section to be bottom.

  1. Architecture. "Unlike most other GUI toolkits, Tcl/Tk consists of a few separate modules with a clear distinction between them, and this is non-transparect to the user:..." Base on https://docs.python.org/3/library/tkinter.html#how-tk-and-tkinter-are-related , but focus on what implements what rather than what _calls_ what; drop Xlib entry (transparent implementation detail, not required for understanding).

  2. Threading model. The above-described general idea and user-visible limitations what can be called where and when. (the letter has details that this refers to, too long to recite)

  3. References for tkinter, tkinter.Tk(), _tkinter.Tkapp (public interface only -- call() at least). Mention which functions cannot be called from other threads. Do not mention the two bpo-33257 bugs.

  • Move widget reference to another page.
  • Drop Python-Tcl mappings unless the reference sections need them as supplemental reference.
  • Drop tutorial: too primitive to be useful. Move tutorials to another page like https://docs.python.org/3/library/logging.html does.
  • Drop https://docs.python.org/3/library/tk.html -- proved hard to find. Make https://docs.python.org/3/library/tkinter.html the head page instead.

• Discuss which of the described behaviour is as intended and which is a bug. Proceed with fixing.

[roseman]

This seems very complicated. The official line on threads for Tk has always been to make all Tk calls from one thread, which is at least predictable and comprehensible. Is there any reason for Tkinter to suggest anything different?

This ignores the compilation issue of course. FYI, the Tcl core group will probably eliminate the possibility of doing non-threaded builds in the future, though with backwards compatibility issues, that's neither here nor there.

[terryjreedy]

The official line is the claim in the docs that tkinter *is* threadsafe, repeated by Martin on at least one issue. See the first paragraph above. Therein lies the problem.

A reason to not just reverse the claim is that it is, at least for practical purposes, true, by default, for 8.6. See the opening post of bpo-11077 for one motivation for calling from threads. (Numerous others have tried and reported failures, though usually without explaining why they tried.)

Mark, do you know how to identify the tcl/tk build on MacOS or Linux? If so, and you have installed 3.7.0 on MacOS since b0, which includes 8.6.8, can you report which it is? Ditto for any Linux distribution you have.

[native-api]

This seems very complicated. The official line on threads for Tk has always been to make all Tk calls from one thread, which is at least predictable and comprehensible. Is there any reason for Tkinter to suggest anything different?

Tcl/Tk doesn't have a notion of a blocking event loop like other GUI toolkits do. Any code using Tcl must essentially call Tcl_DoOneEvent/update regularly at strategic points.
This allows to work completely in one thread -- i.e. even if the OS has no threads whatsoever. Tcl/Tk is very old, and this model made perfect sense back then (and still does in some scenarios -- e.g. https://stackoverflow.com/questions/4083796/how-do-i-run-unittest-on-a-tkinter-app -- so there's no point in dropping it).
If we'll be updating tutorials (the reference is a priority though), we definitly need to demonstrate this option.

The current best practice for GUI programming is different. There's one "GUI" thread that runs just the event loop constantly, and other threads submit GUI-related work items into its queue (whatever they are called - "messages", "events", "futures"...). Likewise, for any lengthy task, the GUI thread spawns worker threads that report back on their progress via the same queue.

All more or less modern GUI toolkits implement and advertize this model as the primary one -- as does Tkinter. So, at least the work item submitting API must be thread-safe (and in other GUI tooltikts, it is -- see https://mail.python.org/pipermail/python-dev/2018-May/153359.html ).
For programmer's convenience, Tkinter does this transparently: whenever and whatever Python thread a call is made from, it makes it look for the Tcl interpreter as if all calls are sequential, and it enforces correct order for interdependent calls.

A great deal of complexity comes from the fact that Tcl's threading model is very unothodox. Tcl's team seem to only have accepted threads reluctantly and to have been campaigning against threads for years before that (https://wiki.tcl.tk/38128 is arguably the most egregious case).
So, what they did is tie a Tcl interpreter to an OS thread (by using thread local storage for its data). Long story short, no-one else does it like this (at least, I've never seen or heard of anything of the kind), and this is not what thread-local storage is meant for (http://www.open-std.org/jtc1/sc22/wg21/docs/papers/2014/n4324.html lists some use cases for TLS). The best practice is to use locks to ensure orderly access to the shared state instead. My guess for the decision is it was the easiest way to migrate the code base (yet still tough as https://wiki.tcl.tk/1370 seems to hint), and it kinda lines up with that "single thread" mindset.

Tkinter thus has to jump through hoops for calls coming from other threads (since for Python, it absolutely doesn't matter which OS thread makes a call).
All the limitations when using threaded Tcl (see the letter attached to the ticket for details) come solely from this tying. With nonthreaded Tcl (bugs nonwithstanding), it's free-for-all, everything can be called from everywhere. The only upside is that with threaded Tcl, calls to different interpreters can run in parallel.

This ignores the compilation issue of course. FYI, the Tcl core group will probably eliminate the possibility of doing non-threaded builds in the future, though with backwards compatibility issues, that's neither here nor there.

I know. Me asking them for clarifications from Tcl/Tk's side seems to have triggered it, in fact. Since the sole offender is their threading model, the way is to show them how it's defective and work towards improving it. We have at least a few years with old versions, and the limitations seem tolerable at first glance anyway, so that's not a priority.

do you know how to identify the tcl/tk build on MacOS or Linux?

The same way Tkinter does @_tkinter.c:624 . The equivalent Python is tkinter.Tk().tk.call("array","get","tcl_platform","threaded")

[roseman]

Hi Ivan, thanks for your detailed response. The approach you're suggesting ("Since the sole offender is their threading model, the way is to show them how it's defective and work towards improving it.") is in the end not something I think is workable.

Some historical context. Ousterhout had some specific ideas about how Tcl/Tk should be used, and that was well-reflected in his early control of the code base. He was certainly outspoken against threads. The main argument is that they're complicated if you don't know what you're doing, which included the "non-professional programmers" he considered the core audience. Enumerating how threads were used at the time, most of the uses could be handled (more simply) in other ways, such as event-driven and non-blocking timers and I/O (so what people today would refer to as the "node.js event model"). Threads (or separate communicating processes) were for long-running computations, things he always envisioned happening in C code (written by more "professional programmers"), not Tcl. His idea of how Tcl and C development would be split didn't match reality given faster machines, more memory, etc.

The second thing is that Tcl had multiple interpreters baked in pretty much from the beginning at the C level and exposed fairly early on (1996?) at the Tcl level, akin to PEP-554. Code isolation and resource management were the key motivators, but of course others followed. Creating and using Tcl interpreters was quick, lightweight (fast startup, low memory overhead, etc.) and easy. So in other words, the notion of multiple interpreters in Tcl vs. Python is completely different. I had one large application I built around that time that often ended up with hundreds of interpreters running.

Which brings me to threads and how they were added to the language. Your guess ("My guess for the decision is it was the easiest way to migrate the code base") is incorrect. The idea of "one thread/one interpreter" was just not seen as a restriction, and was a very natural extension of what had come before. It fit the use cases well (AOLserver was another good example) and was still very understandable from the user level. Contrast with Python's GIL, etc.

With that all said, there would be very little motivation to change the Tcl/Tk side to allow multiple threads to access one interpreter, because in terms of the API and programming model that Tcl/Tk advertises, it's simply not a problem. Keep in mind, the people working on the Tcl/Tk core are very smart programmers, know threads very well, etc., so it's not an issue of "they should know better" or "it's old." In other words, "show them how it's defective" is a non-starter.

The other, more practical matter in pushing for changes in the Tcl/Tk core, is that there are a fairly small number of people working on it, very part-time. Almost all of them are most interested in the Tcl side, not Tk. Changes made in Tk most often amount to bug fixes because someone's running into something in their own work. Expecting large-scale changes to happen to Tk without some way to get dedicated new resources put into it is not realistic.

A final matter on the practical side. As you've carefully noted, certain Tcl/Tk calls now happen to work when called from different threads. Consider those a side-effect of present implementation, not a guarantee. Future core changes could change what can be called from different threads, making the situation better or worse. From the Tcl/Tk perspective, this is not a problem, and would not be caught by any testing, etc. Even if it were, it likely wouldn't be fixed. It would be considered an "abuse" of their API (I think correctly).

My suggestion, given the philosophical and practical mismatch, is that Tkinter move towards operating as if the API Tk provides is inviolate. In other words, all calls into a Tcl interpreter happen from the same thread that created the Tcl interpreter. Tkinter acts as a bridge between Python and Tcl/Tk. It should present an API to Python programs compatible with the Python threading model. It's Tkinter's responsibility to map that onto Tcl/Tk's single threaded API through whatever internal mechanism is necessary (i.e. pass everything to main thread, block caller thread until get response, etc.)

I'd go so far as to suggest that all the Tkapp 'call' code (i.e. every place that Tkinter calls Tcl_Eval) check what thread it's in, and issue a warning or error (at least for testing purposes) if it's being called from the "wrong" thread. Having this available in the near future would help people who are debugging what are fairly inexplicable problems now.

The approach of making Tkinter responsible also has the advantage of dealing with far more Tcl/Tk versions and builds.

Given in practice that few people are really running into things, and that if they are, they know enough to be able to follow the instruction "all Tkinter calls from the same thread" for now, add the warnings/errors in via whatever "turn on debugging" mechanism makes sense. A future version of Python would include a fully thread-safe Tkinter that internally makes all Tcl/Tk calls from a single thread, as per above.

Sorry this is so incredibly long-winded. I hope the context at least is useful information.

[native-api]

@markroseman replied to python-dev since those perspectives are off topic for this ticket.

[native-api]

I'm currently rewriting the docs, too, according to the plan @ #msg316492. WIP @ https://github.com/native-api/cpython/tree/tkinter_docs .

You PR lines up fine though is made redundant by native-api@79b195a -- instead of removing Doc\library\tk.rst, I reduced it to a summary of chapter content without any claims about it whatsoever, like other chapter head pages.

[roseman]

Ivan, thanks for making a good first pass of this. The thread section still feels a lot like 'fighting' with the model. Do you mind if I take a crack at it? Won't get to it for a few days, but in case you have any stuff you're in the middle of.

I should clarify too that Tk apps almost universally do use a blocking event loop (i.e. 'vwait forever' at the end of a script). Application-level event handlers are supposed to respond quickly so control goes back to the event loop.

It's when control doesn't return there that things like the 'update idletasks' hacks are needed. In practice, I've noticed that's what seems to trip people up when they first start, as they try to emulate the flow of their non-GUI code, which frequently blocks. Far better that the program is restructured so that the event handler completes quickly. It's actually worse than it looks, because you can end up having nested event loops if you start randomly throwing this stuff in. That's conceptually hard for most people and a good source of bugs.

[Vano]

On 29.05.2018 23:20, Mark Roseman wrote:

Mark Roseman <mark@markroseman.com> added the comment:

Ivan, thanks for making a good first pass of this. The thread section still feels a lot like 'fighting' with the model. Do you mind if I take a crack at it? Won't get to it for a few days, but in case you have any stuff you're in the middle of.

I should clarify too that Tk apps almost universally do use a blocking event loop (i.e. 'vwait forever' at the end of a script). Application-level event handlers are supposed to respond quickly so control goes back to the event loop.

It's when control doesn't return there that things like the 'update idletasks' hacks are needed. In practice, I've noticed that's what seems to trip people up when they first start, as they try to emulate the flow of their non-GUI code, which frequently blocks. Far better that the program is restructured so that the event handler completes quickly. It's actually worse than it looks, because you can end up having nested event loops if you start randomly throwing this stuff in. That's conceptually hard for most people and a good source of bugs.

I'm pretty much done with the threading section but if you think it
could use more clarification, sure. You can make a PR against my branch
to integrate changes (or vice versa).

In line with the aforementioned plan, the "Threading model" section
needs to tell these things critical to interstanding the module's
behavior and usage patterns:

• There are two basic GUI execution models (stricty speaking, these are
  event-driven models in general, but no need to go that deep): single
  thread with pumping messages by hand (read: cooperative multitasking),
  and UI thread + worker threads (read: preemptive mulitasking). The
  latter is prevalent now so the reader is more likely to be familiar with it.
• Tcl/Tk implements the former model (which is unusual), Tkinter
  emulates the latter with its own logic (so Tcl/Tk docs won't have info
  on this) but supports the former one as well. (So update() is not a
  "hack" at all, it's just intended for a different use case that doesn't
  come up often.)
• Tkinter calls can and should be made from any threads (this is also
  unusual), but there are user-visible limitations.
• Tcl event loop is shared (another unusual gimmick), which is also
  user-visible.

This section is not the place to showcase concrete usage patterns,
that's what tutorials are for. But it can make references to relevant
Tkinter functions as long as this doesn't garble the narration.

I imagine tutorial as a separate page (see the plan how it should be
linked to), with the following sections, each illustrated with code.
It's not meant to be an essential part of this ticket because it's of
lower priority.

• Create initial UI, then run mainloop(). All the rest is done with ui
  commands and events. (a hello world example)
• Start a worker thread for any action that may take more than a
  fraction of a second. Make Tkinter calls from the worker thread to pass
  back info on its progress.
• Collect worker threads and do other cleanup at exit via a cleanup
  function. Call it from both a special exit command, if any, _and_ from
  .protocol("WM_DELETE_WINDOW"). Lengthy/perpetual worker threads' logic
  must be interruptable for this.
• For more complex logic, use the Model-View-Presenter pattern.
• ?Something about exception handling? (Propagating exceptions? Making
  unhandled exceptions visible to the user? I dunno atm)
• An example of using Tcl's execution model, i.e. with
  dooneevent()/update()/update_idletasks() instead of mainloop(), like a
  Tcl program would do.

[roseman]

I've made some changes to what Ivan started, which you can find here: https://github.com/roseman/cpython/tree/tkinter_docs

The first two commits are minor updates/improvements not really related to threading, and I suspect are uncontroversial.

The last commit rewrites the thread model section. The main focus is on how things work in practice, and to eliminate what I previously referred to as "fighting" between the two models.

It explains how Python and Tcl differ on threads, the mappings between threads and interpreters, etc.

It plays down the comparison to other GUI toolkit event models. It still highlights the difference for people familiar with them. It doesn't try to teach both models for those who don't know either (which includes those who may not have done much if any GUI programming) since that is extraneous to their present needs. It talks simply about why event handlers blocking is a bad thing. It avoids what I think are unnecessary details about Tcl in this context.

There is then an area for special cases that highlight the various actual trouble spots when it comes to threads, what can cause them, and how to avoid them.

I hope this covers the real issues without overly complicating things. Any thoughts?

[native-api]

@markroseman I'm about 50% okay with your changes. Could you create a PR against my branch so we can work out the specifics?

[ambv]

New changeset 6b37d0d by Terry Jan Reedy in branch 'main':
bpo-33479: Remove unqualified tkinter threadsafe claim. (GH-6990)
6b37d0d

[miss-islington]

New changeset 2e1fef5 by Miss Islington (bot) in branch '3.10':
bpo-33479: Remove unqualified tkinter threadsafe claim. (GH-6990)
2e1fef5

[ambv]

New changeset c7dfbd2 by Miss Islington (bot) in branch '3.9':
bpo-33479: Remove unqualified tkinter threadsafe claim. (GH-6990) (GH-27705)
c7dfbd2

[ambv]

New changeset 08caf2d by Mark Roseman in branch 'main':
bpo-33479: Add architecture and threading model sections to Tkinter module docs (GH-27717)
08caf2d

[miss-islington]

New changeset 2666d70 by Miss Islington (bot) in branch '3.10':
bpo-33479: Add architecture and threading model sections to Tkinter module docs (GH-27717)
2666d70

[ambv]

This is done. Thanks, Mark!

For remaining discussion on reworking Tkinter docs, please refer to BPO-42560.