[Bug]: pyplot hangs at pause in sonoma 14.3 with backend MacOSX

[cpraveen]

Bug summary

When I use pyplot.pause, the code hangs and I have to kill the python session.

Code for reproduction

import numpy as np
import matplotlib.pyplot as plt

x = np.linspace(0,1,100)
y = np.sin(x)
z = np.cos(x)

plt.plot(x,y)
plt.draw()
plt.pause(1) # <== hangs here
plt.plot(x,z)
plt.show()

Actual outcome

Code hangs at plt.pause.

Expected outcome

Code should run fully.

Additional information

This is happening after I upgrade to sonoma 14.3

Operating system

Mac os sonoma 14.3

Matplotlib Version

3.8.2

Matplotlib Backend

MacOSX

Python version

Python 3.10.12

Jupyter version

No response

Installation

conda

[cpraveen]

Backend TkAgg works fine.

[aadi-bh]

I am on Python 3.8.17 with Matplotlib 3.6.2, on macOS Sonoma 14.3 and I am facing a similar issue, probably related: the session does not hang, but I see only one plot, as if the plot(x,z) line did not exist.

[dstansby]

Thanks for the report - I can reproduce this on the current main branch. I'm on macOS 14.3 (Sonoma). Thanks for sharing that it becomes a problem after updating the OS to macOS 14 too.

Worth noting that

• I can't interrupt the process using cmd+c, but force quitting the Matplotlib window does close it.
• I don't event get a plot showing up, just an empty Matplotlib window with no figure.

[dstansby]

EDIT: Ignore this comment, I did the bisect wrong 🤦

With Matplotlib 3.7.3 this doesn't crash, but the second plot doesn't show up after the plt.pause() call. The change in behaviour from not crashing to crashing bisects to 059c8ee from #25966 (cc @anntzer). As of 059c8ee I get the following error message after the 1 second pause has ended:

Traceback (most recent call last):
  File "/Users/dstansby/software/matplotlib/test.py", line 12, in <module>
    plt.show()
  File "/Users/dstansby/software/matplotlib/lib/matplotlib/pyplot.py", line 519, in show
    return _get_backend_mod().show(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/dstansby/software/matplotlib/lib/matplotlib/backend_bases.py", line 3437, in show
    cls.mainloop()
  File "/Users/dstansby/software/matplotlib/lib/matplotlib/backends/backend_macosx.py", line 185, in start_main_loop
    _macosx.wake_on_fd_write(rsock.fileno())
    ^^^^^^^^^^^^^^^^^^^^^^^^
AttributeError: module 'matplotlib.backends._macosx' has no attribute 'wake_on_fd_write'

[dstansby]

My bad - in the above comment I was forgetting to recompile at each commit.

This actually bisects back to 072226a from #27290 (cc @ksunden)

[greglucas]

I'm on 14.2.1 (intel mac) and it is working as expected for me, so possibly related to 14.3 specifically.

I am on Python 3.8.17 with Matplotlib 3.6.2, on macOS Sonoma 14.3 and I am facing a similar issue, probably related: the session does not hang, but I see only one plot, as if the plot(x,z) line did not exist.

I think this is expected behavior, there should be another plt.draw() before the final `plt.show(), or you can resize your window and the other line should show up.

[ksunden]

Prior to that commit, we were segfaulting on Python 3.12 when the window was resized with the MacOS backend.

I can confirm the following:

• On macOS 14.1.1, the problem is not present
• Upon upgrading to 14.3 I do see the problem
• reverting 072226a does make the pause work, but also does in fact still segfault on resize.

The commit should only be ensuring GIL state (and subsequently releasing it), I'm not at all clear why that would cause this... Especially since the only things happening between acquiring and releasing the GIL are freeing memory.

[greglucas]

Do we need to stick within Python's memory allocator? Some of this came in via: #17263

diff --git a/src/_macosx.m b/src/_macosx.m
index 8038b72899..e8e7711dea 100755
--- a/src/_macosx.m
+++ b/src/_macosx.m
@@ -1141,16 +1141,19 @@ - (void)setCanvas: (PyObject*)newCanvas
 static void _buffer_release(void* info, const void* data, size_t size) {
     PyGILState_STATE gstate = PyGILState_Ensure();
     PyBuffer_Release((Py_buffer *)info);
-    free(info);
+    PyMem_Free(info);
     PyGILState_Release(gstate);
 }
 
 static int _copy_agg_buffer(CGContextRef cr, PyObject *renderer)
 {
-    Py_buffer *buffer = malloc(sizeof(Py_buffer));
+    PyGILState_STATE gstate = PyGILState_Ensure();
+    Py_buffer *buffer = PyMem_Malloc(sizeof(Py_buffer));
+    PyGILState_Release(gstate);
 
     if (PyObject_GetBuffer(renderer, buffer, PyBUF_CONTIG_RO) == -1) {
         PyErr_Print();
+        _buffer_release(buffer, NULL, 0);
         return 1;
     }

[tacaswell]

The commit should only be ensuring GIL state (and subsequently releasing it), I'm not at all clear why that would cause this...

My guess is that there is a second lock involved (maybe in free, maybe in the OSX code that is calling our callback) that we are deadlocking against. We already should have realized there are threads someplace due to the segfault (if no threads there can be no race to lose) so it at least still holds together conceptually.

[tacaswell]

maybe if we take the free(...) out of the GIL lock?

[tacaswell]

@aadi-bh

The issue you are seeing is because we are in a state where are not automatically re-drawing on show. If you interact with the figure (pan, zoom, resize) then the second line will show up. Any of the three changes in the code below should fix that problem:

import numpy as np
import matplotlib.pyplot as plt

x = np.linspace(0,1,100)
y = np.sin(x)
z = np.cos(x)

# plt.ion()  <- adding this
plt.plot(x,y)
plt.draw()
plt.pause(1) # 
plt.plot(x,z)
plt.draw_all()  # <- or this 
# or fig.canvas.draw_idle()  # <- or this if you have a figure object
plt.show()

[aadi-bh]

@aadi-bh

The issue you are seeing is because we are in a state where are not automatically re-drawing on show. If you interact with the figure (pan, zoom, resize) then the second line will show up. Any of the three changes in the code below should fix that problem:

import numpy as np
import matplotlib.pyplot as plt

x = np.linspace(0,1,100)
y = np.sin(x)
z = np.cos(x)

# plt.ion()  <- adding this
plt.plot(x,y)
plt.draw()
plt.pause(1) # 
plt.plot(x,z)
plt.draw_all()  # <- or this 
# or fig.canvas.draw_idle()  # <- or this if you have a figure object
plt.show()

@tacaswell You're right! Panning the figure makes the second line show up. Thanks!

[cpraveen]

Comment out plt.show() still hangs the code. Just plot once and call pause, it hangs

import numpy as np
import matplotlib.pyplot as plt

x = np.linspace(0,1,100)
y = np.sin(x)

plt.plot(x,y)
plt.pause(1) # <== hangs here

Why would this code not run fully. There is some new bug.

[tacaswell]

The summary here:

Prior to mpl 3.8.2 we were not grabbing the GIL when cleaning up buffers ( #27290 ) which was a bug, but a latent one because the object that had the race condition on reference counting was not simultaneously used from more than one thread (the background thread ran only after we were done with it). However in python 3.12 failing to hold the GIL resulted in a segfault due to the per-subinterpreter GIL work (because without the GIL the pointer to parts of Python's internal memory management was NULL which was unconditionally dereference and hence the crash).

In OSX14.3 some lock (I do not know which) appears to have been added and shared between the cleanup callback and something in the event processing code we run in _start_event_loop. The story is:

• thread A holds both mystery osx lock (MOL) and GIL
• thread A release MOL
• thread B grabs MOL
• thread B block trying to grab GIL
• thread A blocks trying to grab MOL

And now we are deadlocked in a way that the UI is completely unresponsive (because thread A is what processes the mouse/keyboard/... events coming in from the window management system). The fix in #27755 is to release the GIL in thread A and hence avoid the deadlock.

@aadi-bh did not see the hang or the segfault because they were on mpl 3.6 and Python 3.8 so completely unrelated behavior nit exposed by the same code!

Things like this are why it is going to be a couple of years (5-10) before the free-threading features in CPython are going to be fully usable by general Scientific Python user (threading is HARD).

[duburcqa]

I don't know if this is related, but I get stuck in deadlock when I call flush_event on Matplotlib>=3.8.0 while it worked fine before. It is not systematic but very frequent. It happens for sure on MacOS and I think also on Linux but I'm not 100% sure.

[tacaswell]

Exactly which version do you have? This was fixed in 3.8.3 so 3.8.0-3.8.2 will have this issue.

Which backend are you seeing it with on linux?