Unable to append to array.array after using it in a plot

[MichaelClerx]

I'm running simulations and logging the output to array.array objects. After I use one of these to plot results, the array's append() method stops working.

It seems something in matplotlib's plotting code causes the array to "export a buffer"

Code for reproduction

#!/usr/bin/env python3
import array
import matplotlib.pyplot as plt

x = array.array('d')
y = array.array('d')

# Appending to array works as expectecd
for i in range(10):
    x.append(i)
    y.append(i**2)

# Now using x and y in a plot
plt.figure()
plt.plot(x, y)

# Side effect has occurred: no longer able to append to array
x.append(11)

Actual outcome

Traceback (most recent call last):
  File "./log_append.py", line 15, in <module>
    x.append(11)
BufferError: cannot resize an array that is exporting buffers

Expected outcome

No errors

Matplotlib version

• Operating system: Fedora linux
• Matplotlib version: 3.0.3
• Matplotlib backend (print(matplotlib.get_backend())): template, Agg
• Python version: 3.7.6
• Other libraries: Myokit (pip installable)

Matplotlib installed via dnf (package manager)

[jklymak]

Can you make a self-contained example that doesn't require myokit or whatever is in example? That error doesn't come from matplotlib, so its pretty hard for us to debug or even know that matplotlib is the source of the problem.

[MichaelClerx]

Fair enough:

#!/usr/bin/env python3
import array
import matplotlib.pyplot as plt

x = array.array('d')
y = array.array('d')

# Appending to array works as expectecd
for i in range(10):
    x.append(i)
    y.append(i**2)

# Now using x and y in a plot
plt.figure()
plt.plot(x, y)

# Side effect has occurred: no longer able to append to array
x.append(11)

The error does not originate in matplotlib, the issue is that matplotlib somehow changes the state of the array.array that was passed into .plot(). It's a side effect. Doesn't happen on Python 2.7 with matplotlib 2.2.5, but happens on Python 3.7.6 with matplotlib 3.0.3

[ImportanceOfBeingErnest]

I suspect that this has to do with the python version in use, rather than matplotlib. As far as I know there is no special support for python's inbuilt array. Edit: invalidated by the below.

Note that you can easily copy your data to one of the supported formates, e.g. a python list

plt.plot(list(x), list(y))

or a numpy array

plt.plot(np.array(x), np.array(y))

[anntzer]

Actually, this is yet another subtle bug arising from (guess what) the unit system. This can be triggered without using array.array:

from matplotlib import pyplot as plt
import numpy as np

t = np.array([1., 2.])
fig, ax = plt.subplots()
l, = ax.plot(t, t)
t[0] = 1.5
l.set_drawstyle("steps")
print(l._yorig)
plt.show()

Note how the x axis uses the "new" x values ([1.5, 2]) but the y axis the initial ones ([1, 2]).
Essentially, this is because Line2D keeps a reference to the "original" data (_xorig, _yorig) to be able to handle unit changes (and this is also triggered by drawstyle changes, as in the demo here), but of course this means that changes to the original data are sometimes subtly brought in sync with the "copy" used by Line2D.

---

Likely this wasn't a problem on Py2 simply because the Py2 buffer didn't check whether there were additional references to it.

[ImportanceOfBeingErnest]

The input for scatter should go through the same unit conversion than for plot; yet, for scatter the problem does not occur.
Would there be a reason to keep input mutable?

So a quick test shows that

diff --git a/lib/matplotlib/axes/_base.py b/lib/matplotlib/axes/_base.py
index 109e2a5b1..00d9a4001 100644
--- a/lib/matplotlib/axes/_base.py
+++ b/lib/matplotlib/axes/_base.py
@@ -333,6 +333,9 @@ class _process_plot_var_args:
         else:
             x, y = index_of(tup[-1])

+        x = np.array(x)
+        y = np.array(y)
+
         if self.axes.xaxis is not None:
             self.axes.xaxis.update_units(x)
         if self.axes.yaxis is not None:

(in line

matplotlib/lib/matplotlib/axes/_base.py

Line 333 in f10bcee

) would fix this. But this might cause performance decrease?!

[anntzer]

That's because scatter is a PolyCollection, which does not support unit reconversion (i.e., in theory, you can plot "meters" (in mpl's unit system sense) and then later convert them to "feet") -- only Line2D does, at it does so at the cost of an extremely complicated system.
By adding calls to np.array, you'd drop the "original" units and lose the ability to change them.
I'm not convinced this feature is really worth the complexity -- especially given that it only works for Line2Ds -- but this has already been discussed to death elsewhere.

[jklymak]

... ooops was just typing what @anntzer was typing, plus or minus.

I agree that we should get rid of unit re-conversion and simply resolve units when the plot method gets called. But perhaps @story645 and @tacaswell have other plans.

[MichaelClerx]

Not an option to just copy any data you want to keep a hold of?