Discussion: Use of masked arrays internally

[scottshambaugh]

Chatting with @timhoffm in #31004 (comment), I'd like to more widely discuss our use of masked arrays in internal code. My position is that they add a lot of maintenance headache and fragility, and we would be best off excising them from the codebase altogether.

Currently we have 185 counts of np.ma in the core library. Most of them are pass-through checks, or are immediately .filled() with nans.

Problems

• Internal APIs do not usually document whether they expect or can handle masked arrays. It's easy to miss needing to handle the masked array case, contributors may not be familiar with them, and numpy functions still work so we can fail silently
• Lots of code clutter and specific paths needed to handle masked arrays
• Poor test coverage of masked array inputs/outputs
• Performance degradation from passing around extra data and not being able to rely on more efficient array operations.
  Note also that a = np.ma.masked_array(a, mask=mask).filled() is slower than a = np.where(mask, np.nan, a) is slower than a[mask] = np.nan.

Proposal

• Replace all instances of masked arrays with float arrays filled with NaNs, or in instances where the underlying data needs to be preserved (eg when clipping) then use an explicitly defined boolean mask variable
• Sanitize all user inputs to pure numpy arrays and do not return masked arrays
• Document a dev policy to minimize internal use of masked arrays, and to never return them

Migration issues

• cbook.safe_masked_invalid() is used a few places and causes some other functions to return a masked array. I haven't traced this all the way through to see where those come out
• All colors.Normalize subclasses preserve input masks in their output (this is the only docstring I see with a masked array output documented for the public API)

Deprecation warnings might be difficult to raise without spamming regular uses of these functions, since we can't tell when users use the mask on returned arrays. We may need to accept some silent breakage and just make the release notes clear.

[jklymak]

Overall I'd be ok with this as I think masked arrays are a concept that never caught on

I think the objection is going to be that we have the concept of "invalid" color in colormaps, and it is not the same as missing or masked data. I think this is a valid distinction but I'm not sure if anyone uses it in real life data analyses or publications. I'd love to see published examples!

[MO-marcomilan]

I come back from 2 days of "masked headache". I had issues in:

• using np.percentile asI had to fill the masked values with nan and use np.nanpercentile.
• usng sklearn functions as sklearn doesn't support masks
• and something else that now I don't remember.

Next time I will think 2 times before introducing masked arrays.

[jfrost-mo]

As a user I'd expect matplotlib to continue supporting masked arrays as input from the typical high level plotting functions, as they can arise from many different sources, especially when plotting meteorological observations.

However, I don't really care whether they are maintained within matplotlib, and I've never considered using the arrays returned from matplotlib for anything other than passing to other plotting functions, as it is typically at the end of my processing.

So if the only thing matplotlib does with a masked array is to infill it with NaNs, that seems fine to me.

[sanrishi]

As a user, I'm fully on board with this. The ecosystem friction with np.ma is real, and simplifying the internals seems like a win for stability.
plt.plot(masked_array) needs to keep working because a lot of upstream libraries (like netCDF4) output masks by default. I really don't care what happens internally or what format gets returned, as long as I don't have to manually call .filled(np.nan) every time I want to make a plot.

The only spot where "sanitize to NaNs" worries me is with masked integers (e.g. categorical data or counts). Converting those to floats just to handle missing values can get messy with precision or memory, so the explicit boolean mask option mentioned in the proposal seems strictly necessary there.

Regarding the colormap question—I honestly can't recall a time I needed to visually distinguish between "invalid" (math error) and "masked" (missing data). They are both just "no data" to the viewer.

[timhoffm]

Colormaps: While we have _get_rgba_and_mask(), the rgba values of masked entries is the "bad" color and the mask is not used in the colormap. I therefore believe we currently do not have any visual distinction between NaN and masked values

matplotlib/lib/matplotlib/colors.py

Line 779 in ec3e68d

rgba, mask = self._get_rgba_and_mask(X, alpha=alpha, bytes=bytes)

The only spot where "sanitize to NaNs" worries me is with masked integers (e.g. categorical data or counts

In those cases, it's likely an option to discard the respective data points.

More generally, I suspect, we currently do not have any special visualization for masked points. In the end, they are always either discarded or handled like NaN (which in some cases is again the same).

[sanrishi]

Re: Colormaps — That makes sense. If the visual output is identical, collapsing the "bad" vs "masked" distinction is safe.

Re: Integers — I would caution against the idea of "discarding" data points for integers.

For scatter, discarding works fine. But for plot (lines), removing a point changes the topology: it connects the neighbors instead of leaving a gap. To draw the gap, the renderer explicitly needs a placeholder in the data stream.

Since we can't use NaN as that placeholder for integers (without casting to float and risking precision loss), we likely need a dedicated mechanism for these paths—perhaps passing an explicit valid boolean array alongside the data, rather than relying on sentinel values or filtering.

[mo-garethsjones]

What does "NaN" stand for? It stands for "Not a Number", and is used in computing to represent a mathematical calculation that cannot be shown as a number. e.g., 0/0.

Clearly "NaN" should not be used to represent missing data in a dataset, as then it would be impossible to tell apart data that has been deliberately set as being "missing", or data that is the result of a mathematical operation that is unrepresentable as a number.

In my field, climate science, missing data in source data (before getting to python) is usually set to a number well outside the range of data being presented and plotted, e.g., -9999, -1e+30. It is extremely problematic when I come across datasets with "NaN"s in them as I don't know if that was deliberate or an error!

Using "NaN" as some sort of replacement for masked arrays is not the way forward.

[rcomer]

@mo-garethsjones this proposal will not affect your data set, only (potentially) your plots. Do you have examples where Matplotlib currently gives you a different result for masked points vs Nan points?

[sanrishi]

@rcomer For floats, NaN and Masked usually render identically (invisible).

However, for integers, there is a hard difference because integers cannot hold NaN. To use NaN, we would have to cast the data to float (changing the type/precision) or "discard" the masked points.

If we "discard" points in a line plot, the rendering changes completely: Masked draws a gap, Discarded connects the neighbors. So for integer line plots, the distinction is visually critical.

[jklymak]

Colormaps: While we have _get_rgba_and_mask(), the rgba values of masked entries is the "bad" color and the mask is not used in the colormap. I therefore believe we currently do not have any visual distinction between NaN and masked values

I'm surprised that is the case, but you are correct. I'd have expected below that missing data (middle cell) would remain "transparent" while the NaN cell would become red.

Given this, I think the argument for keeping the distinction between NaN and masked is even weaker...

X = np.arange(15.0).reshape((3, 5))
fig, ax = plt.subplots(3,1, sharex=True, layout='constrained', figsize=(3, 3))
ax[0].pcolormesh(X, cmap='viridis', shading='nearest')

Xm = np.ma.masked_where(X==7, X)
Xm[2, 3] = np.nan
ax[1].pcolormesh(Xm, cmap='viridis', shading='nearest')

cmap = plt.get_cmap('viridis').with_extremes(bad='red')
ax[2].pcolormesh(Xm, cmap=cmap, shading='nearest')

In my field, climate science, missing data in source data (before getting to python) is usually set to a number well outside the range of data being presented and plotted, e.g., -9999, -1e+30. It is extremely problematic when I come across datasets with "NaN"s in them as I don't know if that was deliberate or an error!

@mo-garethsjones This is indeed the usual argument for distinguishing masked and NaN data. But do you actually find that extremely problematic? I'm in your field, and everyone I know uses xarray to handle multi-dimensional data sets. xarray does not have the concept of masked data versus NaN despite being developed well after masked arrays were introduced to numpy. In fact if you have a netcdf file with both NaN and fill_value in it, the distinction will not survive a round trip with xarray (the NaN will get converted to fill_value). I take this as a pretty strong signal that while formally there is a distinction to be made between NaN and missing data, practically the difference is not worth the extra code headaches.

However, as @rcomer says, I don't think we are proposing to stop accepting masked arrays as an input, just that they will be converted to NaN-filled non-masked arrays for internal use very early in the lifescycle.

With respect to integers - at some point all our data gets converted to floats - there is no reason to keep the arrays internally as integers.

[sanrishi]

@jklymak The pcolormesh demonstration is convincing—visually, the handling of "bad" vs "masked" is effectively consistent there.

at some point all our data gets converted to floats - there is no reason to keep the arrays internally as integers.

While true for final screen coordinates, enforcing an early cast to float64 (to support NaN) poses a data integrity risk for large integers. float64 loses integer precision above $2^{53}$ ($\approx 9 \times 10^{15}$).

Common inputs like nanosecond timestamps (if treated as raw ints) or large database IDs often exceed this range. If we convert these to float internally to handle missing values, we effectively corrupt the data before the Locators or Formatters can see it. Preserving the integer type (via a mask) ensures that the axis labels remain accurate even if the render pipeline eventually floats the coordinates.

[jklymak]

@sanrishi locators and formatters and our conversion pipeline for axes-mappable data types are actually a good question. I'm not sure how we currently handle those, and agree there is no such thing as NaN in integer arrays. Note there is NaT for datetime64 arrays.

[timhoffm]

Options for integers:

1. carry an extra mask
2. use a special sentnel value - Since we control the data, we can basically always find an unused value and use that.
3. If a masked array: If values are not too large, convert to float; error out otherwise (somewhat limits support for ints, so needs a deprecation)
4. drop support for masked int arrays completely (with deprecation).

Remarks:

• 1, 2 will only be feasible after encapsulating the data (see our data prototype). Otherwise we'd need a lot of special casing on internal processing code.
• Apart from Line2D where "missing" breaks the connection between points, are there other Artists that make that distinction? If not, there maybe could be a Line2D internal solution (e.g. introducing an internal notion of segments; i.e. list of arrays, so that "missing" data separate segments).
• pandas has introduced a nullable integer https://pandas.pydata.org/docs/user_guide/integer_na.html. However, from what I've seen, this is a lot of hassle and we should try to avoid that if possible. (Though accepting it like we accept a masked array may be ok).