Skip to content


Subversion checkout URL

You can clone with
Download ZIP
branch: master
Fetching contributors…

Cannot retrieve contributors at this time

813 lines (786 sloc) 47.689 kB
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN"
<html xmlns="">
<meta http-equiv="Content-Type" content="text/html; charset=utf-8" />
<title>Missing data: an orientation &mdash; Numpy</title>
<link rel="stylesheet" type="text/css" href="_static/css/spc-bootstrap.css">
<link rel="stylesheet" type="text/css" href="_static/css/spc-extend.css">
<link rel="stylesheet" type="text/css" href="_static/css/jquery-ui.css">
<link rel="stylesheet" href="_static/scipy.css" type="text/css" />
<link rel="stylesheet" href="_static/pygments.css" type="text/css" />
<script type="text/javascript">
FILE_SUFFIX: '.html',
<script type="text/javascript" src="_static/js/bootstrap.min.js"></script>
<script type="text/javascript" src="_static/js/jquery.min.js"></script>
<script type="text/javascript" src="_static/js/jquery-ui.min.js"></script>
<script type="text/javascript" src="_static/jquery.js"></script>
<script type="text/javascript" src="_static/underscore.js"></script>
<script type="text/javascript" src="_static/doctools.js"></script>
<link rel="shortcut icon" href="_static/favicon.ico"/>
<link rel="top" title="Numpy" href="index.html" />
<div class="container">
<div class="top-scipy-org-logo-header" style="background-color: #a2bae8;">
<a href="index.html">
<img border=0 alt="NumPy" src="_static/numpy_logo.png"></a>
<div class="container">
<div class="main">
<div class="row-fluid">
<div class="span9">
<div class="row-fluid">
<div class="span9">
<div class="spc-navbar">
<ul class="nav nav-pills pull-left">
<li class="active"><a href=""></a></li>
<li class="active"><a href="index.html">Numpy</a></li>
<div class="bodywrapper">
<div class="body" id="spc-section-body">
<div class="admonition note">
<p class="first admonition-title">Note</p>
<p class="last">This page contains the missing data document discussed on the mailing
list -
<a class="reference external" href=""></a></p>
<div class="section" id="missing-data-an-orientation">
<h1>Missing data: an orientation<a class="headerlink" href="#missing-data-an-orientation" title="Permalink to this headline">¶</a></h1>
<p>The debate about how numpy should handle missing data, a subject with
many preexisting approaches, requirements, and conventions, has been long and
contentious. There has been more than one proposal for how to implement
support into numpy, and there is a testable implementation which is
merged into numpy&#8217;s current master. The vast number of emails and differing
points of view has made it difficult for interested parties to understand
the issues and be comfortable with the direction numpy is going.</p>
<p>Here is our (Mark and Nathaniel&#8217;s) attempt to summarize the
problem, proposals, and points of agreement/disagreement in a single
place, to help the community move towards consensus.</p>
<div class="section" id="the-numpy-developers-problem">
<h2>The Numpy developers&#8217; problem<a class="headerlink" href="#the-numpy-developers-problem" title="Permalink to this headline">¶</a></h2>
<p>For this discussion, &#8220;missing data&#8221; means array elements
which can be indexed (e.g. A[3] in an array A with shape (5,)),
but have, in some sense, no value.</p>
<p>It does not refer to compressed or sparse storage techniques where
the value for A[3] is not actually stored in memory, but still has a
well-defined value like 0.</p>
<p>This is still vague, and to create an actual implementation,
it is necessary to answer such questions as:</p>
<ul class="simple">
<li>What values are computed when doing element-wise ufuncs.</li>
<li>What values are computed when doing reductions.</li>
<li>Whether the storage for an element gets overwritten when marking
that value missing.</li>
<li>Whether computations resulting in NaN automatically treat in the
same way as a missing value.</li>
<li>Whether one interacts with missing values using a placeholder object
(e.g. called &#8220;NA&#8221; or &#8220;masked&#8221;), or through a separate boolean array.</li>
<li>Whether there is such a thing as an array object that cannot hold
missing array elements.</li>
<li>How the (C and Python) API is expressed, in terms of dtypes,
masks, and other constructs.</li>
<li>If we decide to answer some of these questions in multiple ways,
then that creates the question of whether that requires multiple
systems, and if so how they should interact.</li>
<p>There&#8217;s clearly a very large space of missing-data APIs that <em>could</em>
be implemented. There is likely at least one user, somewhere, who
would find any possible implementation to be just the thing they
need to solve some problem. On the other hand, much of numpy&#8217;s power
and clarity comes from having a small number of orthogonal concepts,
such as strided arrays, flexible indexing, broadcasting, and ufuncs,
and we&#8217;d like to preserve that simplicity.</p>
<p>There has been dissatisfaction among several major groups of numpy users
about the existing status quo of missing data support. In particular,
neither the component nor use of floating-point NaNs as a
missing data signal fully satisfy the performance requirements and
ease of use for these users. The example of R, where missing data
is treated via an NA placeholder and is deeply integrated into all
computation, is where many of these users point to indicate what
functionality they would like. Doing a deep integration of missing
data like in R must be considered carefully, it must be clear it
is not being done in a way which sacrifices existing performance
or functionality.</p>
<p>Our problem is, how can we choose some incremental additions to
numpy that will make a large class of users happy, be
reasonably elegant, complement the existing design, and that we&#8217;re
comfortable we won&#8217;t regret being stuck with in the long term.</p>
<div class="section" id="prior-art">
<h2>Prior art<a class="headerlink" href="#prior-art" title="Permalink to this headline">¶</a></h2>
<p>So a major (maybe <em>the</em> major) problem is figuring out how ambitious
the project to add missing data support to numpy should be, and which
kinds of problems are in scope. Let&#8217;s start with the
best understood situation where &#8220;missing data&#8221; comes into play:</p>
<div class="section" id="statistical-missing-data">
<h3>&#8220;Statistical missing data&#8221;<a class="headerlink" href="#statistical-missing-data" title="Permalink to this headline">¶</a></h3>
<p>In statistics, social science, etc., &#8220;missing data&#8221; is a term of art
referring to a specific (but extremely common and important)
situation: we have tried to gather some measurements according to some
scheme, but some of these measurements are missing. For example, if we
have a table listing the height, age, and income of a number of
individuals, but one person did not provide their income, then we need
some way to represent this:</p>
<div class="highlight-python"><pre>Person | Height | Age | Income
1 | 63 | 25 | 15000
2 | 58 | 32 | &lt;missing&gt;
3 | 71 | 45 | 30000</pre>
<p>The traditional way is to record that income as, say, &#8220;-99&#8221;, and
document this in the README along with the data set. Then, you have to
remember to check for and handle such incomes specially; if you
forget, you&#8217;ll get superficially reasonable but completely incorrect
results, like calculating the average income on this data set as
14967. If you&#8217;re in one of these fields, then such missing-ness is
routine and inescapable, and if you use the &#8220;-99&#8221; approach then it&#8217;s a
pitfall you have to remember to check for explicitly on literally
<em>every</em> calculation you ever do. This is, obviously, an unpleasant way
to live.</p>
<p>Let&#8217;s call this situation the &#8220;statistical missing data&#8221; situation,
just to have a convenient handle for it. (As mentioned, practitioners
just call this &#8220;missing data&#8221;, and what to do about it is literally an
entire sub-field of statistics; if you google &#8220;missing data&#8221; then
every reference is on how to handle it.) Numpy isn&#8217;t going to do
automatic imputation or anything like that, but it could help a great
deal by providing some standard way to at least represent data which
is missing in this sense.</p>
<p>The main prior art for how this could be done comes from the S/S+/R
family of languages. Their strategy is, for each type they support,
to define a special value called &#8220;NA&#8221;. (For ints this is INT_MAX,
for floats it&#8217;s a special NaN value that&#8217;s distinguishable from
other NaNs, ...) Then, they arrange that in computations, this
value has a special semantics that we will call &#8220;NA semantics&#8221;.</p>
<div class="section" id="na-semantics">
<h3>NA Semantics<a class="headerlink" href="#na-semantics" title="Permalink to this headline">¶</a></h3>
<p>The idea of NA semantics is that any computations involving NA
values should be consistent with what would have happened if we
had known the correct value.</p>
<p>For example, let&#8217;s say we want to compute the mean income, how might
we do this? One way would be to just ignore the missing entry, and
compute the mean of the remaining entries. This gives us (15000 +
30000)/2, or 22500.</p>
<p>Is this result consistent with discovering the income of person 2?
Let&#8217;s say we find out that person 2&#8217;s income is 50000. This means
the correct answer is (15000 + 50000 + 30000)/3, or 31666.67,
indicating clearly that it is not consistent. Therefore, the mean
income is NA, i.e. a specific number whose value we are unable
to compute.</p>
<p>This motivates the following rules, which are how R implements NA:</p>
<dl class="docutils">
<dd><p class="first">NA values are understood to represent specific
unknown values, and thus should have value-like semantics with
respect to assignment and other basic data manipulation
operations. Code which does not actually look at the values involved
should work the same regardless of whether some of them are
missing. For example, one might write:</p>
<div class="highlight-python"><div class="highlight"><pre><span class="n">income</span><span class="p">[:]</span> <span class="o">=</span> <span class="n">income</span><span class="p">[</span><span class="n">np</span><span class="o">.</span><span class="n">argsort</span><span class="p">(</span><span class="n">height</span><span class="p">)]</span>
<p class="last">to perform an in-place sort of the <tt class="docutils literal"><span class="pre">income</span></tt> array, and know that
the shortest person&#8217;s income would end up being first. It turns out
that the shortest person&#8217;s income is not known, so the array should
end up being <tt class="docutils literal"><span class="pre">[NA,</span> <span class="pre">15000,</span> <span class="pre">30000]</span></tt>, but there&#8217;s nothing
special about NAness here.</p>
<dd><p class="first">In the example above, we concluded that an operation like <tt class="docutils literal"><span class="pre">mean</span></tt>
should produce NA when one of its data values was NA.
If you ask me, &#8220;what is 3 plus x?&#8221;, then my only possible answer is
&#8220;I don&#8217;t know what x is, so I don&#8217;t know what 3 + x is either&#8221;. NA
means &#8220;I don&#8217;t know&#8221;, so 3 + NA is NA.</p>
<p>This is important for safety when analyzing data: missing data often
requires special handling for correctness &#8211; the fact that you are
missing information might mean that something you wanted to compute
cannot actually be computed, and there are whole books written on
how to compensate in various situations. Plus, it&#8217;s easy to not
realize that you have missing data, and write code that assumes you
have all the data. Such code should not silently produce the wrong
<p>There is an important exception to characterizing this as propagation,
in the case of boolean values. Consider the calculation:</p>
<div class="highlight-python"><div class="highlight"><pre><span class="n">v</span> <span class="o">=</span> <span class="n">np</span><span class="o">.</span><span class="n">any</span><span class="p">([</span><span class="bp">False</span><span class="p">,</span> <span class="bp">False</span><span class="p">,</span> <span class="n">NA</span><span class="p">,</span> <span class="bp">True</span><span class="p">])</span>
<p>If we strictly propagate, <tt class="docutils literal"><span class="pre">v</span></tt> will become NA. However, no
matter whether we place True or False into the third array position,
<tt class="docutils literal"><span class="pre">v</span></tt> will then get the value True. The answer to the question
&#8220;Is the result True consistent with later discovering the value
that was missing?&#8221; is yes, so it is reasonable to not propagate here,
and instead return the value True. This is what R does:</p>
<div class="last highlight-python"><pre>&gt; any(c(F, F, NA, T))
[1] TRUE
&gt; any(c(F, F, NA, F))
[1] NA</pre>
<dd><p class="first">NaN and NA are conceptually distinct. 0.0/0.0 is not a mysterious,
unknown value &#8211; it&#8217;s defined to be NaN by IEEE floating point, Not
a Number. NAs are numbers (or strings, or whatever), just unknown
ones. Another small but important difference is that in Python, <tt class="docutils literal"><span class="pre">if</span>
<span class="pre">NaN:</span> <span class="pre">...</span></tt> treats NaN as True (NaN is &#8220;truthy&#8221;); but <tt class="docutils literal"><span class="pre">if</span> <span class="pre">NA:</span> <span class="pre">...</span></tt>
would be an error.</p>
<p class="last">In R, all reduction operations implement an alternative semantics,
activated by passing a special argument (<tt class="docutils literal"><span class="pre">na.rm=TRUE</span></tt> in R).
<tt class="docutils literal"><span class="pre">sum(a)</span></tt> means &#8220;give me the sum of all the
values&#8221; (which is NA if some of the values are NA);
<tt class="docutils literal"><span class="pre">sum(a,</span> <span class="pre">na.rm=True)</span></tt> means &#8220;give me the sum of all the non-NA
<div class="section" id="other-prior-art">
<h3>Other prior art<a class="headerlink" href="#other-prior-art" title="Permalink to this headline">¶</a></h3>
<p>Once we move beyond the &#8220;statistical missing data&#8221; case, the correct
behavior for missing data becomes less clearly defined. There are many
cases where specific elements are singled out to be treated specially
or excluded from computations, and these could often be conceptualized
as involving &#8216;missing data&#8217; in some sense.</p>
<p>In image processing, it&#8217;s common to use a single image together with
one or more boolean masks to e.g. composite subsets of an image. As
Joe Harrington pointed out on the list, in the context of processing
astronomical images, it&#8217;s also common to generalize to a
floating-point valued mask, or alpha channel, to indicate degrees of
&#8220;missingness&#8221;. We think this is out of scope for the present design,
but it is an important use case, and ideally numpy should support
natural ways of manipulating such data.</p>
<p>After R, is probably the most mature source of
experience on missing-data-related APIs. Its design is quite different
from R; it uses different semantics &#8211; reductions skip masked values
by default and NaNs convert to masked &#8211; and it uses a different
storage strategy via a separate mask. While it seems to be generally
considered sub-optimal for general use, it&#8217;s hard to pin down whether
this is because the API is immature but basically good, or the API
is fundamentally broken, or the API is great but the code should be
faster, or what. We looked at some of those users to try and get a
better idea.</p>
<p>Matplotlib is perhaps the best known package to rely on It
seems to use it in two ways. One is as a way for users to indicate
what data is missing when passing it to be graphed. (Other ways are
also supported, e.g., passing in NaN values gives the same result.) In
this regard, matplotlib treats and NaN values in the same way
that R&#8217;s plotting routines handle NA and NaN values. For these purposes,
matplotlib doesn&#8217;t really care what semantics or storage strategy is
used for missing data.</p>
<p>Internally, matplotlib uses arrays to store and pass around
separately computed boolean masks containing &#8216;validity&#8217; information
for each input array in a cheap and non-destructive fashion. Mark&#8217;s
impression from some shallow code review is that mostly it works
directly with the data and mask attributes of the masked arrays,
not extensively using the particular computational semantics of So, for this usage they do rely on the non-destructive
mask-based storage, but this doesn&#8217;t say much about what semantics
are needed.</p>
<p>Paul Hobson <a class="reference external" href="">posted some code</a> on the list that uses for
storing arrays of contaminant concentration measurements. Here the
mask indicates whether the corresponding number represents an actual
measurement, or just the estimated detection limit for a concentration
which was too small to detect. Nathaniel&#8217;s impression from reading
through this code is that it also mostly uses the .data and .mask
attributes in preference to performing operations on the MaskedArray
<p>So, these examples make it clear that there is demand for a convenient
way to keep a data array and a mask array (or even a floating point
array) bundled up together and &#8220;aligned&#8221;. But they don&#8217;t tell us much
about what semantics the resulting object should have with respect to
ufuncs and friends.</p>
<div class="section" id="semantics-storage-api-oh-my">
<h2>Semantics, storage, API, oh my!<a class="headerlink" href="#semantics-storage-api-oh-my" title="Permalink to this headline">¶</a></h2>
<p>We think it&#8217;s useful to draw a clear line between use cases,
semantics, and storage. Use cases are situations that users encounter,
regardless of what numpy does; they&#8217;re the focus of the previous
section. When we say <em>semantics</em>, we mean the result of different
operations as viewed from the Python level without regard to the
underlying implementation.</p>
<p><em>NA semantics</em> are the ones described above and used by R:</p>
<div class="highlight-python"><div class="highlight"><pre><span class="mi">1</span> <span class="o">+</span> <span class="n">NA</span> <span class="o">=</span> <span class="n">NA</span>
<span class="nb">sum</span><span class="p">([</span><span class="mi">1</span><span class="p">,</span> <span class="mi">2</span><span class="p">,</span> <span class="n">NA</span><span class="p">])</span> <span class="o">=</span> <span class="n">NA</span>
<span class="n">NA</span> <span class="o">|</span> <span class="bp">False</span> <span class="o">=</span> <span class="n">NA</span>
<span class="n">NA</span> <span class="o">|</span> <span class="bp">True</span> <span class="o">=</span> <span class="bp">True</span>
<p>With <tt class="docutils literal"><span class="pre">na.rm=TRUE</span></tt> or <tt class="docutils literal"><span class="pre">skipNA=True</span></tt>, this switches to:</p>
<div class="highlight-python"><div class="highlight"><pre><span class="mi">1</span> <span class="o">+</span> <span class="n">NA</span> <span class="o">=</span> <span class="n">illegal</span> <span class="c"># in R, only reductions take na.rm argument</span>
<span class="nb">sum</span><span class="p">([</span><span class="mi">1</span><span class="p">,</span> <span class="mi">2</span><span class="p">,</span> <span class="n">NA</span><span class="p">],</span> <span class="n">skipNA</span><span class="o">=</span><span class="bp">True</span><span class="p">)</span> <span class="o">=</span> <span class="mi">3</span>
<p>There&#8217;s also been discussion of what we&#8217;ll call <em>ignore
semantics</em>. These are somewhat underdefined:</p>
<div class="highlight-python"><pre>sum([1, 2, IGNORED]) = 3
# Several options here:
1 + IGNORED = 1
# or
1 + IGNORED = &lt;leaves output array untouched&gt;
# or
<p>The semantics are:</p>
<div class="highlight-python"><div class="highlight"><pre><span class="nb">sum</span><span class="p">([</span><span class="mi">1</span><span class="p">,</span> <span class="mi">2</span><span class="p">,</span> <span class="n">masked</span><span class="p">])</span> <span class="o">=</span> <span class="mi">3</span>
<span class="mi">1</span> <span class="o">+</span> <span class="n">masked</span> <span class="o">=</span> <span class="n">masked</span>
<p>If either NA or ignore semantics are implemented with masks, then there
is a choice of what should be done to the value in the storage
for an array element which gets assigned a missing value. Three
possibilities are:</p>
<ul class="simple">
<li>Leave that memory untouched (the choice made in the NEP).</li>
<li>Do the calculation with the values independently of the mask
(perhaps the most useful option for Paul Hobson&#8217;s use-case above).</li>
<li>Copy whatever value is stored behind the input missing value into
the output (this is what does. Even that is ambiguous in
the case of <tt class="docutils literal"><span class="pre">masked</span> <span class="pre">+</span> <span class="pre">masked</span></tt> &#8211; in this case copies the
value stored behind the leftmost masked value).</li>
<p>When we talk about <em>storage</em>, we mean the debate about whether missing
values should be represented by designating a particular value of the
underlying data-type (the <em>bitpattern dtype</em> option, as used in R), or
by using a separate <em>mask</em> stored alongside the data itself.</p>
<p>For mask-based storage, there is also an important question about what
the API looks like for accessing the mask, modifying the mask, and
&#8220;peeking behind&#8221; the mask.</p>
<div class="section" id="designs-that-have-been-proposed">
<h2>Designs that have been proposed<a class="headerlink" href="#designs-that-have-been-proposed" title="Permalink to this headline">¶</a></h2>
<p>One option is to just copy R, by implementing a mechanism whereby
dtypes can arrange for certain bitpatterns to be given NA semantics.</p>
<p>One option is to copy closely, but with a more optimized
implementation. (Or to simply optimize the existing implementation.)</p>
<p>One option is that described in the <a class="reference external" href="">NEP</a>, for which an implementation
of mask-based missing data exists. This system is roughly:</p>
<ul class="simple">
<li>There is both bitpattern and mask-based missing data, and both
have identical interoperable NA semantics.</li>
<li>Masks are modified by assigning np.NA or values to array elements.
The way to peek behind the mask or to unmask values is to keep a
view of the array that shares the data pointer but not the mask pointer.</li>
<li>Mark would like to add a way to access and manipulate the mask more
directly, to be used in addition to this view-based API.</li>
<li>If an array has both a bitpattern dtype and a mask, then assigning
np.NA writes to the mask, rather than to the array itself. Writing
a bitpattern NA to an array which supports both requires accessing
the data by &#8220;peeking under the mask&#8221;.</li>
<p>Another option is that described in the <a class="reference external" href="">alterNEP</a>, which is to implement
bitpattern dtypes with NA semantics for the &#8220;statistical missing data&#8221;
use case, and to also implement a totally independent API for masked
arrays with ignore semantics and all mask manipulation done explicitly
through a .mask attribute.</p>
<p>Another option would be to define a minimalist aligned array container
that holds multiple arrays and that can be used to pass them around
together. It would support indexing (to help with the common problem
of wanting to subset several arrays together without their becoming
unaligned), but all arithmetic etc. would be done by accessing the
underlying arrays directly via attributes. The &#8220;prior art&#8221; discussion
above suggests that something like this holding a .data and a .mask
array might actually be solve a number of people&#8217;s problems without
requiring any major architectural changes to numpy. This is similar to
a structured array, but with each field in a separately stored array
instead of packed together.</p>
<p>Several people have suggested that there should be a single system
that has multiple missing values that each have different semantics,
e.g., a MISSING value that has NA semantics, and a separate IGNORED
value that has ignored semantics.</p>
<p>None of these options are necessarily exclusive.</p>
<div class="section" id="the-debate">
<h2>The debate<a class="headerlink" href="#the-debate" title="Permalink to this headline">¶</a></h2>
<p>We both are dubious of using ignored semantics as a default missing
data behavior. <strong>Nathaniel</strong> likes NA semantics because he is most
interested in the &#8220;statistical missing data&#8221; use case, and NA semantics
are exactly right for that. <strong>Mark</strong> isn&#8217;t as interested in that use
case in particular, but he likes the NA computational abstraction
because it is unambiguous and well-defined in all cases, and has a
lot of existing experience to draw from.</p>
<p>What <strong>Nathaniel</strong> thinks, overall:</p>
<ul class="simple">
<li>The &#8220;statistical missing data&#8221; use case is clear and compelling; the
other use cases certainly deserve our attention, but it&#8217;s hard to say what
they <em>are</em> exactly yet, or even if the best way to support them is
by extending the ndarray object.</li>
<li>The &#8220;statistical missing data&#8221; use case is best served by an R-style
system that uses bitpattern storage to implement NA semantics. The
main advantage of bitpattern storage for this use case is that it
avoids the extra memory and speed overhead of storing and checking a
mask (especially for the common case of floating point data, where
some tricks with NaNs allow us to effectively hardware-accelerate
most NA operations). These concerns alone appears to make a
mask-based implementation unacceptable to many NA users,
particularly in areas like neuroscience (where memory is tight) or
financial modeling (where milliseconds are critical). In addition,
the bit-pattern approach is less confusing conceptually (e.g.,
assignment really is just assignment, no magic going on behind the
curtain), and it&#8217;s possible to have in-memory compatibility with R
for inter-language calls via rpy2. The main disadvantage of the
bitpattern approach is the need to give up a value to represent NA,
but this is not an issue for the most important data types (float,
bool, strings, enums, objects); really, only integers are
affected. And even for integers, giving up a value doesn&#8217;t really
matter for statistical problems. (Occupy Wall Street
notwithstanding, no-one&#8217;s income is 2**63 - 1. And if it were, we&#8217;d
be switching to floats anyway to avoid overflow.)</li>
<li>Adding new dtypes requires some cooperation with the ufunc and
casting machinery, but doesn&#8217;t require any architectural changes or
violations of numpy&#8217;s current orthogonality.</li>
<li>His impression from the mailing list discussion, esp. the <a class="reference external" href="">&#8220;what can
we agree on?&#8221; thread</a>, is that many users specifically
like the combination of masked storage, the mask being easily
accessible through the API, and ignored semantics. He could be
wrong, of course. But he cannot remember seeing anybody besides Mark
advocate for the specific combination of masked storage and NA
semantics, which makes him nervous.</li>
<li>Also, he personally is not very happy with the idea of having two
storage implementations that are almost-but-not-quite identical at
the Python level. While there likely are people who would like to
temporarily pretend that certain data is &#8220;statistically missing
data&#8221; without making a copy of their array, it&#8217;s not at all clear
that they outnumber the people who would like to use bitpatterns and
masks simultaneously for distinct purposes. And honestly he&#8217;d like
to be able to just ignore masks if he wants and stick to
bitpatterns, which isn&#8217;t possible if they&#8217;re coupled together
tightly in the API. So he would say the jury is still very much out
on whether this aspect of the NEP design is an advantage or a
disadvantage. (Certainly he&#8217;s never heard of any R users complaining
that they really wish they had an option of making a different
trade-off here.)</li>
<li>R&#8217;s NA support is a <a class="reference external" href="">headline feature</a> and its target audience
consider it a compelling advantage over other platforms like Matlab
or Python. Working with statistical missing data is very painful
without platform support.</li>
<li>By comparison, we clearly have much more uncertainty about the use
cases that require a mask-based implementation, and it doesn&#8217;t seem
like people will suffer too badly if they are forced for now to
settle for using numpy&#8217;s excellent mask-based indexing, the new
where= support, and even</li>
<li>Therefore, bitpatterns with NA semantics seem to meet the criteria
of making a large class of users happy, in an elegant way, that fits
into the original design, and where we can have reasonable certainty
that we understand the problem and use cases well enough that we&#8217;ll
be happy with them in the long run. But no mask-based storage
proposal does, yet.</li>
<p>What <strong>Mark</strong> thinks, overall:</p>
<ul class="simple">
<li>The idea of using NA semantics by default for missing data, inspired
by the &#8220;statistical missing data&#8221; problem, is better than all the
other default behaviors which were considered. This applies equally
to the bitpattern and the masked approach.</li>
<li>For NA-style functionality to get proper support by all numpy
features and eventually all third-party libraries, it needs to be
in the core. How to correctly and efficiently handle missing data
differs by algorithm, and if thinking about it is required to fully
support numpy, NA support will be broader and higher quality.</li>
<li>At the same time, providing two different missing data interfaces,
one for masks and one for bitpatterns, requires numpy developers
and third-party numpy plugin developers to separately consider the
question of what to do in either case, and do two additional
implementations of their code. This complicates their job,
and could lead to inconsistent support for missing data.</li>
<li>Providing the ability to work with both masks and bitpatterns through
the same C and Python programming interface makes missing data support
cleanly orthogonal with all other numpy features.</li>
<li>There are many trade-offs of memory usage, performance, correctness, and
flexibility between masks and bitpatterns. Providing support for both
approaches allows users of numpy to choose the approach which is
most compatible with their way of thinking, or has characteristics
which best match their use-case. Providing them through the same
interface further allows them to try both with minimal effort, and
choose the one which performs better or uses the least memory for
their programs.</li>
<li>Memory Usage<ul>
<li>With bitpatterns, less memory is used for storing a single array
containing some NAs.</li>
<li>With masks, less memory is used for storing multiple arrays that
are identical except for the location of their NAs. (In this case a
single data array can be re-used with multiple mask arrays;
bitpattern NAs would need to copy the whole data array.)</li>
<li>With bitpatterns, the floating point type can use native hardware
operations, with nearly correct behavior. For fully correct floating
point behavior and with other types, code must be written which
specially tests for equality with the missing-data bitpattern.</li>
<li>With masks, there is always the overhead of accessing mask memory
and testing its truth value. The implementation that currently exists
has no performance tuning, so it is only good to judge a minimum
performance level. Optimal mask-based code is in general going to
be slower than optimal bitpattern-based code.</li>
<li>Bitpattern integer types must sacrifice a valid value to represent NA.
For larger integer types, there are arguments that this is ok, but for
8-bit types there is no reasonable choice. In the floating point case,
if the performance of native floating point operations is chosen,
there is a small inconsistency that NaN+NA and NA+NaN are different.</li>
<li>With masks, it works correctly in all cases.</li>
<li>The bitpattern approach can work in a fully general way only when
there is a specific value which can be given up from the
data type. For IEEE floating point, a NaN is an obvious choice,
and for booleans represented as a byte, there are plenty of choices.
For integers, a valid value must be sacrificed to use this approach.
Third-party dtypes which plug into numpy will also have to
make a bitpattern choice to support this system, something which
may not always be possible.</li>
<li>The mask approach works universally with all data types.</li>
<div class="section" id="recommendations-for-moving-forward">
<h2>Recommendations for Moving Forward<a class="headerlink" href="#recommendations-for-moving-forward" title="Permalink to this headline">¶</a></h2>
<p><strong>Nathaniel</strong> thinks we should:</p>
<ul class="simple">
<li>Go ahead and implement bitpattern NAs.</li>
<li><em>Don&#8217;t</em> implement masked arrays in the core &#8211; or at least, not
yet. Instead, we should focus on figuring out how to implement them
out-of-core, so that people can try out different approaches without
us committing to any one approach. And so new prototypes can be
released more quickly than the numpy release cycle. And anyway,
we&#8217;re going to have to figure out how to experiment with such
changes out-of-core if numpy is to continue to evolve without
forking &#8211; might as well do it now. The existing code can live in
master, disabled, or it can live in a branch &#8211; it&#8217;ll still be there
once we know what we&#8217;re doing.</li>
<p><strong>Mark</strong> thinks we should:</p>
<ul class="simple">
<li>The existing code should remain as is, with a global run-time experimental
flag added which disables NA support by default.</li>
<p>A more detailed rationale for this recommendation is:</p>
<ul class="simple">
<li>A solid preliminary NA-mask implementation is currently in numpy
master. This implementation has been extensively tested
against scipy and other third-party packages, and has been in master
in a stable state for a significant amount of time.</li>
<li>This implementation integrates deeply with the core, providing an
interface which is usable in the same way R&#8217;s NA support is. It
provides a compelling, user-friendly answer to R&#8217;s NA support.</li>
<li>The missing data NEP provides a plan for adding bitpattern-based
dtype support of NAs, which will operate through the same interface
but allow for the same performance/correctness tradeoffs that R has made.</li>
<li>Making it very easy for users to try out this implementation, which
has reasonable feature coverage and performance characteristics, is
the best way to get more concrete feedback about how numpy&#8217;s missing
data support should look.</li>
<p>Because of its preliminary state, the existing implementation is marked
as experimental in the numpy documentation. It would be good for this
to remain marked as experimental until it is more fleshed out, for
example supporting struct and array dtypes and with a fuller set of
numpy operations.</p>
<p>I think the code should stay as it is, except to add a run-time global
numpy flag, perhaps numpy.experimental.maskna, which defaults to
False and can be toggled to True. In its default state, any NA feature
usage would raise an &#8220;ExperimentalError&#8221; exception, a measure which
would prevent it from being accidentally used and communicate its
experimental status very clearly.</p>
<p>The <a class="reference external" href=";">ABI issues</a> seem very tricky to deal with effectively in the 1.x
series of releases, but I believe that with proper implementation-hiding
in a 2.0 release, evolving the software to support various other
ABI ideas that have been discussed is feasible. This is the approach
I like best.</p>
<p><strong>Nathaniel</strong> notes in response that he doesn&#8217;t really have any
objection to shipping experimental APIs in the main numpy distribution
<em>if</em> we&#8217;re careful to make sure that they don&#8217;t &#8220;leak out&#8221; in a way
that leaves us stuck with them. And in principle some sort of &#8220;this
violates your warranty&#8221; global flag could be a way to do that. (In
fact, this might also be a useful strategy for the kinds of changes
that he favors, of adding minimal hooks to enable us to build
prototypes more easily &#8211; we could have some &#8220;rapid prototyping only&#8221;
hooks that let prototype hacks get deeper access to numpy&#8217;s internals
than we were otherwise ready to support.)</p>
<p>But, he wants to point out two things. First, it seems like we still
have fundamental questions to answer about the NEP design, like
whether masks should have NA semantics or ignore semantics, and there
are already plans to majorly change how NEP masks are exposed and
accessed. So he isn&#8217;t sure what we&#8217;ll learn by asking for feedback on
the NEP code in its current state.</p>
<p>And second, given the concerns about their causing (minor) ABI issues,
it&#8217;s not clear that we could really prevent them from leaking out. (He
looks forward to 2.0 too, but we&#8217;re not there yet.) So maybe it would
be better if they weren&#8217;t present in the C API at all, and the hoops
required for testers were instead something like, &#8216;we have included a
hacky pure-Python prototype accessible by typing &#8220;import
numpy.experimental.donttrythisathome.NEP&#8221; and would welcome feedback&#8217;?</p>
<p>If so, then he should mention that he did implement a horribly klugy,
pure Python implementation of the NEP API that works with numpy
1.6.1. This was mostly as an experiment to see how possible such
prototyping was and to test out a possible ufunc override mechanism,
but if there&#8217;s interest, the module is available here:
<a class="reference external" href=""></a></p>
<p>It passes the maskna test-suite, with some minor issues described
in a big comment at the top.</p>
<p><strong>Mark</strong> responds:</p>
<p>I agree that it&#8217;s important to be careful when adding new
features to numpy, but I also believe it is essential that the project
have forward development momentum. A project like numpy requires
developers to write code for advancement to occur, and obstacles
that impede the writing of code discourage existing developers
from contributing more, and potentially scare away developers
who are thinking about joining in.</p>
<p>All software projects, both open source and closed source, must
balance between short-term practicality and long-term planning.
In the case of the missing data development, there was a short-term
resource commitment to tackle this problem, which is quite immense
in scope. If there isn&#8217;t a high likelihood of getting a contribution
into numpy that concretely advances towards a solution, I expect
that individuals and companies interested in doing such work will
have a much harder time justifying a commitment of their resources.
For a project which is core to so many other libraries, only
relying on the good will of selfless volunteers would mean that
numpy could more easily be overtaken by another project.</p>
<p>In the case of the existing NA contribution at issue, how we resolve
this disagreement represents a decision about how numpy&#8217;s
developers, contributers, and users should interact. If we create
a document describing a dispute resolution process, how do we
design it so that it doesn&#8217;t introduce a large burden and excessive
uncertainty on developers that could prevent them from productively
contributing code?</p>
<p>If we go this route of writing up a decision process which includes
such a dispute resolution mechanism, I think the meat of it should
be a roadmap that potential contributers and developers can follow
to gain influence over numpy. Numpy development needs broad support
beyond code contributions, and tying influence in the project to
contributions seems to me like it would be a good way to encourage
people to take on tasks like bug triaging/management, continuous
integration/build server administration, and the myriad other
tasks that help satisfy the project&#8217;s needs. No specific meritocratic,
democratic, consensus-striving system will satisfy everyone, but the
vigour of the discussions around governance and process indicate that
something at least a little bit more formal than the current status
quo is necessary.</p>
<p>In conclusion, I would like the numpy project to prioritize movement
towards a more flexible and modular ABI/API, balanced with strong
backwards-compatibility constraints and feature additions that
individuals, universities, and companies want to contribute.
I do not believe keeping the NA code in 1.7 as it is, with the small
additional measure of requiring it to be enabled by an experimental
flag, poses a risk of long-term ABI troubles. The greater risk I see
is a continuing lack of developers contributing to the project,
and I believe backing out this code because these worries would create a
risk of reducing developer contribution.</p>
<div class="section" id="references-history">
<h2>References/history<a class="headerlink" href="#references-history" title="Permalink to this headline">¶</a></h2>
<p>The NEP describes Mark&#8217;s NA-semantics/mask
implementation/view based mask handling API:
<a class="reference external" href=""></a></p>
<p>The alterNEP was Nathaniel&#8217;s initial attempt at separating MISSING and
IGNORED handling into bit-patterns versus masks, though there&#8217;s a
bunch he would change about the proposal at this point:
<a class="reference external" href=""></a></p>
<p>miniNEP 2 was a later attempt by Nathaniel to sketch out an
implementation strategy for NA dtypes:
<a class="reference external" href=""></a></p>
<p>A discussion overview page is here:
<a class="reference external" href=""></a></p>
<div class="spc-rightsidebar span3">
<div class="sphinxsidebarwrapper">
a.extlink:after {
margin-left: 5px;
content: url("_static/external-link.png");
<div class="well sidebar-nav">
<ul class="nav nav-list">
<li >
<a href="index.html">About Numpy</a></li>
<li >
<a href="license.html">License</a></li>
<li >
<a href="old_array_packages.html">Old array packages</a></li>
<h3><a href="index.html">Table Of Contents</a></h3>
<li><a class="reference internal" href="#">Missing data: an orientation</a><ul>
<li><a class="reference internal" href="#the-numpy-developers-problem">The Numpy developers&#8217; problem</a></li>
<li><a class="reference internal" href="#prior-art">Prior art</a><ul>
<li><a class="reference internal" href="#statistical-missing-data">&#8220;Statistical missing data&#8221;</a></li>
<li><a class="reference internal" href="#na-semantics">NA Semantics</a></li>
<li><a class="reference internal" href="#other-prior-art">Other prior art</a></li>
<li><a class="reference internal" href="#semantics-storage-api-oh-my">Semantics, storage, API, oh my!</a></li>
<li><a class="reference internal" href="#designs-that-have-been-proposed">Designs that have been proposed</a></li>
<li><a class="reference internal" href="#the-debate">The debate</a></li>
<li><a class="reference internal" href="#recommendations-for-moving-forward">Recommendations for Moving Forward</a></li>
<li><a class="reference internal" href="#references-history">References/history</a></li>
<div class="container container-navbar-bottom">
<div class="spc-navbar">
<div class="container">
<div class="footer">
<div class="row-fluid">
<ul class="inline pull-left">
&copy; Copyright 2013 Numpy developers.
Created using <a href="">Sphinx</a> 1.1.3.
<script type="text/javascript">
//Handles menu drop down
$('.dropdown-menu').find('form').click(function (e) {
<script type="text/javascript">
//handles accordion arrow-up and down in pages
$('.accordion-group').on('show hide', function(e)
$('.accordion-heading').find('.accordion-toggle i').toggleClass('icon-arrow-down icon-arrow-up', 200);
Jump to Line
Something went wrong with that request. Please try again.