ENH: Overhaul of NumPy lib namespace [NEP 52]

[mtsokol]

Hi @rgommers @ngoldbaum,

This issue is meant for monitoring and discussing cleanup of np.lib namespace. (Tracking parent issue: #23999)

Main goal is to make each public function, which lives in np.lib, available either from the main namespace np or "local lib" namespace e.g. np.lib.npyio. As a result, there will be only one way to access a specific public function in NumPy.
To achieve it, for each relevant lib submodule, there will be a private file (that starts with _) imported by the main namespace, and a public file that allows to be accessed as a public submodule.

# Example:

np.broadcast_to  # imported from numpy/lib/_stride_tricks.py

np.lib.stride_tricks.normalize_axis_tuple  # imported from numpy/lib/stride_tricks.py

Let's call this mechanism "main or submodule namespace access". (maybe a better name?)

---

The scope of the milestone:

"ns" stands for "namespace". ☑️ means that this submodule has been refactored.

• lib PR
  main ns: N/A
  local ns: NumpyVersion, Arrayterator, add_newdoc, add_docstring, tracemalloc_domain
• array_utils PR
  main ns: N/A
  local ns: byte_bounds, discover_array_parameters, normalize_axis_index, normalize_axis_tuple
• histograms PR
  main ns: histogram, histogramdd, histogram_bin_edges
  local ns:
• type_check PR
  main ns: iscomplexobj, isrealobj, imag, iscomplex, isreal, nan_to_num, real, real_if_close, typename, mintypecode, common_type
  local ns:
• nanfunctions PR
  main ns: nansum, nanmax, nanmin, nanargmax, nanargmin, nanmean, nanmedian, nanpercentile, nanvar, nanstd, nanprod, nancumsum, nancumprod, nanquantile
  local ns:
• function_base PR
  main ns: select, piecewise, trim_zeros, copy, iterable, percentile, diff, gradient, angle, unwrap, sort_complex, flip, rot90, extract, place, vectorize, asarray_chkfinite, average, bincount, digitize, cov, corrcoef, median, sinc, hamming, hanning, bartlett, blackman, kaiser, i0, meshgrid, delete, insert, append, interp, quantile
  local ns:
• shape_base PR
  main ns: column_stack, row_stack, dstack, array_split, split, hsplit, vsplit, dsplit, apply_over_axes, expand_dims, apply_along_axis, kron, tile, get_array_wrap, take_along_axis, put_along_axis
  local ns:
• arraypad PR
  main ns: pad
  local ns:
• arraysetops PR
  main ns: ediff1d, intersect1d, setxor1d, union1d, setdiff1d, unique, isin
  local ns:
• twodim_base PR
  main ns: diag, diagflat, eye, fliplr, flipud, tri, triu, tril, vander, histogram2d, mask_indices, tril_indices, tril_indices_from, triu_indices, triu_indices_from
  local ns:
• ufunclike PR
  main ns: fix, isneginf, isposinf
  local ns:
• utils PR
  main ns: get_include, info, show_runtime
  local ns:
• npyio PR
  main ns: savetxt, loadtxt, genfromtxt, load, save, savez, savez_compressed, packbits, unpackbits, fromregex,
  local ns: DataSource, NpzFile, BagObj
• polynomial PR
  main ns: poly, roots, polyint, polyder, polyadd, polysub, polymul, polydiv, polyval, poly1d, polyfit
  local ns:
• stride_tricks PR
  main ns: broadcast_to, broadcast_arrays, broadcast_shapes
  local ns: as_strided, sliding_window_view
• index_tricks PR
  main ns: diag_indices_from, diag_indices, fill_diagonal, ndindex, ndenumerate, ix_, ogrid, mgrid, unravel_index, ravel_multi_index, s_, c_, r_, index_exp
  local ns:

This work should be less disturbing than main namespace cleanup, as I think a function is rarely imported from np.lib if it's already available in np.

[mhvk]

@mtsokol - be careful - lib.stride_tricks is meant to remain (I didn't check all others, but this is one I use (too?) regularly, and the contents are not all exported; see NEP).

[mtsokol]

@mtsokol - be careful - lib.stride_tricks is meant to remain (I didn't check all others, but this is one I use (too?) regularly, and the contents are not all exported; see NEP).

Hi @mhvk! Sure, that's correct.
For example np.lib.stride_tricks.as_strided or normalize_axis_tuple are the ones that are public and available only from the submodule.

The ones from lib.stride_tricks that are being exported and will be accessible only from main namespace are broadcast_to, broadcast_arrays and broadcast_shapes.

[mhvk]

Yes, as_strided remains a favourite... There's also sliding_window_view (but normalize_axis_tuple comes from elsewhere).

[rgommers]

Focusing on what remains or becomes a public name would be useful. stride_tricks indeed, but there is more. array_utils is new. What about index_tricks, I think we were considering keeping that and exposing some of the more niche or advanced indexing related utilities there?

The checklist of what to make private is useful too, but that's more a todo list. I'd suggest going into more detail on what will be officially public. And then on what the TBDs are or potential pain points. If some current .py file contains only objects that are in the main namespace already, then it's easy - that .py file does not need to be a public module. When that's not the case, we have to surface the changes.

[rgommers]

I am not sure, but I suspect that normalize_axis_tuple being in stride_tricks was an accident - it has nothing to do with strides, and I'd expect it in another place (array_utils maybe?).

[mhvk]

normalize_axis_tuple is imported from numpy.core.numeric, no worries about that (though there was a separate question of where it should be exposed - but not here for sure!)

[mtsokol]

The checklist of what to make private is useful too, but that's more a todo list. I'd suggest going into more detail on what will be officially public. And then on what the TBDs are or potential pain points. If some current .py file contains only objects that are in the main namespace already, then it's easy - that .py file does not need to be a public module. When that's not the case, we have to surface the changes.

@rgommers Sure! I updated the PR's description.
For each submodule, I defined what goes to the main namespace and what goes to the local one, as of now. It looks that the concept of public items available only in the local namespace applies only to stride_tricks, index_tricks and utils (and future array_utils).

All "main ns" public items are in the agreed "keep" list, so main namespaces shouldn't shrink. I think it's only local namespaces that we will discuss then.

[rgommers]

Thanks Mateusz, that's a clear list. There really is very little in "local ns", so basically we're hiding almost everything that's there now and have more or less a clean slate to add new niche functionality that doesn't qualify for inclusion in the main namespace. That'll be quite nice, and has been on our wish list for a long time.

Maybe add local ns items for numpy.lib itself? I think that there's a couple of items in it, like NumpyVersion and Arrayterator.

[ngoldbaum]

It looks like there's still two minor usages of import * in numpy/lib/__init__.py that could be replaced but other than that this is done. Going to close this as I don't think those usages are bad, they're doing import * from modules with an __all__.