API: pseudo-public internals API for downstream libraries

[jbrockmendel]

Downstream libraries accessing core.internals causes headaches pretty regularly (#40149 (comment), #38134,). The proposal: define a public-ish stable-ish API to expose for these libraries, leaving us free-er to make changes in internals.

This exposes make_block. I expect we'll also need to expose one or both of create_block_manager_from_arrays/create_block_manager_from_blocks (comment in internals.__init__ says they are there for downstream xref #33892)

[jreback]

can you add a test which exercises this from a downstream author POV (e.g. create an manipulate things, e.g. like what pyarrow is doing), but only do this by importing the new api you are establishing so they can't cheat

[jreback]

ideally we remove 2 as soon as possible.

[jreback]

I would have these import to pandas.api.internals i think (then you can really control the exports)

[jbrockmendel]

i know pyarrow accesses the pd.core.internals namespace. not sure about others. we can ask them to change, but for the forseeable future will need these in the namespace.

[jreback]

right what i mean is let's expose a wrapper api namespace and then we can change the downstream packages when we have released.

[jbrockmendel]

fair enough. keep it in this file though? im wary of adding it to the pd.api namespace lest new downstream packages adopt bad habits

[jreback]

ok sure

[jbrockmendel]

can you add a test which exercises this from a downstream author POV (e.g. create an manipulate things, e.g. like what pyarrow is doing), but only do this by importing the new api you are establishing so they can't cheat

i think the way to do this is to rename the internal version, then our existing downstream/io tests will test for this

[jorisvandenbossche]

Thanks, it's probably a good idea to make a public-ish make_block, so you are more free to make changes/optimizations to the one used internally. Now, then we do need to add tests for this, because it won't longer be tested through our own test suite.

In pyarrow, we currently make use of BlockManager(..) constructor as well. And further, we access CategoricalBlock, DatetimeTZBlock etc, but only as object, eg for isinstance(block, ObjectBlock) (never calling the __init__ of those). Now, I suppose that we could remove this usage (instead of checking for CategoricalBlock, we could also check that block.values is a Categorical, for example)

[jreback]

can you add a test for this model that asserts the exact name that we expose, similar to https://github.com/pandas-dev/pandas/blob/master/pandas/tests/api/test_api.py

[jreback]

can you also assert the entire namespace of api itself e.g. [name for name in dir(api) if not name.startswith('_')] and then compare to a hard coded list.

[jbrockmendel]

i dont think we want to hard-code the existing list bc whats in there is somewhat haphazard. Once we hear back from all the relevant parties on #40226 we'll be able to make that list

[jreback]

i would fix it now with all of the symbols that are currently exposed

[jbrockmendel]

picking my battles; updated.