[Python] avoid using pandas internals

[jbrockmendel]

• [Python] pandas internals: avoid using DatetimeTZBlock  #38341
• [Python] pandas internals: Passing a manager object to DataFrame() constructor is deprecated #38342

---

Describe the bug, including details regarding any error messages, version, and platform.

ATM pyarrow passes a BlockManager to the pd.DataFrame constructor, but in doing so accesses pandas functions/classes that are not public and that some pandas maintainers (me) would like to ween arrow off of (xref pandas-dev/pandas#52419).

@jorisvandenbossche tells me the current usage is performance motivated, particularly (but not exclusively?) the performance hit associated with pandas silently consolidating, which it no longer does in 2.0.

Let's see if we can find an alternative using pandas' public API. Starting bid: is pd.DataFrame.from_arrays a viable alternative?

Component(s)

Python

[graingert]

pandas 2.2.0.dev0+390.g59d4e84128 raises a DeprecationWarning: Passing a BlockManager to DataFrame is deprecated and will raise in a future version. Use public APIs instead for this use now:

import pyarrow.dataset
import fsspec

paths = [
    "https://github.com/Parquet/parquet-compatibility/raw/master/parquet-testdata/impala/1.1.1-NONE/nation.impala.parquet"
]
(
    pyarrow.dataset.dataset(paths, filesystem=fsspec.filesystem("http"))
    .schema.empty_table()
    .to_pandas()
)

this landed in pandas-dev/pandas#52419

[jorisvandenbossche]

@graingert yes, although you shouldn't see that as an end user, fixing that on the pandas side (-> pandas-dev/pandas#52419 (comment))

You will see the warning when enabling all warnings to show or error (like when running tests), so we have to switch to a different API in pyarrow.

[jorisvandenbossche]

Issue resolved by pull request 40897
#40897