[Python][CI] Failures on latest/nightly pandas

[AlenkaF]

Describe the bug, including details regarding any error messages, version, and platform.

The tests on the CI started failing with the new release of pandas (19 jan 2024, version 2.2.0) with:

=================================== FAILURES ===================================
__________________ test_array_protocol_pandas_extension_types __________________

monkeypatch = <_pytest.monkeypatch.MonkeyPatch object at 0x7fbb44648c40>

    def test_array_protocol_pandas_extension_types(monkeypatch):
        # ARROW-7022 - ensure protocol works for Period / Interval extension dtypes
    
        storage = pa.array([1, 2, 3], type=pa.int64())
        expected = pa.ExtensionArray.from_storage(DummyExtensionType(), storage)
    
        monkeypatch.setattr(pd.arrays.PeriodArray, "__arrow_array__",
                            PandasArray__arrow_array__, raising=False)
        monkeypatch.setattr(pd.arrays.IntervalArray, "__arrow_array__",
                            PandasArray__arrow_array__, raising=False)
        for arr in [pd.period_range("2012-01-01", periods=3, freq="D").array,
                    pd.interval_range(1, 4).array]:
            result = pa.array(arr)
            assert result.equals(expected)
            result = pa.array(pd.Series(arr))
            assert result.equals(expected)
            result = pa.array(pd.Index(arr))
            assert result.equals(expected)
            result = pa.table(pd.DataFrame({'a': arr})).column('a').chunk(0)
>           assert result.equals(expected)
E           assert False
E            +  where False = <bound method Array.equals of <pyarrow.lib.Int64Array object at 0x7fbb446b7690>\n[\n  1,\n  2,\n  3\n]>(<pyarrow.lib.ExtensionArray object at 0x7fbb446b7a10>\n[\n  1,\n  2,\n  3\n])
E            +    where <bound method Array.equals of <pyarrow.lib.Int64Array object at 0x7fbb446b7690>\n[\n  1,\n  2,\n  3\n]> = <pyarrow.lib.Int64Array object at 0x7fbb446b7690>\n[\n  1,\n  2,\n  3\n].equals

usr/local/lib64/python3.11/site-packages/pyarrow/tests/test_pandas.py:4161: AssertionError
___________________ test_table_factory_function_args_pandas ____________________

    @pytest.mark.pandas
    def test_table_factory_function_args_pandas():
        import pandas as pd
    
        # from_pandas not accepting names or metadata:
>       with pytest.raises(ValueError):
E       Failed: DID NOT RAISE <class 'ValueError'>

See:
https://github.com/ursacomputing/crossbow/actions/runs/7604845464/job/20708281882
https://github.com/ursacomputing/crossbow/actions/runs/7604845540/job/20708282208

Component(s)

Python

[jorisvandenbossche]

Hmm, the reason for the failure is (I suppose) related to the fact that I added __arrow_c_stream__ to pd.DataFrame.

[AlenkaF]

Ah, thanks for the info! I was a bit lost to be honest =)
I guess a change in the ordering of the defined method options when converting would do?

[AlenkaF]

Here:

arrow/python/pyarrow/table.pxi

Line 5205 in c33ffb0

elif hasattr(data, "__arrow_c_stream__"):

?