[CI][Python] Nightly test for PySpark 3.2.0 fail with AttributeError on numpy.bool

[raulcd]

Describe the bug, including details regarding any error messages, version, and platform.

Nightly integration tests with PySpark 3.2.0 are failing with the following error:
test-conda-python-3.8-spark-v3.2.0

ERROR: test_with_key_complex (pyspark.sql.tests.test_pandas_cogrouped_map.CogroupedMapInPandasTests)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/spark/python/pyspark/sql/tests/test_pandas_cogrouped_map.py", line 160, in test_with_key_complex
    result = self.data1 \
  File "/spark/python/pyspark/sql/pandas/conversion.py", line 168, in toPandas
    pandas_type = PandasConversionMixin._to_corrected_pandas_type(field.dataType)
  File "/spark/python/pyspark/sql/pandas/conversion.py", line 238, in _to_corrected_pandas_type
    return np.bool
  File "/opt/conda/envs/arrow/lib/python3.8/site-packages/numpy/__init__.py", line 284, in __getattr__
    raise AttributeError("module {!r} has no attribute "
AttributeError: module 'numpy' has no attribute 'bool'

Component(s)

Continuous Integration, Python

[raulcd]

cc @AlenkaF @jorisvandenbossche

[AlenkaF]

Numpy deprecated the use of np.bool in version 1.24:

• NumPy 1.24.0 release notes
• DEP: Next step in scalar type alias deprecations/futurewarnings numpy/numpy#22607
• https://github.com/numpy/numpy/pull/22607/files#diff-43b932a4beb6c6bc01b8cbb341d52dbc3d6d89c8f41ff044f66146489697171c

the CI build for Nightly integration tests with PySpark 3.2.0 is using this (1.24) version of NumPy, but the fix in Apache Spark has already been merged: apache/spark#37817.

Am continuing to look into it ...

[AlenkaF]

... ah yes, the version of PySpark is 3.2.0 in the CI build (could just look at the title, not the CI setup :D).

I guess we should change numpy version till the new release of PySpark is out with the fix?

[jorisvandenbossche]

I guess we should change numpy version till the new release of PySpark is out with the fix?

That sounds correct

[AlenkaF]

Great, will make a PR.