[python-package] Add support for NumPy 2.0, test against nightly versions of dependencies (fixes #6454)

[jameslamb]

Fixes #6454.

This PR makes lightgbm compatible with numpy 2.x

• converts np.array(..., copy=False) calls (which are no longer supported in NumPy 2.x) with np.asarray(...).
• adds a CI job testing lightgbm against nightlies of matplotlib, numpy, pandas, pyarrow, scipy, and scikit-learn

Also fixes some other smalls things I noticed thanks to the new CI job.

• makes it clearer when test_arrow.py tests are failing because pyarrow is installed but cffi is not
• skips on test on pandas>=3.0 (which is not release yet), to deal with a change to pd.DataFrame(...)'s copying behavior when built from a numpy array

Notes for Reviewers

What does it mean that np.array(..., copy=False) is "no longer supported"?

Raises an exception if you try to ask it to do an impossible cast.

import numpy as np

X = np.array([1, 2, 3, 4], dtype=np.float32)
np.array(X, dtype=np.float64, copy=False)

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ValueError: Unable to avoid copy while creating an array as requested.
If using `np.array(obj, copy=False)` replace it with `np.asarray(obj)` to allow a copy when needed (no behavior change in NumPy 1.x).
For more details, see https://numpy.org/devdocs/numpy_2_0_migration_guide.html#adapting-to-changes-in-the-copy-keyword.

So lightgbm is going to start creating unnecessary copies again on pandas input, if you use numpy 2.0`?

No.

pandas is going to start creating copies on NumPy input under some conditions. Follow the conversation at pandas-dev/pandas#58913.

cc @jmoralez, since you did so much work on this e.g. in #4927 and #5612

[jameslamb]

I found myself in a situation where pyarrow was installed but cffi was not.

That caused this import to fail:

LightGBM/python-package/lightgbm/compat.py

Line 212 in e0cda88

from pyarrow.cffi import ffi as arrow_cffi

which led to all of the pyarrow imports being defined with the placeholders like this:

LightGBM/python-package/lightgbm/compat.py

Lines 227 to 231 in e0cda88

	class pa_ChunkedArray: # type: ignore
	"""Dummy class for pa.ChunkedArray."""
	def __init__(self, *args: Any, **kwargs: Any):
	pass

which led to every Arrow test failing in a hard-to-understand way.

pytest tests/python_package_test/test_arrow.py

==== short test summary info ====
FAILED tests/python_package_test/test_arrow.py::test_dataset_construct_fuzzy[<lambda>-dataset_params0] - AssertionError: assert False
... 
FAILED tests/python_package_test/test_arrow.py::test_predict_ranking - TypeError: Wrong type(ChunkedArray) for label.
==== 139 failed in 11.77s ====

As of this change, it'll fail with a slightly clearer error message (and earlier, without running all the tests).

E   AssertionError: 'pyarrow' and its dependencies must be installed to run the arrow tests

[jameslamb]

This is one that I missed in #6458, noticed it when replicating some of the configuration for the new CI job here

[guolinke]

Thank you!

[jameslamb]

Thanks very much for the reviews! I'm very excited to now have this CI job testing against nightlies of lightgbm's dependencies. I hope it won't be too noisy, and will help us catch issues before any lightgbm users do.