[Train] Handle Arrow-backed pandas dtypes in LightGBM examples

[pseudo-rnd-thoughts]

Description

#63017 updated Ray Data's Arrow-to-pandas conversion to preserve Arrow-backed pandas dtypes, such as int64[pyarrow], so dtypes can roundtrip more faithfully.

This exposed an incompatibility with LightGBM's pandas input path. Ray Train's LightGBM examples and legacy trainer code convert Ray Data shards to pandas before constructing lightgbm.Dataset. With Arrow-backed Ray Data inputs, those pandas DataFrames can now contain Arrow-backed dtypes, and LightGBM rejects them during pandas dtype validation even when the logical column type is numeric.

This PR updates the LightGBM paths to normalize pandas DataFrames to NumPy-nullable pandas dtypes before passing them to LightGBM. It also updates documentation and examples to show the same conversion for user-authored LightGBM training loops.

Changes

• Normalize pandas DataFrames in the legacy LightGBMTrainer path before constructing lightgbm.Dataset.
• Update LightGBM docs and docstring examples to call convert_dtypes(dtype_backend="numpy_nullable") after to_pandas().
• Add a regression test covering Arrow-backed Ray Data input from ray.data.from_items(...).
• Keep the restore test focused on trainer restore behavior by using a pandas-backed test dataset.
• Update the LightGBM release benchmark to avoid passing Arrow-backed pandas dtypes to LightGBM.

[gemini-code-assist]

Code Review

This pull request introduces the use of convert_dtypes() across LightGBM trainer implementations, documentation examples, and tests to ensure compatibility with Arrow-backed pandas DataFrames from Ray Data. It also re-enables the test_trainer_restore test. The review feedback recommends simplifying the code by removing redundant materialize() calls and omitting the explicit dtype_backend argument to maintain compatibility with pandas versions prior to 2.0.0. Additionally, the reviewer suggests applying these fixes to the legacy LightGBM trainer and warns about potential performance overhead when using convert_dtypes() on very large datasets.

[cursor]

Cursor Bugbot has reviewed your changes and found 1 potential issue.

^{Reviewed by Cursor Bugbot for commit 17abd91. Configure here.}

[cursor]

Float16 arrow dtype causes TypeError in normalize function

Low Severity

pa.types.is_floating returns True for float16 (half-float) Arrow types, so the function generates "Float16" as the target pandas dtype. However, pandas only supports Float32 and Float64 as nullable float extension types — there is no Float16. This causes a TypeError at the df.astype(dtype_mapping) call if any column has pd.ArrowDtype(pa.float16()). The pa.types.is_float16() check could be used to skip or handle this case separately.

 

^{Reviewed by Cursor Bugbot for commit 17abd91. Configure here.}