[Data] Improve handling of pandas.NA

[bveeramani]

Why are these changes needed?

This change standardizes null handling across internal block formats by converting pd.NA to None. Since different formats (e.g., Pandas and Arrow) use different null representations, this ensures consistent semantics when converting between them. In particular, we treat pd.NA as a null value, while preserving np.nan as a distinct floating-point NaN.

Related issue number

Checks

• I've signed off every commit(by using the -s flag, i.e., git commit -s) in this PR.
• I've run scripts/format.sh to lint the changes in this PR.
• I've included any doc changes needed for https://docs.ray.io/en/master/.
  • I've added any new APIs to the API Reference. For example, if I added a
    method in Tune, I've added it in doc/source/tune/api/ under the
    corresponding .rst file.
• I've made sure the tests are passing. Note that there might be a few flaky tests, see the recent failures at https://flakey-tests.ray.io/
• Testing Strategy
  • Unit tests
  • Release tests
  • This PR is not tested :(

[Copilot]

Pull Request Overview

This PR enhances how pandas.NA values are handled by converting them to Python None when rows are materialized as dictionaries.

• Adds a new test verifying pd.NA → None conversion in iter_rows.
• Makes TableRow.as_pydict an abstract method.
• Implements as_pydict in the pandas block accessor to translate pd.NA to None.

Reviewed Changes

Copilot reviewed 3 out of 3 changed files in this pull request and generated no comments.

File	Description
python/ray/data/tests/test_pandas_block.py	Added test_iter_rows_with_na to cover pd.NA → None conversion
python/ray/data/_internal/row.py	Marked as_pydict as an abstract method on TableRow
python/ray/data/_internal/pandas_block.py	Implemented as_pydict to map pd.NA values to None

Comments suppressed due to low confidence (3)

python/ray/data/tests/test_pandas_block.py:465

• The comment mentions 'NaN values' but this test is for pd.NA. Please update it to 'NA values' for clarity.

# We should return None for NaN values.

python/ray/data/_internal/pandas_block.py:113

• Type annotations Dict and Any are used here but not imported in this module; add from typing import Any, Dict at the top to prevent a NameError.

    def as_pydict(self) -> Dict[str, Any]:

python/ray/data/_internal/row.py:34

• [nitpick] Using an Ellipsis stub in an abstract method can mask missing implementations. Consider raising NotImplementedError or using pass to make the intent explicit.

        ...