[data] Add orc datasource

[laysfire]

Description

Support read orc format

Related issues

Link related issues: "Fixes #1234", "Closes #1234", or "Related to #1234".

Additional information

Optional: Add implementation details, API changes, usage examples, screenshots, etc.

[gemini-code-assist]

Warning

Gemini encountered an error creating the review. You can try again by commenting /gemini review.

[laysfire]

/gemini review

[gemini-code-assist]

Code Review

This pull request introduces a concrete DataSourceV2 implementation for ORC files (OrcDatasourceV2) with worker-side stripe chunking. It adds OrcFileChunker, OrcFileReader, OrcScanner, OrcInMemorySizeEstimator, and exposes the public read_orc API. Feedback on the implementation suggests optimizing the file reading loop in OrcFileReader by checking if the row limit has already been reached before opening subsequent files, thereby avoiding unnecessary I/O overhead.

[gemini-code-assist]

To avoid unnecessary file opens and I/O overhead, we should check if the row limit has already been reached before calling _read_path for subsequent files in the outer loop.

Suggested change

	for path, chunk_metadatas in self._iter_contiguous_path_groups(input_split):
	for table in self._read_path(path, chunk_metadatas):
	for path, chunk_metadatas in self._iter_contiguous_path_groups(input_split):
	if self._limit is not None and rows_read >= self._limit:
	return
	for table in self._read_path(path, chunk_metadatas):

[github-actions]

This pull request has been automatically marked as stale because it has not had
any activity for 14 days. It will be closed in another 14 days if no further activity occurs.
Thank you for your contributions.

You can always ask for help on our discussion forum or Ray's public slack channel.

If you'd like to keep this open, just leave any comment, and the stale label will be removed.

[cursor]

Cursor Bugbot has reviewed your changes using default effort and found 1 potential issue.

^{Reviewed by Cursor Bugbot for commit 4ba3909. Configure here.}

[cursor]

Path filter runs too early

Medium Severity

_read_stripe applies _predicate before _append_synthetic_columns. Filters that reference the synthesized path column (when include_paths=True) see null placeholders first, so matching rows are dropped incorrectly.

Additional Locations (1)

• python/ray/data/_internal/datasource_v2/readers/orc_file_reader.py#L230-L255

 

^{Reviewed by Cursor Bugbot for commit 4ba3909. Configure here.}

[richardliaw]

@laysfire thanks for the contribution. we're currently merging #37891 , which should be for V1. It'd be great to rebase on top of that when done.