[Data] Making get_parquet_dataset configurable in # of fragments to scan

[alexeykudinkin]

Description

• Making get_parquet_dataset configurable in # of fragments to scan
• Minor clean ups

Related issues

Link related issues: "Fixes #1234", "Closes #1234", or "Related to #1234".

Additional information

Optional: Add implementation details, API changes, usage examples, screenshots, etc.

[gemini-code-assist]

Code Review

This pull request introduces a valuable feature to configure the number of fragments scanned for schema inference in get_parquet_dataset. The implementation is well-structured, involving a significant refactoring that correctly handles PyArrow version differences and schema promotion for null types. Additionally, the PR includes important correctness fixes for predicate evaluation and filesystem compatibility checks. My feedback is minor and focuses on a small code cleanup opportunity.

[gemini-code-assist]

The Union in Optional[Union["pyarrow.lib.Schema"]] is redundant. You can simplify this to Optional["pyarrow.lib.Schema"] for better readability.

Suggested change

	schema: Optional[Union["pyarrow.lib.Schema"]] = None,
	schema: Optional["pyarrow.lib.Schema"] = None,

[cursor]

Cursor Bugbot has reviewed your changes and found 1 potential issue.

^{Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, have a team admin enable autofix in the Cursor dashboard.}

[cursor]

Explicit schema parameter breaks dataset_kwargs containing schema

Low Severity

The new pq.ParquetDataset call passes schema=schema explicitly alongside **dataset_kwargs. Previously, schema was not an explicit keyword argument, so users could pass "schema" inside dataset_kwargs and it would be forwarded correctly. Now, if dataset_kwargs contains a "schema" key, it will raise TypeError: got multiple values for keyword argument 'schema'. This is a regression even though dataset_kwargs is deprecated.