ARROW-10134: [Python][Dataset] Add ParquetFileFragment.num_row_groups

[bkietz]

No description provided.

[github-actions]

https://issues.apache.org/jira/browse/ARROW-10134

[jorisvandenbossche]

I think the main reason such a property would be interesting for dask's use case is to get the number of row groups in a case all statistics are not available / not yet parsed. So the way that this PR returns None in that case is not super useful, I think.
I think ideally if the number of row groups is not know (the row_groups are not set), it would retrieve this information from the FileMetaData.

[bkietz]

@jorisvandenbossche just confirming: you want f.num_row_groups to potentially perform IO?

[bkietz]

@jorisvandenbossche PTAL

[jorisvandenbossche]

you want f.num_row_groups to potentially perform IO?

Yes, that's indeed the consequence for now (if the metadata was not yet parsed before). Long term I would like us to cache the metadata, though, without the need to necessarily directly parse all statistics etc (https://issues.apache.org/jira/browse/ARROW-10131-

[jorisvandenbossche]

Could you add test for the case I commented about? I think this should do it (didn't run the code though):

@pytest.mark.parquet
def test_parquet_fragment_num_row_groups(tempdir):
    import pyarrow.parquet as pq

    table = pa.table({'a': range(8)})
    pq.write_table(table, tempdir / "test.parquet", row_group_size=2)
    dataset = ds.dataset(tempdir / "test.parquet", format="parquet")
    original_fragment = list(dataset.get_fragments())[0]

    # create fragment with subset of row groups
    fragment = original_fragment.format.make_fragment(
        original_fragment.path, original_fragment.filesystem,
          row_groups=[1, 3])
    assert fragment.num_row_groups == 2
    # ensure that parsing metadata preserves correct number of row groups
    fragment.ensure_complete_metadata()
    assert fragment.num_row_groups == 2
    assert len(fragment.row_groups) == 2

[bkietz]

CI failure is unrelated. Merging