FIX-#6936: Fix read_parquet when dataset is created with to_parquet and index=False

[anmyachev]

What do these changes do?

• first commit message and PR title follow format outlined here

  NOTE: If you edit the PR title to match this format, you need to add another commit (even if it's empty) or amend your last commit for the CI job that checks the PR title to pick up the new PR title.

• passes flake8 modin/ asv_bench/benchmarks scripts/doc_checker.py
• passes black --check modin/ asv_bench/benchmarks scripts/doc_checker.py
• signed commit with git commit -s
• Resolves read_parquet failed to read the dataset created by to_parquet with index = False. #6936
• tests added and passing
• module layout described at docs/development/architecture.rst is up-to-date

[anmyachev]

In case of index=False the list is empty

[YarShev]

We do the same in parquet_dispatcher.py. Am I right that we will have correct columns for the Modin DataFrame but wrong columns for partitions?

[anmyachev]

The assigned columns will be propagated to partitions by synchronize_labels function.

[YarShev]

Should we do this in parquet_dispatcher.py like with sync_index flag?

[anmyachev]

IIRC pyarrow handled this on its end by calling .to_pandas and using pandas metadata:

modin/modin/core/storage_formats/pandas/parsers.py

Lines 747 to 758 in 56fb47e

	return (
	ParquetFile(f)
	.read_row_groups(
	range(
	row_group_start,
	row_group_end,
	),
	columns=columns,
	use_pandas_metadata=True,
	)
	.to_pandas(**to_pandas_kwargs)
	)

[YarShev]

Why do we need then this line in the test if internal and external column labels match?

[anmyachev]

In this case, the index is not saved, the pyarrow cannot obtain the same index type as the original dataframe had, but since I need to compare the original and read dataframe, I explicitly cast the type.

The fact that Modin reads the dataframe in this case with the another type of index is not Modin’s problem, the same behavior occurs in pandas. This is simply a consequence of the fact that due to the lack of metadata, the index type is lost.

Why do we need then this line in the test if internal and external column labels match?

Both the internal type and the external type (of the columns) are the same, but they do not match what was in the original dataframe.

[YarShev]

I see, thanks for the explanation!