[C++] exception caused by dataset_writer.cc:587: Check failed: (largest) != (nullptr)

[pawelz188]

Describe the bug, including details regarding any error messages, version, and platform.

The exception is repeated every time and
can be triggered by e.g. modifying the example by modification of CreateExampleParquetHivePartitionedDataset() example in arrow/cpp/examples/arrow/dataset_documentation_example.cc
by setting the recording parameters:

  write_options.base_dir = base_path;
  write_options.partitioning = partitioning;
  write_options.basename_template = "part{i}.parquet";
+   write_options.max_open_files = 2;
+   write_options.max_rows_per_file = 2;
+   write_options.max_rows_per_group = 2;
  ARROW_RETURN_NOT_OK(ds::FileSystemDataset::Write(write_options, scanner));
  return base_path;

Output error message:
vcpkg\buildtrees\arrow\src\e-arrow-13-0a65b46298.clean\cpp\src\arrow\dataset\dataset_writer.cc:587: Check failed: (largest) != (nullptr)

Exception is triggered by DCHECK_NE(largest, nullptr) in CloseLargestFile() procedure in dataset_writter.cc.

The exception call indicates a synchronization problem when arrow is closing files and a limit on the number of open files is reached.

Component(s)

C++

[pawelz188]

My local commit with changes described above:
0829fd5
that triggered exception.

[kou]

Thanks for your report.
Do you want to work on this?

[mapleFU]

Seems this is because the writer would like to close a file, but all in-flight file has written zero rows...🤔

I can fix this if you're not willing to work on this

[pawelz188]

Please fix it, I am not yet expert in arrow.
Now I'm trying to make a patch to fix the problem but I'm not sure of the impact on the overall code.

[mapleFU]

I'll submit a fixing. I'm in National Day so maybe slow to reply

[pawelz188]

This patch eliminates the problem of exception occurrence.
It seems that the files are handled correctly.
Please confirm the impact of the patch on the dataset at overall.
CloseLargestFile-nullptr-fix.patch

[pawelz188]

I'll submit a fixing. I'm in National Day so maybe slow to reply

If You submit a fix I will test it within my scope.

[mapleFU]

#38030 I've submit a basic fixing here, will accept review and maybe working more after national holiday :-)