Thread Safety for parquet::arrow::FileReader::GetRecordBatch

[pjones]

I posted the following on the Zulip chat and Antoine asked me to repost it here:

Is it safe for multiple threads to use the same parquet::arrow::FileReader object and each call GetRecordBatchReader with different row group IDs separately?

Also, how can I figure this out for myself in the future so I don't have to bug others?

Thank you!

[pitrou]

@wgtmac @adamreeve @HuaHuaY @mapleFU If you have a more precise idea than I have :)

[wgtmac]

I don't think we can do this blindly. Codex has spotted evidence like below:

---

With the current default settings, concurrent calls to GetRecordBatchReader() on the same FileReader are not safe. ArrowReaderProperties::pre_buffer defaults to true, and each call invokes ParquetFileReader::PreBuffer(). That method replaces SerializedFile::cached_source_ and clears/rebuilds prebuffered_column_chunks_, while GetRowGroup() reads both without synchronization. Different row-group IDs do not avoid the race because this state is file-wide.

If pre-buffering is disabled before constructing the FileReader, then separate RecordBatchReader instances for different row groups appear safe to consume concurrently in the current implementation, provided that:

• the FileReader outlives all returned readers;
• its properties are not modified concurrently;
• each RecordBatchReader is consumed by only one thread; and
• the underlying RandomAccessFile follows the thread-safe ReadAt() contract.

[HuaHuaY]

It sounds to me like this has nothing to do with whether the row group IDs of multiple RecordBatchReaders overlap. I suspect it depends on the actual capabilities supported by the InputStream. If we were to make a thread-safety guarantee regarding the interface, it would likely be: "Whether the result of GetRecordBatchReader is thread-safe depends on the safety of certain underlying parameters."