[Go][Parquet] Inaccurate RowGroupTotalCompressedBytes/RowGroupTotalBytesWritten with go parquet file writer

[matthewmcnew]

Describe the bug, including details regarding any error messages, version, and platform.

There does not appear to be an accurate way to identify or estimate the size of the current row group with pqarrow.FileWriter.

RowGroupTotalCompressedBytes()provides the total bytes from created data pages but, when the dictionary page size limit is reached the buffered data pages are flushed and the total size is reset to "0". This means the RowGroupTotalCompressedBytes will only provide the size of pages created after the dictionary page size was reached. Ideally the size the TotalCompressedBytes size should include all created data pages.

RowGroupTotalBytesWritten() will provide the total bytes of DataPages when they are written but, not if the the page is buffered due to the dictionary page still being created. This causes the RowGroupTotalBytesWritten to inaccurately provide a "0" bytes estimate until the dictionary page size limit is reached.

Perhaps related to: #39789.

Component(s)

Go

[zeroshade]

Thanks for tracking down the cause! It's likely related if not directly the cause of the issue you linked.

Is this something you think you'd be willing to contribute a PR for? I don't have the bandwidth immediately, so it'll take me a week or so to dig into this. But I can definitely review any PRs

[mapleFU]

About totalBytesWritten, I've summarize similiar interface in C++:
#33897 (comment)

Don't know if there're better solutions. Let me fix RowGroupTotalCompressedBytes in another separate issue

[zeroshade]

@matthewmcnew Did #39922 fix this issue? or is it still exhibiting this problem?

[matthewmcnew]

#39922 should fix the inaccuracy with RowGroupTotalCompressedBytes being reset.

However, it would still be useful to estimate buffered data pages within RowGroupTotalBytesWritten. Would an approach like this make sense? To include the buffered data pages within the size of RowGroupTotalBytesWritten?

[zeroshade]

@matthewmcnew That makes sense to me

[matthewmcnew]

@zeroshade #40105 should be ready for review now.