[Go][Parquet] Check number of rows written by current row writer #38506
Comments
zeroshade
pushed a commit
that referenced
this issue
Dec 8, 2023
…leWriter (#38507) ### Rationale for this change When using a chunked column reader to read from one Parquet file and a chunked column writer to write to another Parquet file, it can be useful to keep track of the number of rows written. ### What changes are included in this PR? This branch adds a new `RowGroupNumRows` method to the `pqarrow.FileWriter`. This is somewhat similar to the existing `RowGroupTotalBytesWritten` function. ### Are these changes tested? A new `file_writer_test.go` file is added that adds a test for the new method. ### Are there any user-facing changes? The new method is exported and documented. * Closes: #38506 Authored-by: Tim Schaub <tim@planet.com> Signed-off-by: Matt Topol <zotthewizard@gmail.com>
mapleFU
pushed a commit
to mapleFU/arrow
that referenced
this issue
Dec 13, 2023
…row.FileWriter (apache#38507) ### Rationale for this change When using a chunked column reader to read from one Parquet file and a chunked column writer to write to another Parquet file, it can be useful to keep track of the number of rows written. ### What changes are included in this PR? This branch adds a new `RowGroupNumRows` method to the `pqarrow.FileWriter`. This is somewhat similar to the existing `RowGroupTotalBytesWritten` function. ### Are these changes tested? A new `file_writer_test.go` file is added that adds a test for the new method. ### Are there any user-facing changes? The new method is exported and documented. * Closes: apache#38506 Authored-by: Tim Schaub <tim@planet.com> Signed-off-by: Matt Topol <zotthewizard@gmail.com>
clayburn
pushed a commit
to clayburn/arrow
that referenced
this issue
Jan 23, 2024
…row.FileWriter (apache#38507) ### Rationale for this change When using a chunked column reader to read from one Parquet file and a chunked column writer to write to another Parquet file, it can be useful to keep track of the number of rows written. ### What changes are included in this PR? This branch adds a new `RowGroupNumRows` method to the `pqarrow.FileWriter`. This is somewhat similar to the existing `RowGroupTotalBytesWritten` function. ### Are these changes tested? A new `file_writer_test.go` file is added that adds a test for the new method. ### Are there any user-facing changes? The new method is exported and documented. * Closes: apache#38506 Authored-by: Tim Schaub <tim@planet.com> Signed-off-by: Matt Topol <zotthewizard@gmail.com>
dgreiss
pushed a commit
to dgreiss/arrow
that referenced
this issue
Feb 19, 2024
…row.FileWriter (apache#38507) ### Rationale for this change When using a chunked column reader to read from one Parquet file and a chunked column writer to write to another Parquet file, it can be useful to keep track of the number of rows written. ### What changes are included in this PR? This branch adds a new `RowGroupNumRows` method to the `pqarrow.FileWriter`. This is somewhat similar to the existing `RowGroupTotalBytesWritten` function. ### Are these changes tested? A new `file_writer_test.go` file is added that adds a test for the new method. ### Are there any user-facing changes? The new method is exported and documented. * Closes: apache#38506 Authored-by: Tim Schaub <tim@planet.com> Signed-off-by: Matt Topol <zotthewizard@gmail.com>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Describe the enhancement requested
When using the
pqarrow.FileWriter, it would be nice to be able to check the number of rows that have been written by the current row group writer.Component(s)
Go, Parquet
The text was updated successfully, but these errors were encountered: