[C++][Parquet] Interface total_bytes_written is Confusing

[mapleFU]

Describe the enhancement requested

Hi, all, when using parquet's total_bytes_written, I found it really confusing.

In RowGroupWriter:

class PARQUET_EXPORT RowGroupWriter {
 public:
  // Forward declare a virtual class 'Contents' to aid dependency injection and more
  // easily create test fixtures
  // An implementation of the Contents class is defined in the .cc file
  struct Contents {
    virtual ~Contents() = default;
    virtual int num_columns() const = 0;
    virtual int64_t num_rows() const = 0;

    // to be used only with ParquetFileWriter::AppendRowGroup
    virtual ColumnWriter* NextColumn() = 0;
    // to be used only with ParquetFileWriter::AppendBufferedRowGroup
    virtual ColumnWriter* column(int i) = 0;

    virtual int current_column() const = 0;
    virtual void Close() = 0;

    // total bytes written by the page writer
    virtual int64_t total_bytes_written() const = 0;
    // total bytes still compressed but not written
    virtual int64_t total_compressed_bytes() const = 0;

    virtual bool buffered() const = 0;
  };

1. Syntax of total_compressed_bytes() is clear
2. total_bytes_written means "total bytes written by page writer".

And in ColumnWriter, the interface says:

  /// \brief The total size of the compressed pages + page headers. Some values
  /// might be still buffered and not written to a page yet
  virtual int64_t total_compressed_bytes() const = 0;

  /// \brief The total number of bytes written as serialized data and
  /// dictionary pages to the ColumnChunk so far
  virtual int64_t total_bytes_written() const = 0;

In ColumnWriterImpl, it's:

  // Records the total number of uncompressed bytes written by the serializer
  int64_t total_bytes_written_;

  // Records the current number of compressed bytes in a column
  int64_t total_compressed_bytes_;

So, in fact, the total_bytes_written() in RowGroup just means the uncompressed bytes it output? Is there any interface for compressed bytes? In non-buffer mode, we can just call sink_.Tell(), but in buffer mode, how can we get compressed-output length?

Component(s)

C++, Parquet

[mapleFU]

@pitrou @wjones127 Hi, would you mind me add a compressed_written_bytes api? Or did I miss some api?

[wjones127]

I'm don't think you are missing any API, although I am not sure the use case for exposing it during the write. Why not call Tell() on the sink after flushing?

[mapleFU]

I'm don't think you are missing any API, although I am not sure the use case for exposing it during the write. Why not call Tell() on the sink after flushing?

Because we have "buffered" page writer.

[wgtmac]

I'm don't think you are missing any API, although I am not sure the use case for exposing it during the write. Why not call Tell() on the sink after flushing?

I think compressed_written_bytes would be useful if we want to limit compressed size of row group if we buffer row group in the parquet writer.

For now we can only get size of flushed row groups via Tell() and raw (uncompressed) size of buffered row group via total_bytes_written(). As we cannot precisely estimate the compression ratio, it is not easy to limit the actual row group size after compression.

[mapleFU]

@wgtmac @wjones127

To detail explain this problem, let me assume a schema: (a: required int, b: required int).

Assume the data below:

a:  | Dict Page | Page 1 |  Page 2 | Unbuffered Written Values |
b:  | Page 1 |  Page 2 | Page 3 | Unbuffered Written Values |

So, we have a RowGroupWriter, and two ColumnWriter here. Each column writer holds a PageWriter.

There are two kinds of page writer:

1. SerializedPageWriter, which just write "compressed page" to sink.
2. BufferedPageWriter, which buffer the writing pages, and write multiple compressed page to sink.

So, assume we want to get "currently buffered value size" and "Unbuffered estimated size", sink_.Tell() will not be enough, because we have buffered page writer, which haven't write to sink.

We still have interface in this way, yes, total_bytes_written. But, as you can see, it's just "uncompressed size". For example, assume column b uses DELTA_BINARY_PACKED, it size would be 50B, but the total_bytes_written may be 5KB, which cannot tell use the size it written

[wjones127]

As we cannot precisely estimate the compression ratio, it is not easy to limit the actual row group size after compression.

That makes sense. It sounds like you want to be able to get the data size while writing and not wait until you have finished and flushed all data. That makes sense. 👍