Add file writer and row group writer #149

sadikovi · 2018-08-16T07:18:05Z

This PR is final part of the core implementation of Parquet writes, adds:

FileWriter and its serialised version SerializedFileWriter.
RowGroupWriter and its serialised version SerializedRowGroupWriter.

I tried to maintain a fairly simple API. The idea and steps are as follows:

Create file writer based on file std::fs::File, schema TypePtr and writer properties WriterPropertiesPtr that you want to write.
Call file_writer.next_row_group to get a mutable reference to a row group writer to add data.
Call row_group_writer.next_column to add values to a column. Keep iterating until all columns have been written - we do not support partially written row group!
After that you can call close() on column writer and/or row group writer. They will be closed regardless, before either the next column is requested or the next row group is requested.
Keep adding row groups until you have added enough.
Call file_writer.close to write final metadata to a file.
That is it, after that you should have a complete file, so you can check schema and read records from it.

I added a set of unit tests. Please review them as well, they show API in action.

sadikovi · 2018-08-16T07:19:20Z

@sunchao Could you review this PR when you have time. I have several high-level questions and workarounds that I would like to discuss below:

I had to modify API of a column writer, so I can return a mutable reference and update it within row group writer. It is mainly around the fact that close takes &mut self now instead of mut self. My goal was mainly around the fact that I can store a reference to a current column writer in row group writer and call close() multiple times (user might call close, and we need to ensure that also close column writer in case she did not). Is it okay or should I look for other ways of implementing it somehow?
Please, check ergonomics of the new API. I tried my best to come up with easy to use and expressive API, but I might have missed something. Is API good enough?
We currently do not use limit on a row group size or DEFAULT_MAX_ROW_GROUP_SIZE from WriterProperties. Should we use it?

Let me know what you think. Thanks!

coveralls · 2018-08-16T07:40:43Z

Pull Request Test Coverage Report for Build 612

0 of 0 changed or added relevant lines in 0 files are covered.
36 unchanged lines in 3 files lost coverage.
Overall coverage increased (+0.1%) to 95.606%

Files with Coverage Reduction	New Missed Lines	%
column/writer.rs	3	95.93%
file/reader.rs	13	96.97%
file/writer.rs	20	95.91%

Totals
Change from base Build 610:	0.1%
Covered Lines:	12359
Relevant Lines:	12927

💛 - Coveralls

sunchao · 2018-08-21T16:37:50Z

@sadikovi : really sorry that I haven't got chance to look at this yet. Will take a look today.

sadikovi · 2018-08-21T17:33:51Z

No worries, take as much time as needed.

…

On Tue, 21 Aug 2018 at 6:37 PM, Chao Sun ***@***.***> wrote: @sadikovi <https://github.com/sadikovi> : really sorry that I haven't got chance to look at this yet. Will take a look today. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#149 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AHbY3vuyRRb6CI-LUG8TIF2iTgpCybYVks5uTDdegaJpZM4V_Uzh> .

sunchao · 2018-08-21T16:35:50Z

src/file/writer.rs

+  ///
+  /// This method finalises previous column writer, so make sure that all writes are
+  /// finished before calling this method.
+  fn next_column(&mut self) -> Result<&mut ColumnWriter>;


is it better to return a Option here? this more align with a iterator API.

I would have to return Result<Option<&mut ColumnWriter>>, is it okay?

Yes, I think this will be better. We should differentiate the error case and the case where there's no next.

Yes, I changed the code to return Result<Option<Something>> so looks like iterator and it is similar to how we read pages.

sunchao · 2018-08-21T16:48:15Z

src/file/writer.rs

+use util::io::{FileSink, Position};
+
+/// File writer interface.
+pub trait FileWriter {


nit: perhaps document more on how FileWriter should be used. For instance, how many row group writers should one create? does user need to explicitly call close the row group writers?

Yes, will do.

I updated the method docs to emphasise what you mentioned. There is a follow-up sub-task to update all of the documentation related to writes after we have merged the implementation.

OK. Sounds good then.

sunchao · 2018-08-22T05:15:26Z

src/file/writer.rs

+impl SerializedFileWriter {
+  /// Creates new file writer.
+  pub fn new(mut file: File, schema: TypePtr, properties: WriterPropertiesPtr) -> Self {
+    Self::start_file(&mut file).expect("Start Parquet file");


nit: the error message needs improvement.

I will update the new method to return Result<Self> similar to FileReader and improve the error message.

sunchao · 2018-08-22T06:34:19Z

src/file/writer.rs

+
+  #[inline]
+  fn next_column(&mut self) -> Result<&mut ColumnWriter> {
+    if self.row_group_metadata.is_some() {


nit: we can just call has_next_column at the beginning.

I am going to make it iterator-like, this condition will be updated.

sunchao · 2018-08-22T06:46:41Z

src/file/writer.rs

+  }
+
+  #[inline]
+  fn close(&mut self) -> Result<RowGroupMetaDataPtr> {


what if user calls close() before iterating over all columns. Will that cause some error?

Yes, it will raise an error, because there will be mismatch between expected columns (must be greater than 0) in the schema and actual number of columns written.

sunchao · 2018-08-22T06:52:53Z

src/file/writer.rs

+/// File writer interface.
+pub trait FileWriter {
+  /// Creates new row group from this file writer.
+  fn create_row_group(&mut self) -> Result<&mut RowGroupWriter>;


The good thing with this is we cannot call close until all the returned RowGroupWriter are out of scope. The downside is the restrictiveness: now it's hard to pass the RowGroupWriter to some struct which lives beyond the current call stack. Not sure if this will be an issue.

Yes, I had the same thought, but it looks like this is one of the few options to keep track of row group writers ourselves. Do you have other ideas? Should we just keep a reference count of row group writers and check later when asking for the next one?

Yes, I think another approach is to return a unique reference to the RowGroupWriter:

fn create_row_group(&mut self) -> Result<RowGroupWriter>;

And a separate function to explicitly close it:

fn close_row_group(&mut self, RowGroupWriter) -> Result<()>;

This will call finalise_row_group_writer to close the row group writer and set some flag in the file writer indicating that the current row group writer has been closed. We can do similar things for RowGroupWriter to make sure all column writers are handed back before the close is called.

Let me know what you think. This is just some rough thought.

Sounds good, I will make changes, see what it looks like, and let you know. Thanks!

sadikovi · 2018-08-24T07:27:52Z

@sunchao I addressed your comments, but might have missed some. Would you mind doing another review pass, including tests (I feel they are a bit overcomplicated)? And could you also give your thoughts on the questions I raised in my first comment?

Thanks!

sadikovi · 2018-08-29T20:59:28Z

@sunchao I made the changes to the API as per your suggestion. Can you have a look? Thanks!

Now we have next_XYZ and close_XYZ, we will raise an error if next method is called before close method is called for the previous XYZ.

Let me know if I should add more unit tests.

sunchao · 2018-08-31T08:46:14Z

@sadikovi : the latest PR looks good to me in general. However, the test test_column_writer_check_metadata is failing. Can you fix that?

sadikovi · 2018-08-31T08:47:39Z

Yes, thanks. I just checked the tests, will fix shortly, make some other minor changes and ping you again.

…

On Fri, 31 Aug 2018 at 10:46 AM, Chao Sun ***@***.***> wrote: @sadikovi <https://github.com/sadikovi> : the latest PR looks good to me in general. However, the test test_column_writer_check_metadata is failing. Can you fix that? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#149 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AHbY3uJ5jEm7WWh0bZ7u1q2WaJObELlLks5uWPfXgaJpZM4V_Uzh> .

sadikovi · 2018-08-31T11:08:21Z

@sunchao I fixed the test - it failed because I added test on encodings in this branch, but in master we have updated to add levels encoding as well, so it was failing with missing encoding for levels.

Should be all good now, would appreciate if you could look over the code again. There is also an open question about limiting row groups somehow, either based on size or number of rows - currently we do not limit them in any way. We could also add this as a follow-up issue.

sunchao · 2018-08-31T18:47:58Z

Should be all good now, would appreciate if you could look over the code again. There is also an open question about limiting row groups somehow, either based on size or number of rows - currently we do not limit them in any way. We could also add this as a follow-up issue.

Yes we should have some config properties for this. Good for a follow-up. :)

sunchao · 2018-08-31T18:48:13Z

Merged. Thanks @sadikovi !

sadikovi · 2018-09-01T07:54:07Z

Thanks!

…

On Fri, 31 Aug 2018 at 8:48 PM, Chao Sun ***@***.***> wrote: Merged. Thanks @sadikovi <https://github.com/sadikovi> ! — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#149 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AHbY3s1WzMAaaSJdyY6K9K4a31reIExWks5uWYTtgaJpZM4V_Uzh> .

sadikovi added 5 commits August 15, 2018 22:25

add row group writer and file writer

99bf7ad

update file writer

0e78864

update column writer and file writer, add tests

dcf0adb

update tests

d545201

rebase branch

b45ce7c

sadikovi mentioned this pull request Aug 16, 2018

Write support for Parquet (low-level writes) #116

Closed

9 tasks

sadikovi requested a review from sunchao August 16, 2018 15:24

sunchao reviewed Aug 22, 2018

View reviewed changes

address comments

4294f2e

sadikovi added 5 commits August 24, 2018 22:05

update doc comments

60526cf

revert changes

b9fb226

update file writer api

d0268b0

update tests

6464576

update test

b782b2e

sadikovi added 2 commits August 31, 2018 13:03

Merge branch 'master' into add-file-writer

a7197d2

update column writer test

a49649c

sunchao merged commit 9a46f6c into sunchao:master Aug 31, 2018

sadikovi deleted the add-file-writer branch September 1, 2018 07:54

xrl mentioned this pull request Oct 12, 2018

Implement Drop for RowGroupWriter, ColumnWriter, and friends #173

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add file writer and row group writer #149

Add file writer and row group writer #149

sadikovi commented Aug 16, 2018 •

edited

Loading

sadikovi commented Aug 16, 2018

coveralls commented Aug 16, 2018 •

edited

Loading

sunchao commented Aug 21, 2018

sadikovi commented Aug 21, 2018 via email

sunchao Aug 21, 2018

sadikovi Aug 23, 2018

sunchao Aug 25, 2018

sadikovi Aug 25, 2018

sunchao Aug 21, 2018

sadikovi Aug 23, 2018

sadikovi Aug 24, 2018

sunchao Aug 25, 2018

sunchao Aug 22, 2018

sadikovi Aug 23, 2018

sunchao Aug 22, 2018

sadikovi Aug 23, 2018

sunchao Aug 22, 2018

sadikovi Aug 23, 2018

sunchao Aug 22, 2018

sadikovi Aug 23, 2018

sunchao Aug 25, 2018

sadikovi Aug 25, 2018

sadikovi commented Aug 24, 2018

sadikovi commented Aug 29, 2018 •

edited

Loading

sunchao commented Aug 31, 2018

sadikovi commented Aug 31, 2018 via email

sadikovi commented Aug 31, 2018

sunchao commented Aug 31, 2018

sunchao commented Aug 31, 2018

sadikovi commented Sep 1, 2018 via email

Add file writer and row group writer #149

Add file writer and row group writer #149

Conversation

sadikovi commented Aug 16, 2018 • edited Loading

sadikovi commented Aug 16, 2018

coveralls commented Aug 16, 2018 • edited Loading

Pull Request Test Coverage Report for Build 612

💛 - Coveralls

sunchao commented Aug 21, 2018

sadikovi commented Aug 21, 2018 via email

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

sadikovi commented Aug 24, 2018

sadikovi commented Aug 29, 2018 • edited Loading

sunchao commented Aug 31, 2018

sadikovi commented Aug 31, 2018 via email

sadikovi commented Aug 31, 2018

sunchao commented Aug 31, 2018

sunchao commented Aug 31, 2018

sadikovi commented Sep 1, 2018 via email

sadikovi commented Aug 16, 2018 •

edited

Loading

coveralls commented Aug 16, 2018 •

edited

Loading

sadikovi commented Aug 29, 2018 •

edited

Loading