Parquet Writer

[xiaodaigh]

No description provided.

[tanmaykm]

Thanks for initiating this. This seems like a good start. Any idea why the CI has failed?
I had a quick look and have added a few suggestions.
Will be looking at it more in the next few days.

[xiaodaigh]

Thanks for initiating this. This seems like a good start. Any idea why the CI has failed?
I had a quick look and have added a few suggestions.
Will be looking at it more in the next few days.

Seems to be failing due to nonmissingtype which is not available in Base prior to Julia 1.3. Added. Hope no further issues. Missings.jl is added to deps for this reason.

I added CategoricalyArrays.jl purely so that I can use it to tell users it's not supported, please let me know if you think it's ok.

Update:
It's passing all tests now.

[xiaodaigh]

Resolved conflict with master

[xiaodaigh]

Broke by #81 #82

Tried to accommodate #81 but #82 is definitely a bug, as R and Python can read it OK.

[tanmaykm]

I feel we should have a write API that allows data to be written iteratively. Something like:

parwriter = ParFileWriter(path)

for chunk in table_chunks # chunk being a table itself
    write(parwriter, chunk)
end

Each iteration will add a row group to the file. And the file metadata can be updated at the end of the iteration. Looks like the underlying functions are already there here. So it should not be difficult. What do you think?

[xiaodaigh]

What do you think?

I am ok with that in addition to an API for writing a full table. Cos the table_chunks have to come from somewhere, so for ppl to use it they would need a way to chunk tables, which is rarely done. I'd say 90% of use cases are where the user just provides a Table so it's most straightforward to just write it.

How about this? We merge this one first, if no objection to the write API as it is, then in a future release we write this chunk based writer (to cover what I guess is just 10% of use cases).

[xiaodaigh]

More serious issue is that tests broke by #82

[xiaodaigh]

Merged latest changes from master and passing all tests. If there are no further changes requested from the writer, can I suggest we merge this. And implement the chunk based writer in the next release?

[tanmaykm]

👍 will merge in a bit

[tanmaykm]

Oops, should have squashed and merged. Maybe I should rebase on the master, else it will become too unwieldy?

[tanmaykm]

doesn't look possible now

[xiaodaigh]

Maybe I should rebase on the master, else it will become too unwieldy?

Feel free. To me it's not that big a deal. But it can be harder to pin down an issue