WIP/ENH: Writer

[martindurant]

I don't know if you saw this, but I attempted to make a parquet writer. It works well enough to make parquet files which the same reader can read. The files cannot be read by true parquet tools (e.g., spark...), however. Any thoughts on what is missing?

[jcrobak]

Awesome! will try to take a look this weekend.

[martindurant]

@jcrobak have you have a look through what I did here?
You maybe want to take out the vectorised reading and py3 adaptation (see #15) for considered merger and leave only the writing stuff in WIP.

[martindurant]

... or anybody else out there?
I don't think my code is far off a real and fast parquet writer for simple column types - any experts can tell me what I'm doing wrong?

[peterbe]

How are you supposed to use it? I'm a newbie. I tried:

>>> grouped = pd.DataFrame([{'foo':'A', 'bar':'B'}, {'foo':'C', 'bar':'D'}], columns=['foo', 'bar'])
>>> df_to_parquet(grouped, '/tmp/foo.parquet')
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "parquet/writer.py", line 42, in df_to_parquet
    rle_string = make_rle_string(nrows, 1)
  File "parquet/writer.py", line 87, in make_rle_string
    header_bytes = make_unsigned_var_int(header)
  File "parquet/writer.py", line 82, in make_unsigned_var_int
    return bit + result.to_bytes(1, 'little')
AttributeError: 'int' object has no attribute 'to_bytes'

[peterbe]

You'll get a NameError here if typ not in ('int64','int32','float64').

[martindurant]

Correct, only int32/64 and float64 are supported. When I say supported, I couldn't get anything to read the parquet files created except the reader right here... which is why WIP, and not recommended for newbies. The apache feather/arrow shows promise for providing parquet usage for pandas, I stopped trying here because of the forthcoming work described here: http://wesmckinney.com/blog/pandas-and-apache-arrow/

[jcrobak]

Thanks again for your work @martindurant. I haven't had a chance to debug the write path, but I'm certainly interested in trying to.

The main reason I haven't tried to integrate this code is that I believe there's value in a pure python implementation with few dependencies. For instance, folks tend to use the avro python tools for quick data checks locally. pip install take a few seconds and can be used to dump small files. I think that's the primary value of this tool in its current shape, since it's far from production-ready.

For high-performance/production, Wes' arrow + cpp bindings will likely be the best bet... and will integrate with pandas/numpy AFAICT. I'm also interested in pandas/numpy bindings, but not as part of the core implementation (not sure how much code reuse their could be...).

I'm going to close this PR for now so that folks have a better sense of viability. Hopefully someone will come along at some point and rebase on top of all the api refactors. I'll certainly try to do so if I ever have some spare cycles!