Writing Partitioned Datasets recipe for Python #47
Conversation
There was a problem hiding this comment.
I understand that you are using the dummy "chunk" column to be able to have a partitioned dataset without having a natural column to partition on, but my feeling is that this is a bit of a workaround because we don't give a better alternative (yet).
Personally, I would maybe use a more useful partitioning column as the main example, and then mention the "chunk" column as a way to do it if there is no column already present that can be used for partitioning (and then we can easily update this when a better API becomes available)
Switched to a real column ( |
|
@jorisvandenbossche can you re-review? Should have addressed your concerns |
python/source/io.rst
Outdated
|
|
||
| Arrow will partition datasets in subdirectories by default, which will | ||
| result in 10 different directories named with the value of the partitioning | ||
| column and with file containing the data partition inside: |
There was a problem hiding this comment.
the "and with file containing the data partition inside" reads a bit strange. Maybe something like "each with a file containing the subset of the data for that partition"
Fixes #46