Skip to content

Fix for parquet file schemes#2667

Merged
jakirkham merged 1 commit intodask:masterfrom
martindurant:parquet_file_schema
Sep 8, 2017
Merged

Fix for parquet file schemes#2667
jakirkham merged 1 commit intodask:masterfrom
martindurant:parquet_file_schema

Conversation

@martindurant
Copy link
Copy Markdown
Member

No description provided.

if append:
pf = fastparquet.api.ParquetFile(path, open_with=myopen, sep=sep)
if pf.file_scheme != 'hive':
if pf.file_scheme not in ['hive', 'empty', 'flat']:
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is there a "flat" scheme somewhere? Am only finding "empty" and "simple". Is it suppose to be "simple"?

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

"flat" is new, when there are multiple files, but all are in one directory

@jakirkham
Copy link
Copy Markdown
Member

The change seems clean and fine to me from a general perspective, but admittedly I don't know lots about parquet. Trusting your judgement and merging.

@jakirkham jakirkham merged commit 74ecf25 into dask:master Sep 8, 2017
@jakirkham
Copy link
Copy Markdown
Member

jakirkham commented Sep 8, 2017

Just realized I forgot we merge squash. Would it make sense to just make this the default (or only) kind of merge in the repo settings?

@martindurant
Copy link
Copy Markdown
Member Author

There was only one commit, squash would not have made a difference

@mrocklin
Copy link
Copy Markdown
Member

mrocklin commented Sep 8, 2017 via email

@jcrist
Copy link
Copy Markdown
Member

jcrist commented Sep 8, 2017

Just switched off merge-commits for this repo so this can't happen again.

@jakirkham
Copy link
Copy Markdown
Member

Great! Thanks so much @jcrist. Sorry for the trouble.

@martindurant martindurant deleted the parquet_file_schema branch September 29, 2017 12:45
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants