New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
PARQUET-1599: Fix to-avro to respect the overwrite option #650
Conversation
@@ -96,9 +97,8 @@ public int run() throws IOException { | |||
|
|||
Path outPath = qualifiedPath(outputPath); | |||
FileSystem outFS = outPath.getFileSystem(getConf()); | |||
if (overwrite && outFS.exists(outPath)) { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
SchemaCommand has the same problem.
Instead of throwing an exception here if the file already exists, how about adding a new method to BaseCommand for no overwrite case, and using that when creating the file in this command (and in SchemaCommand)?
Something like this: a new createWithNoOverwrite, which calls the private create with an extra overwrite=false flag, and it will use fs.create with this parameter value. How does this sound? We can also remove this delete file section in this case, and just call createWithNoOverwrite when overwrite is true, and call create otherwise.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thank you for the review @nandorKollar, that sounds good to me.
I'll update the PR a bit later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@nandorKollar Sorry for my long silence, just addressed your comments :)
Rebased on master and resolve conflicts. |
Rebased on master and resolved conflicts. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM, one small comment
for (Record record : reader) { | ||
writer.append(record); | ||
count += 1; | ||
try (OutputStream os = overwrite ? |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I think we can combine the two try
's here
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Sorry that I missed your comment @Fokko! Just updated the PR.
I've retriggered the failing build, it looked unrelated. |
Thanks @sekikn ! |
Make sure you have checked all steps below.
Jira
Tests
My PR adds the following unit tests OR does not need testing for this extremely good reason:
Commits
Documentation