Avoid redundant stat call when writing Parquet files #1746
Conversation
| if (!abort) { | ||
| length = target.getFileSystem(conf).getFileStatus(target).getLen(); | ||
| } | ||
| length = fileWriter.getPos(); |
There was a problem hiding this comment.
Previously, we were getting the size by doing a stat call, which wouldn't work after an abort, since the file would have been deleted. But now we just ask the writer how much it wrote, which works in any case. (it's probably not useful in the case of an abort, but it doesn't hurt anything, and makes the code more consistent)
There was a problem hiding this comment.
[...] but it doesn't hurt anything
can this depend in impl of the writer?
can a writer still do some non O(1) call underneath?
(when aborting, you need to be defensive, especially that this code path is less frequent and less tested)
There was a problem hiding this comment.
Given this is ParquetRecordWriterUtil I would assume we only have one writer implementation to care about, so we just check.
There was a problem hiding this comment.
I'll leave the check because it doesn't hurt. Hopefully this will be gone soon, once we merge the native Parquet writer code.
| if (!abort) { | ||
| length = target.getFileSystem(conf).getFileStatus(target).getLen(); | ||
| } | ||
| length = fileWriter.getPos(); |
There was a problem hiding this comment.
Given this is ParquetRecordWriterUtil I would assume we only have one writer implementation to care about, so we just check.
No description provided.