DRILL-5733: Unable to SELECT from parquet file with Hadoop 2.7.4 #1969
Conversation
| .map(filename -> new Path(p, filename)) | ||
| .collect(Collectors.toList()); | ||
| for (String filename : Metadata.OLD_METADATA_FILENAMES) { | ||
| // Read the older version of metadata file if the current version of metadata cache files do not exist. |
| return metaFilepaths; | ||
| } | ||
| metaFilepaths.clear(); | ||
| metaFilepaths.add(new Path(p, filename)); |
There was a problem hiding this comment.
Does this mean that for the last iteration we’ll add file name to metaFilepaths but will never check if it exists and return empty collection?
There was a problem hiding this comment.
No, we will check whether the metadata files from the last iteration exist below the loop and in this case return list with files or an empty collection if they do not exist.
There was a problem hiding this comment.
Ok, thanks. Missed check in the end.
vvysotskyi
force-pushed
the
DRILL-5733
branch
from
b0730d8
to
159a125
Compare
There was a problem hiding this comment.
@arina-ielchiieva, thanks for the review, I have addressed CR comments.
| .map(filename -> new Path(p, filename)) | ||
| .collect(Collectors.toList()); | ||
| for (String filename : Metadata.OLD_METADATA_FILENAMES) { | ||
| // Read the older version of metadata file if the current version of metadata cache files do not exist. |
| return metaFilepaths; | ||
| } | ||
| metaFilepaths.clear(); | ||
| metaFilepaths.add(new Path(p, filename)); |
There was a problem hiding this comment.
No, we will check whether the metadata files from the last iteration exist below the loop and in this case return list with files or an empty collection if they do not exist.
|
+1, LGTM. |
|
Sorry, missed a question about Hadoop 3. I have checked also with Hadoop 3.2.0 (previously was checked with Hadoop 2.8), and it works fine, even without this fix. |
DRILL-5733: Unable to SELECT from parquet file with Hadoop 2.7.4
Description
For the case when a single parquet file was selected, Drill built pass for metadata cache files like this is a directory. It caused errors for Hadoop. Added check to verify that the parent path is a directory before constructing a path for metadata cache files.
Documentation
NA
Testing
Checked manually on Apache Hadoop cluster, checked that fix doesn't break existing unit tests. No test is provided since the error is not reproduced even with MiniDFSCluster.