You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
PrestoFileSystemCache maintained the file system client in a map, when filesystem key is remove from the map, then FinalizerService will call the filesystem's close to cleanup resource. Because PrestoExtendedFileSystemCache overwrite the methed 'createPrestoFileSystemWrapper' and use HadoopExtendedFileSystem to wrapper the original Filesystem, But the open and create methods is not wrapper with InputStreamWrapper and OutputStreamWrapper which have the reference of HadoopExtendedFileSystem object. If somewhere use HadoopExtendedFileSystem to open a file, and if the filesystem object is removed from PrestoFileSystemCache, then is FinalizerService will close the filesystem, finally the input/output stream will encounter exception.
exception:
Caused by: java.io.IOException: Filesystem closed
at org.apache.hadoop.hdfs.DFSClient.checkOpen(DFSClient.java:482)
at org.apache.hadoop.hdfs.DFSInputStream.close(DFSInputStream.java:723)
at java.base/java.io.FilterInputStream.close(FilterInputStream.java:180)
at org.apache.hadoop.util.LineReader.close(LineReader.java:164)
at org.apache.hadoop.mapred.LineRecordReader.close(LineRecordReader.java:291)
PrestoFileSystemCache maintained the file system client in a map, when filesystem key is remove from the map, then FinalizerService will call the filesystem's close to cleanup resource. Because PrestoExtendedFileSystemCache overwrite the methed 'createPrestoFileSystemWrapper' and use HadoopExtendedFileSystem to wrapper the original Filesystem, But the open and create methods is not wrapper with InputStreamWrapper and OutputStreamWrapper which have the reference of HadoopExtendedFileSystem object. If somewhere use HadoopExtendedFileSystem to open a file, and if the filesystem object is removed from PrestoFileSystemCache, then is FinalizerService will close the filesystem, finally the input/output stream will encounter exception.
exception:
The code change should be:
(
presto/presto-hive-common/src/main/java/org/apache/hadoop/fs/HadoopExtendedFileSystem.java
Lines 125 to 129 in e01d5ca
to:
The text was updated successfully, but these errors were encountered: