Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

The Filesystem is abnormally closed #17356

Open
gjhkael opened this issue Feb 25, 2022 · 0 comments
Open

The Filesystem is abnormally closed #17356

gjhkael opened this issue Feb 25, 2022 · 0 comments

Comments

@gjhkael
Copy link
Contributor

gjhkael commented Feb 25, 2022

PrestoFileSystemCache maintained the file system client in a map, when filesystem key is remove from the map, then FinalizerService will call the filesystem's close to cleanup resource. Because PrestoExtendedFileSystemCache overwrite the methed 'createPrestoFileSystemWrapper' and use HadoopExtendedFileSystem to wrapper the original Filesystem, But the open and create methods is not wrapper with InputStreamWrapper and OutputStreamWrapper which have the reference of HadoopExtendedFileSystem object. If somewhere use HadoopExtendedFileSystem to open a file, and if the filesystem object is removed from PrestoFileSystemCache, then is FinalizerService will close the filesystem, finally the input/output stream will encounter exception.

exception:

Caused by: java.io.IOException: Filesystem closed
		at org.apache.hadoop.hdfs.DFSClient.checkOpen(DFSClient.java:482)
		at org.apache.hadoop.hdfs.DFSInputStream.close(DFSInputStream.java:723)
		at java.base/java.io.FilterInputStream.close(FilterInputStream.java:180)
		at org.apache.hadoop.util.LineReader.close(LineReader.java:164)
		at org.apache.hadoop.mapred.LineRecordReader.close(LineRecordReader.java:291)

The code change should be:
(

public FSDataInputStream open(Path f, int bufferSize)
throws IOException
{
return fs.open(f, bufferSize);
}
)

to:

    @Override
    public FSDataInputStream open(Path f, int bufferSize)
            throws IOException
    {
        return new InputStreamWrapper(this.getRawFileSystem().open(f, bufferSize), this);
    }
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant