[common] Forward close through PluginFileIO and ResolvingFileIO - #9472
[common] Forward close through PluginFileIO and ResolvingFileIO#9472wombatu-kun wants to merge 1 commit into
Conversation
| @Override | ||
| public SeekableInputStream newInputStream(Path path) throws IOException { | ||
| return fileIO().newInputStream(path); | ||
| Lease lease = acquire(); |
There was a problem hiding this comment.
Why need to modify RESTTokenFileIO?
|
I think this PR can be simplified and kept focused:
Forwarding Removing the listener preserves the previous effective behavior for the |
|
After reconsidering the scope and current priority, I do not think this PR is worth continuing at this point. The original issue has not been accepted for now, so the benefit does not seem large enough to justify introducing and maintaining this fairly broad lifecycle change, especially with the additional iterator and compatibility concerns discussed above. I would prefer to stop here and close this PR rather than split it or expand the scope further. We can revisit the problem if the original issue is accepted and prioritized later, ideally with a smaller design. |
Purpose
Split out of #8962 at @JingsongLi's request. Answers this review comment.
PluginFileIOandResolvingFileIOnever forwardedclose()to their delegates, unlikeCachingFileIO, soOSSFileIO.close()has never run andRESTCatalog'stry (FileIO fileIO = fileIOFromOptions(path))releases nothing. Both forward now, andclose()is terminal so the wrappers cannot silently rebuild a delegate nobody will release.That also makes
RESTTokenFileIO's removal listener close a delegate for real. Its cache is JVM wide and hands out rawFileIOs and streams, so an eviction could close an OSS file system while another table is still reading through it. Values are reference counted now: the listener hands back only the cache's own reference, and the delegate is closed when the last lease goes. Leases cover each operation and the lifetime of every stream and listing handed out, and the caller's lease is taken before the put, since the admission policy can evict a just-inserted entry.BaseMultiPartUploadCommitter,LanceUtilsandVortexUtilswork inside a lease instead of unwrapping the cached instance.fileIO()is deprecated, since a raw reference carries no lifetime to track, but kept because engines cast the result to their own implementation.Cut from current master and shares no file with #8962 or #9471, so the three can be reviewed in any order.
Tests
RESTTokenFileIOTest(new): an eviction keeps a leasedFileIOalive and closes an unleased one; an open input, output or two-phase stream keeps its delegate alive until the stream ends; the wrapper carriesVectoredReadablethrough and does not fake it; a failed open hands the lease back; a spent entry is rebuilt; a lease closed twice releases once; the multipart committer holds a lease across the call.PluginFileIOTest,ResolvingFileIOTest: close releases every delegate under the plugin classloader, keeps going when one fails, and use after close is rejected.