-
Notifications
You must be signed in to change notification settings - Fork 1.6k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[Kernel] Use LogStore
s in listFrom
implementation in default FileSystemClient
#2770
Conversation
...rnel-defaults/src/main/java/io/delta/kernel/defaults/internal/logstore/LogStoreProvider.java
Outdated
Show resolved
Hide resolved
|
||
/** | ||
* Get the {@link LogStore} instance for the given schema and configuration. Callers can set | ||
* {code io.delta.kernel.logStore.class} to specify the LogStore implementation to use. If not |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
why support both io.delta.kernel.logStore.class
and io.delta.kernel.logStore.<scheme>.impl
?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This should be io.delta.kernel.logStore.<scheme>.impl
. Updated the javadoc
## Description Prep for #2770 PR
## Description Build changes from PR: #2770
1e6a1c0
to
c5b22ca
Compare
Description
LogStore
s instorage
module have file system operations (needed for reading/writing DeltaLogs) implemented for each storage (e.g. s3, GCS etc.) to take into account of the behavior of storage and also efficiently implement certain operations depending upon the storage system support (e.g. fast listing in S3).This PR creates
LogStoreProvider
to get the specific implementation of theLogStore
for givenscheme
. Also updates theDefaultFileSystemClient.listFrom
to use theLogStore.listFrom
.The majority of the code here is copied from the
delta-spark
andstandalone
projects.How was this patch tested?
Unittests for
LogStoreProvider
and existing integration tests forDefaultFileSystemClient.listFrom
changes.