-
Notifications
You must be signed in to change notification settings - Fork 136
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Remove file reading responsibility from loaders #737
Merged
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
23856c5
to
36a4ca8
Compare
36a4ca8
to
744df73
Compare
collindutter
requested changes
Apr 8, 2024
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Nice work!
collindutter
requested changes
Apr 9, 2024
vasinov
requested changes
Apr 9, 2024
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Looks great overall!
collindutter
approved these changes
Apr 9, 2024
vasinov
approved these changes
Apr 9, 2024
Merged
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
We will want to create a
BaseFileManagerDriver
to enable theFileManager
Tool to interact with alternative (e.g. non-local) filesystems. We'd like to be able to reuse the existing loaders to load in artifacts from files, though some of the loaders used by theFileManager
currently do not support loading files from memory, and thus would subclasses ofBaseFileManagerDriver
to write to a file before invoking the loader.To make this easier, this change will remove the responsibility of reading from the filesystem from all loaders. Since this is the only reason the
FileLoader
loader exists, we will remove that.Changes:
IO
andPath
type options as source. All loaders that are related to specific file formats will acceptbytes
as source (andstr
for text file formats).FileLoader
FileManager
to have no default loader and to return aBlobArtifact
by default if there is no default loader. This is essentially the same behavior. (The default used to beFileLoader
with no args, which would always return aBlobArtifact
and if you changed the default loader to beFileLoader
withencoding
set, then it would be equivalent to passing aTextLoader
for the default)to_key
method onBaseLoader
with a reasonable implementation and moveload_collection
implementation in to theBaseLoader
. The idea here is to encapsulate the part ofload_collection
that varies. This allows there to be a single implementation that just submitsload
to the executor for each source.load_collection
for sources that are listed more than once in the input (or at least that have the same key).