Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Remove file reading responsibility from loaders #737

Merged
merged 5 commits into from
Apr 9, 2024

Conversation

dylanholmes
Copy link
Contributor

@dylanholmes dylanholmes commented Apr 8, 2024

We will want to create a BaseFileManagerDriver to enable the FileManager Tool to interact with alternative (e.g. non-local) filesystems. We'd like to be able to reuse the existing loaders to load in artifacts from files, though some of the loaders used by the FileManager currently do not support loading files from memory, and thus would subclasses of BaseFileManagerDriver to write to a file before invoking the loader.

To make this easier, this change will remove the responsibility of reading from the filesystem from all loaders. Since this is the only reason the FileLoader loader exists, we will remove that.

Changes:

  • Remove IO and Path type options as source. All loaders that are related to specific file formats will accept bytes as source (and str for text file formats).
  • Remove FileLoader
  • Edit FileManager to have no default loader and to return a BlobArtifact by default if there is no default loader. This is essentially the same behavior. (The default used to be FileLoader with no args, which would always return a BlobArtifact and if you changed the default loader to be FileLoader with encoding set, then it would be equivalent to passing a TextLoader for the default)
  • Add to_key method on BaseLoader with a reasonable implementation and move load_collection implementation in to the BaseLoader. The idea here is to encapsulate the part of load_collection that varies. This allows there to be a single implementation that just submits load to the executor for each source.
  • Remove duplicate loads in load_collection for sources that are listed more than once in the input (or at least that have the same key).
  • Edit CHANGELOG

@dylanholmes dylanholmes force-pushed the feature/loaders-bytes branch 2 times, most recently from 23856c5 to 36a4ca8 Compare April 8, 2024 22:50
@dylanholmes dylanholmes marked this pull request as ready for review April 8, 2024 23:11
@dylanholmes dylanholmes requested review from collindutter, vasinov and andrewfrench and removed request for vasinov April 8, 2024 23:11
Copy link
Member

@collindutter collindutter left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice work!

griptape/loaders/csv_loader.py Show resolved Hide resolved
griptape/loaders/email_loader.py Show resolved Hide resolved
griptape/loaders/text_loader.py Show resolved Hide resolved
griptape/loaders/csv_loader.py Outdated Show resolved Hide resolved
griptape/loaders/text_loader.py Outdated Show resolved Hide resolved
griptape/tools/file_manager/tool.py Outdated Show resolved Hide resolved
griptape/utils/futures.py Outdated Show resolved Hide resolved
griptape/loaders/base_loader.py Show resolved Hide resolved
griptape/loaders/csv_loader.py Outdated Show resolved Hide resolved
griptape/loaders/text_loader.py Outdated Show resolved Hide resolved
griptape/tools/file_manager/tool.py Outdated Show resolved Hide resolved
Copy link
Member

@vasinov vasinov left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks great overall!

CHANGELOG.md Show resolved Hide resolved
griptape/loaders/base_loader.py Outdated Show resolved Hide resolved
@dylanholmes dylanholmes removed the request for review from andrewfrench April 9, 2024 20:03
@dylanholmes dylanholmes merged commit c9edaff into dev Apr 9, 2024
6 checks passed
@dylanholmes dylanholmes deleted the feature/loaders-bytes branch April 9, 2024 21:39
@collindutter collindutter mentioned this pull request Apr 15, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

3 participants