Skip to content

Add option to use local filesystem for storage #217

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 1 commit into from
Dec 17, 2022

Conversation

trueleo
Copy link
Contributor

@trueleo trueleo commented Dec 15, 2022

Description

This PR enables Parseable server to accept additional arguments ( drive or s3) to switch between storage option.
Local filesystem behaves as an storage provider through implementation of ObjectStorage trait. All
methods of LocalFS are implemented using tokio::fs

Configuration provided on runtime is used to select a storage provider. The storage object is handled as
trait objects hence the requirement of ?Sized on some methods. Calls to S3 are replaced with appropriate methods.


This PR has:

  • been tested to ensure log ingestion and log query works.
  • added comments explaining the "why" and the intent of the code wherever would not be obvious for an unfamiliar reader.
  • added documentation for new or modified features or behaviors.

This PR enables Parseable server to accept additional arguments ( drive or s3) to switch between storage.
Local filesystem behaves like an object storage through implementation of ObjectStorage trait. All
methods of LocalFS are implemented using tokio::fs

Configuration provided on runtime is used to select a storage provider. The storage object is handled as
trait objects hence the requirement of ?Sized on some methods. Calls to S3 are replaced with appropriate methods.
@trueleo trueleo changed the title Add option to use local filesystem as object storage Add option to use local filesystem for storage Dec 15, 2022
@nitisht nitisht self-requested a review December 17, 2022 12:32
Copy link
Member

@nitisht nitisht left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, there are several improvements required, we can do those incrementally.

@nitisht nitisht merged commit b4b5ca2 into parseablehq:main Dec 17, 2022
@trueleo trueleo deleted the add_localfs branch December 17, 2022 15:01
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants