Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Introduce extension points for share storage #31

Open
tdikland opened this issue Jan 24, 2024 · 6 comments · May be fixed by #37
Open

Introduce extension points for share storage #31

tdikland opened this issue Jan 24, 2024 · 6 comments · May be fixed by #37
Assignees
@tdikland

Comments

@tdikland
Copy link
Contributor

Currently the Delta Sharing server depends on a postgres database to store information about the defined shares, schemas, and tables. For a variety of reasons users may want to use a different store. My proposed solution if to introduce a ShareStore trait (open to naming suggestions) and let the app state depend on a concrete implementation defined in a library or user supplied.
@tdikland
Copy link
Contributor Author

I'd be interested to work on this.

@roeap
Copy link
Collaborator

roeap commented Jan 28, 2024

This makes a lot of sense!

In fact I am hoping, that we can introduce extension points for various components of the sharing server - including the share store.

Regarding the naming - while I don't really have a better name, conceptually this should mirror catalogues, such that we can for instance populate the shares and their children from unity catalogue ...

I don't really remember the UC APIs, but we may want to have the option to add some more descriptive metadata on top of this. Main application would be to be able to populate UIs o.a. with some information consumers may be interested in around the shared datasets.

@tdikland
Copy link
Contributor Author

take

@tdikland
Copy link
Contributor Author

Thanks for the feedback! Let me put up a draft and discuss further from there.

@roeap
Copy link
Collaborator

roeap commented Jan 28, 2024

Not sure if this makes any sense at all, but I just had a thought 😆.

What if we more or less mirror the CatalogList, SchemaProvider, and TableProvider from datafusion - of course adjusting some types to out specific needs...

and then add some sort of metadata trait on top of this. This would allow us to re-use the StorageCatalog from delta-rs directly and potentially extend this to other catalogues that are implemented for datafusion.

Then again, just a quick thought ...

@tdikland tdikland linked a pull request Jan 30, 2024 that will close this issue
Draft
@tdikland
Copy link
Contributor Author

I've put up a draft for now. Let me have a think about your suggestions and see how they would fit in here:)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@tdikland tdikland

Labels
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

ShareStore extension tdikland/delta-sharing-rs-official
2 participants
@tdikland @roeap