Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add support for serialize/deserialize #95

Open
SamantazFox opened this issue Feb 15, 2024 · 2 comments
Open

Add support for serialize/deserialize #95

SamantazFox opened this issue Feb 15, 2024 · 2 comments

Comments

@SamantazFox
Copy link

Could you please add support for sqlite3_serialize and sqlite3_deserialize?

As a downstream app, that would allow us to manipulate database in memory only, removing the need to write data on disk!

@bcardiff
Copy link
Member

Can you give me an example of how would you use this? First time I see those functions.

In particular sqlite3_deserialize seems it will not play nice with the semantics of the db pool:

The sqlite3_deserialize(D,S,P,N,M,F) interface causes the database connection D to disconnect from database S and then reopen S as an in-memory database based on the serialization

Either the operations are just available as C functions (which they can happen outside crystal-sqlite3 if needed) or we need some reasonable Crystal api for these operations. An example would be a great starting point. I see there are many options and I am not sure what would be best for the general use case.

@SamantazFox
Copy link
Author

SamantazFox commented Feb 16, 2024

Our use case would be importing/exporting user data from/to a sister app.

As an example, here is our current code to import said user data. As you can see, we have to create a file on disk, open it, read the user data, close, and make sure it is deleted.
https://github.com/iv-org/invidious/blob/1e6ec605e88d1874e1b8b99294312a3c51f07beb/src/invidious/user/imports.cr#L292-L335

The goal would be to replace that code with something like this:

DB.open "sqlite3::memory" do |db|
  db.deserialize(file_io) # or maybe this?: sqlite3_deserialize(db, file_io)

  watched_raw = db.query_all("SELECT url FROM streams", as: String)
  subscriptions_raw = db.query_all("SELECT url FROM subscriptions", as: String)
  
  # The in-memory database is safely closed here
end

# Use the imported data

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants