New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Microsoft.Data.Sqlite: Incremental blob I/O #13824

Closed
bricelam opened this Issue Apr 2, 2014 · 8 comments

Comments

Projects
None yet
4 participants
@bricelam
Copy link
Member

bricelam commented Apr 2, 2014

See Open A BLOB For Incremental I/O.

This would light up the GetBytes, GetChars & GetStream methods on SqliteDataReader.

@bricelam bricelam changed the title Enable incremental blob i/o Incremental blob I/O Apr 2, 2014

@bricelam bricelam added this to the Backlog milestone May 3, 2014

@bricelam bricelam added the enhancement label Jul 1, 2014

@bricelam

This comment has been minimized.

Copy link
Member

bricelam commented Dec 20, 2017

@buybackoff Would you be interested in contributing in this area? (Possibly just GetBytes)

@buybackoff

This comment has been minimized.

Copy link

buybackoff commented Dec 20, 2017

I have it my fork Spreads/Spreads.SQLite@21add79

I do not use much functionality of SQL, only as ordered KV blob storage, so I am pretty much all set already.

@bricelam

This comment has been minimized.

Copy link
Member

bricelam commented Dec 20, 2017

I noticed you had a commit that mentioned it, but I can't/shouldn't look at it unless you submit a PR and sign the CLA. No worries if you're not interested; I just thought I'd ping you to see if you were.

@AlexanderTaeschner

This comment has been minimized.

Copy link
Contributor

AlexanderTaeschner commented Apr 9, 2018

sqlite3_blob_open is only usable to open a BLOB in a certain row in a specific column of an existing table. Since the SqliteDataReader can only get the column and table name but not the rowid, it can not be used to implement GetBytes!
From my point of view the only use case could be to access a really large BLOB incrementally. Would this be interesting?

@bricelam

This comment has been minimized.

Copy link
Member

bricelam commented Apr 9, 2018

Oh wow, that is restrictive. It's a lower-priority feature, but I think it's still interesting for certain applications.

We could make the APIs use sqlite3_column_blob and a MemoryStream by default. But if you happen to SELECT rowid it can use a more optimized sqlite3_blob_open codepath.

@bricelam

This comment has been minimized.

Copy link
Member

bricelam commented Apr 9, 2018

It's not very discoverable, but at least it unblocks applications that need to either...

  • Peek at the first few bytes/characters (e.g. file headers)
  • Do some forward-only parsing/querying/interpreting in a memory-optimized way (e.g. XML, JSON, BAML, etc.)
@bricelam

This comment has been minimized.

Copy link
Member

bricelam commented Apr 9, 2018

sqlite3_bind_zeroblob() is related to these scenarios. To stream a BLOB into the database, you'd INSERT using a zeroblob parameter of the correct size then call sqlite3_blob_open and sqlite3_blob_write.

@bricelam bricelam modified the milestones: Backlog, 2.2.0 Apr 13, 2018

@bricelam bricelam self-assigned this Apr 13, 2018

@bricelam bricelam modified the milestones: 2.2.0, Backlog Jun 6, 2018

@ajcvickers ajcvickers transferred this issue from aspnet/Microsoft.Data.Sqlite Oct 31, 2018

@ajcvickers ajcvickers added this to the Backlog milestone Oct 31, 2018

@ajcvickers ajcvickers changed the title Incremental blob I/O Microsoft.Data.Sqlite: Incremental blob I/O Oct 31, 2018

@bricelam bricelam modified the milestones: Backlog, 3.0.0 Nov 12, 2018

@bricelam

This comment has been minimized.

Copy link
Member

bricelam commented Nov 20, 2018

Initial implementation merged. See #13987 for further improvements.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment