present a file-like interface to blobs #101

Open
glyph opened this Issue Apr 20, 2016 · 3 comments

Comments

Projects
None yet
3 participants
@glyph

glyph commented Apr 20, 2016

http://www.sqlite.org/c3ref/blob_open.html and http://www.sqlite.org/c3ref/blob_write.html allow users to incrementally write large objects, which can (at least in principle) allow for smaller memory footprint when processing large blob storage requests. Given that sqlite supports 64-bit blobs, i.e. larger than would fit in memory, this is necessary to be able to access all of its functionality.

@glyph glyph changed the title from make blobs file-like to present a file-like interface to blobs Apr 20, 2016

@rogerbinns

This comment has been minimized.

Show comment
Hide comment
@rogerbinns

rogerbinns Apr 20, 2016

@glyph your 64 bit portion is not correct. The default maximum blob size is 1GB. The maximum you can make it by raising limits is a few bytes shy of 2GB. This applies to 32 and 64 bit versions of SQLite and is from the file format.

Blobs also have several restrictions. For example you cannot change their length with the blob I/O interface. (You can use zeroblob to create one of the right size, and then overwrite portions of that.) I recommend having a look at the APSW doc to get an idea of a file like API will look like, and the various quirks: https://rogerbinns.github.io/apsw/blob.html#blob-class

@glyph your 64 bit portion is not correct. The default maximum blob size is 1GB. The maximum you can make it by raising limits is a few bytes shy of 2GB. This applies to 32 and 64 bit versions of SQLite and is from the file format.

Blobs also have several restrictions. For example you cannot change their length with the blob I/O interface. (You can use zeroblob to create one of the right size, and then overwrite portions of that.) I recommend having a look at the APSW doc to get an idea of a file like API will look like, and the various quirks: https://rogerbinns.github.io/apsw/blob.html#blob-class

@glyph

This comment has been minimized.

Show comment
Hide comment
@glyph

glyph Apr 20, 2016

@rogerbinns thanks for all that information. I suppose the interface here is just for limiting memory usage during a bulk load of a large file, not actually for doing things like having a log file that can be appended to inside the database. Still, it's useful to have :)

glyph commented Apr 20, 2016

@rogerbinns thanks for all that information. I suppose the interface here is just for limiting memory usage during a bulk load of a large file, not actually for doing things like having a log file that can be appended to inside the database. Still, it's useful to have :)

@palaviv

This comment has been minimized.

Show comment
Hide comment
@palaviv

palaviv Apr 20, 2016

hi @glyph you may want to look at pull request #93.

palaviv commented Apr 20, 2016

hi @glyph you may want to look at pull request #93.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment