apsw as alternative sqlite3 binding (for full text search)

[mhalle]

Hey there,

Have you considered providing apsw support as an alternative to stock python sqlite3? I use apsw because it keeps up with sqlite3 and is straightforward to bring in extensions like FTS5. FTS really accelerates the kind of searching often done by web clients.

I may be able to help (it shouldn't be much code), but there are a couple of stylistic questions that come up when supporting an optional package. Also, apsw is tricky in that it doesn't have a pypi package (author says limitations in providing options to setup.py).

[simonw]

I have a solution for FTS already, but I'm interested in apsw as a mechanism for allowing custom virtual tables to be written in Python (pysqlite only lets you write custom functions)

Not having PyPI support is pretty tough though. I'm planning a plugin/extension system which would be ideal for things like an optional apsw mode, but that's a lot harder if apsw isn't in PyPI.

[mhalle]

Thanks. There is a way to use pip to grab apsw, which also let's you configure it (flags to build extensions, use an internal sqlite, etc). Don't know how that works as a dependency for another package, though.

…

On November 22, 2017 11:38:06 AM EST, Simon Willison ***@***.***> wrote: I have a solution for FTS already, but I'm interested in apsw as a mechanism for allowing custom virtual tables to be written in Python (pysqlite only lets you write custom functions) Not having PyPI support is pretty tough though. I'm planning a plugin/extension system which would be ideal for things like an optional apsw mode, but that's a lot harder if apsw isn't in PyPI. -- You are receiving this because you authored the thread. Reply to this email directly or view it on GitHub: #144 (comment)

[simonw]

The other major limitation of APSW is its treatment of unicode: https://rogerbinns.github.io/apsw/types.html - it tells you that it is your responsibility to ensure that TEXT columns in your SQLite database are correctly encoded.

Since Datasette is designed to work against ANY SQLite database that someone may have already created, I see that as a show-stopping limitation.

Thanks to https://github.com/coleifer/sqlite-vtfunc I now have a working mechanism for virtual tables (I've even built a demo plugin with them - https://github.com/simonw/datasette-sql-scraper ) which was the main thing that interested me about APSW.

I'm going to close this as WONTFIX - I think Python's built-in sqlite3 is good enough, and is now so firmly embedded in the project that making it pluggable would be more trouble than it's worth.