[SQLite] Support linking against a system SQLite library

[mehcode]

• Is this a cargo feature flag? How does it work? Perhaps have sqlite (bundled) and sqlite_sys (unbundled) ?

• What is the default inclusion strategy? My preference leans heavily towards bundled.

[mehcode]

This is blocked on libsqlite3-sys allowing us to specify a much higher minimum SQLite version.

[jcgruenhage]

Ftr: This makes packaging software using sqlx a bit of a pain, because it breaks cross compilation in some situations. If you could make static linking of sqlite optional, I'd really appreciate it. Your reason for bundling sqlite doesn't apply to all distros, in this case, Void ships sqlite-3.32.3 at the moment, so that's more up to date than what libsqlite3-sys bundles.

Additionally, when you start to support this: Making this configurable via features only doesn't help distro packagers much. An env var would be much nicer for that. This would require a Cargo build script, but that's very doable. If you want this to mainly be a cargo feature, you can also set (or unset) the cargo feature from the build script. An env var override would really be appreciated.

[mehcode]

Making this configurable via features only doesn't help distro packagers much [...]

Can you help me understand how you'd use this? What does the environment variable look like?

---

[...] Your reason for bundling sqlite doesn't apply to all distros [...]

The libsqlite3-sys crate has 3 primary modes of compilation: bindgen where it generates the rust bindings from header files for the target and links against a system library; bundled where it compiles and statically links the bundled amalgamation; and the default, where it uses pre-generated rust bindings and links against a system library. I would rather not use bindgen as it greatly increases compile time and requires the header files to be present. However, the pre-generated bindings in libsqlite3-sys does not have several functions that we depend on: sqlite3_bind_blob64, sqlite3_bind_text64, sqlite3_prepare_v3 and sqlite3_value_dup.

[jcgruenhage]

Can you help me understand how you'd use this? What does the environment variable look like?

Sure. The variable could be SQLITE_COMPILATION_MODE_OVERRIDE=bindgen. We'd set that in the template of the application that depends on sqlx when building that. The build script could then read that env var and use it to override the selected feature. Does that make it clear enough? Otherwise I can try to build an example.

[...] I would rather not use bindgen as it greatly increases compile time and requires the header files to be present. However, the pre-generated bindings in libsqlite3-sys does not have several functions that we depend on: sqlite3_bind_blob64, sqlite3_bind_text64, sqlite3_prepare_v3 and sqlite3_value_dup.

That sounds quite sensible. Having bundled as the default sounds good then, and bindgen for distros that want to package software.

Now that I think more about it, maybe it'd make more sense for that override to live in libsqlite3-sys though? Does bundled vs bindgen result in enough difference in the API surface for your crate to fail compiling? If not, then putting that override into libsqlite3-sys sounds better.

[mehcode]

bundled vs bindgen shouldn't cause any differences to SQLx as long as a recent enough version of SQLite is used (v3.8.11+)

you're right that an env var override to change compilation mode makes more sense in libsqlite3-sys

once support for 3.8.11+ pre-generated bindings is available in libsqlite3-sys, we plan to remove the bundled feature flag from sqlx which means that an application using sqlx that wants bundled would do:

sqlx = { version = "0.5", features = [ "sqlite" ] }

# the * means we don't care about the version, its being constrained by sqlx anyway
libsqlite3-sys = { version = "*", features = [ "bundled" ] }

[thomcc]

once support for 3.8.11+ pre-generated bindings is available in libsqlite3-sys

I've meant to reach out to you about this, and do a write up, but we have an open issue in the rusqlite repository around the minimum/default supported version: rusqlite/rusqlite#706. (FWIW there's no rule that says that rusqlite and libsqlite3-sys need to have the same values here, although it would certainly be convenient for me if they did).

That said it's been a rough year in general and I haven't found the time to do so yet. That said, if you have opinions here they'd be valuable. Even the knowledge that you care about 3.8.11+ is very helpful.

In general I think the way libsqlite3-sys / rusqlite handles versions to be suboptimal. In an ideal world, there would be only a few versions that control underlying features, and then we'd have feature flags for them. clang-sys does this like https://github.com/KyleMayes/clang-sys/blob/master/Cargo.toml#L22-L42 and is fairly okay as a result, although you can see the fractures of that approach starting to not scale very well.

However, the world we live in is one in which there are very very many versions we would have to do this way, far more than clang-sys has to deal with...

Edit: P.S. in the future please feel welcome to file bugs in the rusqlite repo against libsqlite3-sys for anything sqlx finds lacking (such as the versioning situation). In general we'd be more than happy to make changes to accommodate your needs.