Provide an easy way to compile with official sqlite3 amalgamation

[JoshuaWise]

The README states:

To use sqleet as a library, the recommended way is to download a preconfigured release package instead of cloning the master. The contained sqleet.c and sqleet.h files are drop-in replacements for the official sqlite3.c amalgamation and sqlite3.h header.

However, I want to use my own custom amalgamation (some compile-time options must be provided when generating the amalgamation itself).

The official SEE extension makes it very easy to turn a regular amalgamation into one that supports encryption:

1. prepend see-prefix.txt
2. append see.c

Could there be a simple way to do the same thing using sqleet?

[resilar]

Prepare your own SQLite3 amalgamation and just replace sqlite3.c & sqlite3.h in sqleet's master. Compile and you should be good to go. If you want to produce sqleet release style drop-in replacements of your custom amalgamation version, then commit the changed SQLite3 files to master branch and run ./script/release.sh in a Linux environment. It should look like this:

[sqleet]$ ls
config.c  README.md      script   sqleet.c   sqlite3.h
crypto.c  rekeyvacuum.c  shell.c  sqlite3.c  UNLICENSE
[sqleet]$ ./script/release.sh
[+] SQLite version 3.26.0
[+] Generating rekeyvacuum.c
[+] Generating sqleet.c amalgamation
[+] Updating shell.c #include "sqlite3.h" -> "sqleet.h"
[+] Moving files around a bit
[+] Success!
git checkout -b v0.26.0
git commit -m "Release v0.26.0"
git push -u origin v0.26.0
[sqleet]% ls
README.md  shell.c  sqleet.c  sqleet.h  UNLICENSE

Now sqleet.c and sqleet.h contain your version of SQLite3 amalgamation with sqleet encryption support. Note that the printed git commands are not executed. They are there to simplify the release procedure for sqleet maintainers (i.e., a fancy way to say 'me').

If you cannot run the release.sh script in your environment for whatever reason, you can still manually make the sqleet-enabled amalgamation quite easily: Replace the few #includes in the beginning of sqleet.c with the contents of the included files and you're done. The only nasty part is updating rekeyvacuum.c to match the SQLite3 version used, but it is not mandatory most of the time (the release script executes ./script/rekeyvacuum.sh which tries to automate this process).

Also, if you are unlucky, the result may not compile straight out of the box if the used SQLite3 version differs enough. Fortunately, the required changes have been so far trivial across several years in the past (e.g., slightly adjust calls to SQLite3 functions which have been renamed or updated to take one additional parameter etc).

---

I agree that the procedure could be easier as it is in the official SEE extension case. However, I don't like the idea of making sqleet concatenable like SEE, because then we would lose the control of SQLite3 compile-time configuration (config.c), which is crucial for security reasons. Maybe writing a separate script or improving release.sh might be acceptable route forward. What do you think?

[JoshuaWise]

Prepare your own SQLite3 amalgamation and just replace sqlite3.c & sqlite3.h in sqleet's master. Compile and you should be good to go.

I originally created this issue with the goal of creating drop-in replacement amalgamations, but on second thought, this method actually does satisfy my needs.

I'll be using the same versions of sqlite as sqleet (no mis-matched versions)—the only difference is that my amalgamation will be created using various non-default configuration (e.g., OMIT_* flags, which must be present during sh configure within the sqlite source repository).

As far as I can tell, none of those OMIT_* flags affect sqlite3RunVacuum() until the C preprocessor actually runs. So I should just be able to use the existing rekeyvacuum.c file from the branch I'm using (currently sqleet-v0.26.0), without needing to run ./script/rekeyvacuum.sh.

While it turns out my needs are already met, I would recommend simplifying release.sh (or providing an alternative to it) that exhibits more friendly behaviors:

• It shouldn't require that the master branch is checked out, or that the working directory is clean. People would more likely want to check out a reliable tag (e.g., sqleet-v0.26.0) rather than the ever-changing master branch. And in my case, the working directory won't be clean if I move in my own sqlite3.c file.
• It should be more portable. For example, sed on Mac OSX doesn't have the -r flag. That's the first thing I noticed, but I bet there are other non-portable elements in that script as well. Of course, I'm biased because I develop on Mac OSX.

In any case, thanks for making this great library.

[resilar]

As far as I can tell, none of those OMIT_* flags affect sqlite3RunVacuum() until the C preprocessor actually runs. So I should just be able to use the existing rekeyvacuum.c file from the branch I'm using (currently sqleet-v0.26.0), without needing to run ./script/rekeyvacuum.sh.

That sounds OK to me.

While it turns out my needs are already met, I would recommend simplifying release.sh (or providing an alternative to it) that exhibits more friendly behaviors: ...

Commit 0dc5a18 added script/amalgamate.sh which performs amalgamation of sqleet master branch sources (instructions added to README.md as well). release.sh still has the annoying master branch & cleanliness check, but that is probably no longer an issue because the amalgamation procedure can be executed directly via the new script.

I also removed the -r flag from sed invocations. Unfortunately I do not have access to Mac OS X so I cannot solve remaining portability problems (if there's any). At least bash, busybox sh, fish & zsh successfully produce identical amalgamation using the script/amalgamate.sh script, which is promising. If you want to test on Mac OS X using current master:

[sqleet]$ git rev-parse HEAD
0dc5a18dbf93972316ea21315767323b2509befb
[sqleet]$ ./script/amalgamate.sh <sqleet.c | md5sum -
4ea5dedea10d9d538d2f06097edb45cc  -

[JoshuaWise]

This is great, thanks!

[JoshuaWise]

@resilar I just found out (via #20) that release branches and tags are supposed to be post-amalgamation. I started on sqleet-v0.26.0, and saw that it wasn't amalgamated, and assumed that was an intentional pattern, but it turns out I was mistaken.

Since I need to compile against my own version of sqlite3.c, I have to do the amalgamation myself—I cannot use those amalgamated branches. However, I would like to use stable released versions of sqleet, rather than just using master which is inherently unstable. So it turns out this issue is actually unsolved.

[resilar]

Yes, there was an issue with sqlite-v0.26.0 tag. A quick workaround for your use-case is to checkout this way: git checkout -b asdf sqleet-v0.26.0~1 (all release branches & tags point to an amalgamation commit that directly forks from master branch). This works for now, but in the future we'd like to have a better system, so leave this issue open until that happens.

Sorry in advance for the wall of text. And big thanks for your patience with the project so far, especially since the user experience has not been the smoothest possible. Feedback about how people use sqleet is valuable in order to improve the library.

The core issue is that branches, tags & releases are a bit abused in the repo for reason(s) not immediately obvious. The main requirement is that we want to offer old sqleet release amalgamations targeting specific SQLite3 versions in case some users are locked into a specific SQLite3 version. Currently the old releases are unmaintained, i.e., bug fixes are not backported to them. This is clearly not ideal but I do not see an easy way to address this without substantial amount of boring work needed to support (already) ten releases. *

Other requirements are imposed by Github's clunky release feature. Using a tag name v0.27.1 yields a source package named v0.27.1.zip, thereby requiring the sqleet- prefix to get sane file names (like sqleet-v0.27.1.zip). This on the other hand has the drawback that the source directory contained in the package is named sqleet-sqleet-v0.27.1/ which is maybe even more stupid. In other words, one simply cannot win (although it is possible that I'm just a bit OCD to care about these details at all).

Anyway, I have been recently planning to resolve the current releases/branches/tags mess as follows:

• Get rid of the release branches and make release tags point to a location in the master branch instead of an amalgamation commit (this is the expected behavior by you if I understood correctly).
• In the unlikely case that an old sqleet release needs support, fork a release branch from the master (specifically, from the location pointed by the existing release tag) and create a bug fix release.
• Offer release amalgamations (sqleet.c, sqleet.h) as uploaded "release binaries" because Github's source code packages are fundamentally broken unless I'm missing some detail of how releases work.
  • Consequently, source code packages can be fully ignored so the file names become unimportant. The packages would contain unamalgamated sqleet sources matching the amalgated files.
• Figure out what to do with releases published to date. Migrating them into this new system is perhaps feasible and better than leaving them as is.

This is what I would do now knowing the intricacies of Github more thoroughly. Any comments or ideas before I rush to implement this?

---

(*) As a side note:

If someone were to request support for an old sqleet release (via the Github issue tracker), then I could make it happen. However, it is still quite straightforward to take a bug-fixed newer version of sqleet and compile it against old SQLite3 (some trivial changes to the code may be needed; for example, sqlite3RunVacuum() interface changed in 3.27.0 which broke backward compatibility with previous SQLite3 versions -- this could definitely be improved by #ifdeffing against SQLite3 version numbers, but I'm unsure if it is worth the effort).

[JoshuaWise]

A quick workaround for your use-case is to checkout this way: git checkout -b asdf sqleet-v0.26.0~1 (all release branches & tags point to an amalgamation commit that directly forks from master branch)

This does solve my problem for now, thanks.

Sorry in advance for the wall of text. And big thanks for your patience with the project so far, especially since the user experience has not been the smoothest possible.

Thank you for making this great library and putting it in the public domain!

Get rid of the release branches and make release tags point to a location in the master branch instead of an amalgamation commit

This would be ideal. From my perspective, release tags/branches should always be on a regular git timeline, and special pre-built deliverables should be delivered separately.

In the unlikely case that an old sqleet release needs support, fork a release branch from the master (specifically, from the location pointed by the existing release tag) and create a bug fix release

In this case, what would the bugfix release be named/tagged as? Given how the semver numbers are used to match the sqlite3 version, I assume the sqleet patch would have to be indicated by some other naming convention, perhaps appended to the sqlite3 version number?

• Figure out what to do with releases published to date. Migrating them into this new system is perhaps feasible and better than leaving them as is

Doing this on a per-request basis is probably good enough. I can't imagine that too many people would have such a rigid need for backwards compatibility, given that this project is relatively young.

[resilar]

Thanks for your input.

In this case, what would the bugfix release be named/tagged as? Given how the semver numbers are used to match the sqlite3 version, I assume the sqleet patch would have to be indicated by some other naming convention, perhaps appended to the sqlite3 version number?

Yes, naming bug fix tags can be tricky because of the weird way sqleet versions are numbered to match SQLite3 versions (although I don't see an easy way to fix the tag naming issue by changing the version numbering convention). AFAIK semver allows appending a suffix to version number (e.g., 0.27.1-fix0) so maybe that could be used as you suggested. Of course simply recreating the old release tags without changing names is also an option, but that can cause major confusion in the long run if the same version number can correspond to multiple commits. Thus, appending suffixes is probably the way to go.

I can't imagine that too many people would have such a rigid need for backwards compatibility, given that this project is relatively young.

Yes. Given the age of the project, I'm considering updating existing releases to the new system. It should not cause significant breakage for users, I think.

[resilar]

Now the latest release v0.27.1 follows the new release convention. I decided to compress the deliverables because amalgamated sqleet.c is quite big (7.6MB) and including README.md & shell.c in release archives is nice. I also deleted the two oldest releases (v0.1.0 & v0.1.1) that did not use the current version numbering scheme, which hopefully avoids confusion.

Older releases (<0.27.1) can remain in the old release format.

Closing the issue. Reopen if you find mistakes or shortcomings in the new release approach.