core: Save schema version for intermediate steps

[digitalcircuit]

In short

• Update schemaversion for each successful intermediate upgrade
  • Better chance of auto-recovering by running core after interrupt
  • Aids debugging after the fact
  • Makes downgrades after interrupted upgrade more explicit, requiring setting schema version
• Update logging to be more explicit about what schema version fails

Criteria	Rank	Reason
Impact	★★★ 3/3	Reduces risk of breaking database with interrupted upgrades
Risk	★★☆ 2/3	Schema version might not be stored correctly
Intrusiveness	★☆☆ 1/3	Minor changes, shouldn't interfere with other pull requests

Rationale

These changes

When running upgrade queries, update the schema version for each intermediate step. This ensures that any interrupted upgrades have a greater chance of resuming correctly after core restart.

Almost all databases make single queries atomic (fully works or fully fails, no partial), and with many of the longest migrations being a single query, this makes upgrade interruptions much more likely to leave the database in a valid intermediate schema version.

As a side effect, downgrading from a partial upgrade will now require setting schema version manually, instead of it possibly happening to work depending on what changed. An unsupported operation failing in a known, recoverable way seems better than failing in an unknown, possibly recoverable way.

Future work

In the future, for databases that support it (e.g. almost only PostgreSQL), we may want to wrap upgrades in a transaction. This will need careful testing of potential additional space requirements and any database modifications that might not be allowed in a transaction.

For sake of time, the above was copied from the commit message.

Examples

Interrupted upgrade procedure

Steps

1. Show current schema version
2. Run core, forcibly (kill -9) interrupting upgrade when it reaches a long single-query migration
3. Run core, letting it finish

Commands run
run-profile.sh is a script to simplify managing multiple testing copies of Quassel and is equivalent to calling the right quasselcore with the right options

sqlite> select * from coreinfo;
schemaversion|26

$ (sleep 0.5 && pkill -9 quasselcore ) & \
  ./run-profile.sh master core local

[terminated]

sqlite> select * from coreinfo;
schemaversion|30

$ ./run-profile.sh master core local

sqlite> select * from coreinfo;
schemaversion|31

Interrupted upgrade log

2018-05-31 12:49:20 Warning: Installed Schema (version 26) is not up to date. Upgrading to version 31...  This may take a while for major upgrades. 
[terminated]
2018-05-31 12:54:29 Warning: Installed Schema (version 30) is not up to date. Upgrading to version 31...  This may take a while for major upgrades. 
2018-05-31 12:54:29 Warning: Installed Schema successfully upgraded to version 31. 
2018-05-31 12:54:29 Info: SQLite storage backend is ready. Schema version: 31

[digitalcircuit]

Pretty sure we're already in favor of this, but another point for the release notes:
Upgrades may take a long time, and may temporarily more than double the size of the database.

Just had someone on IRC run into a failed upgrade due to out of space - 22 GB database, 26 GB free space.