Add a check for testing that the upgrade scripts are correct

[brendanheywood]

This might be a dup of #156 but I don't think so.

What I have in mind is a way of checking that the install.xml and upgrade.php script are perfectly in sync where it would check out some very old version of the repo, install it, then checkout the latest and upgrade it and then assert it is in the same state as a fresh install.

One challenges to overcome is often plugins in early development don't maintain the upgrade script and then at some point when things have stabilized a line is drawn in the sand for the first release. So we can only ever install from this point on so we need to pass that commit hash in as a setting.

[stronk7]

Just noting that we already have that for core, where we continuously check that an upgraded Moodle instance (from any upgradeable version to target version Y) leads to 100% the same DB schema than the very same Y installed version.

This is, for master as version Y, for example: https://integration.moodle.org/view/master/job/25.%20Compare%20databases%20from%2039-310-311-400%20(master)/

And this is the shell script in charge of all the installations, upgrades and comparisons: https://github.com/moodlehq/moodle-local_ci/tree/master/compare_databases

Problem (that I can see) for plugins is that, while in core it's clear (and restricted by environmental checks) which versions are upgradeable to which versions.... in pluginland it's not so clear for me. So we would need something like:

1. Install upgraded site
   1.1 Install Moodle W with your_plugin X
   1.2 Upgrade it to Moodle Y with your_plugin Z
2. Install target site (Moodle Y with your plugin Z)
3. Compare upgraded and target sites effective db schema

And, right now, all those W, X, Y, Z (specifically the old W, X ones) is something that should be, somehow, parametrised in the runs and, for sure, diverges with the normal behaviour of moodle-plugin-ci that exclusively tests "actual" versions.

Note I'm not sure if that's the sort of comparison/check that you are proposing here. Just sharing how we do it in core and what would it imply here, for plugins.

Ciao :-)

[brendanheywood]

thanks @stronk7 I was aware that hq did this for core but not really across the details. And yes those parameters for the 'base' install version are what I was expecting we'd have to do