tests/kola: add extended upgrade test #2310
Conversation
There was a problem hiding this comment.
This looks sane overall (thanks for writing it!). But I think ideally we'd rework this so that it works with the locally built artifacts. I've been thinking about merging the kola upgrade test into an external test, which is relevant here.
One thing we could do is add a new startingImage YAML key, which can take e.g. previous (for the current upgrade test) and first (for this one). Then kola would implement the logic to resolve the version and fetch the image to boot (it already has most of the code for this). Then the external test could use injectContainer: true to have access to the OCI archive.
tests/kola/upgrade/extended/test.sh
Outdated
| # in the compose OSTree repo. | ||
| if vereq $version $last_release; then | ||
| systemctl stop zincati | ||
| rpm-ostree rebase fedora-compose: |
There was a problem hiding this comment.
Shouldn't we check here that the version we staged matches $target_version? Or alternatively, we can make this rpm-ostree rebase fedora-compose: $target_version. That way rpm-ostree itself errors out if the version somehow isn't in the compose repo.
tests/kola/upgrade/extended/test.sh
Outdated
| timeout 60s journalctl -b 0 -f --no-tail -u zincati.service \ | ||
| | grep --max-count=1 'proceeding to stage it' && \ | ||
| /tmp/autopkgtest-reboot-prepare reboot && sleep infinity | ||
|
|
||
| # If we make it here then there was no update; let's error. | ||
| fatal "Updating the system stalled out on version: $version" |
There was a problem hiding this comment.
This looks cryptic. Why not
if ! timeout ... | grep ...; then
fatal "Updating the system stalled out on version: $version"
fi
/tmp/autopkgtest-reboot-prepare reboot
sleep infinity?
There was a problem hiding this comment.
I didn't think what I had was too bad, but I reworked it a bit now
dustymabe
force-pushed
the
dusty-upgrade-test
branch
2 times, most recently
from
9a4e55e
to
59c4b7a
Compare
No problem.
I support something better than this, but probably can't invest much time in it for now.
👍 |
tests/kola/upgrade/extended/test.sh
Outdated
| max_version=38 # hardcode for now because `next` went from | ||
| # 37.20230303.1.0 -> 38.20230310.1.0 -> 37.20230303.1.1. | ||
| # Can drop once a new 38 commit is at the tip of `next`. |
There was a problem hiding this comment.
Since we're about to run the next build we can just hold this PR for 4-12 hours and I can drop this hack/workaround (I think).
There was a problem hiding this comment.
This is still there, did you mean to drop it?
dustymabe
force-pushed
the
dusty-upgrade-test
branch
3 times, most recently
from
266649a
to
3d7195c
Compare
This test will attempt to test an upgrade from a given starting point (assumed by the caller passing in a specific `cosa kola run --build=x.y.z`) all the way to the latest build that is staged to be released. The test is basic in that it essentially tests 1) updates work 2) boot works. An example invocation for this test would look like: ``` cosa buildfetch --stream=next --build=34.20210904.1.0 --artifact=qemu cosa decompress --build=34.20210904.1.0 cosa kola run --build=34.20210904.1.0 --tag extended-upgrade ``` You can then monitor the console.txt or journal.txt to watch the upgrades happen via `tail -f tmp/kola/ext.config.upgrade.extened/*/console.txt'. Another useful thing to do is to just monitor the major events: ``` $ tail -f tmp/kola/ext.config.upgrade.extended/*/journal.txt | grep --color -i 'ok reached version' Mar 20 02:25:12.584096 kola-runext-test.sh[2085]: ok Reached version: 34.20210904.1.0 Mar 20 02:26:37.107738 kola-runext-test.sh[3538]: ok Reached version: 35.20210924.1.0 Mar 20 02:28:08.509482 kola-runext-test.sh[1668]: ok Reached version: 36.20220418.1.0 Mar 20 02:28:41.242555 kola-runext-test.sh[1649]: ok Reached version: 36.20220906.1.0 Mar 20 02:29:33.635979 kola-runext-test.sh[1635]: ok Reached version: 37.20221111.1.0 Mar 20 02:30:16.657854 kola-runext-test.sh[1469]: ok Reached version: 37.20230303.1.0 ```
dustymabe
force-pushed
the
dusty-upgrade-test
branch
from
3d7195c
to
7d6d879
Compare
This test will attempt to test an upgrade from a given starting point (assumed by the caller passing in a specific
cosa kola run --build=x.y.z) all the way to the latest build that is staged to be released. The test is basic in that it essentially tests 1) updates work 2) boot works.An example invocation for this test would look like:
You can then monitor the console.txt or journal.txt to watch the upgrades happen via
tail -f tmp/kola/ext.config.upgrade.extened/*/console.txt.Another useful thing to do is to just monitor the major events: