-
Notifications
You must be signed in to change notification settings - Fork 165
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
kola: Replace run-ext-bin with kola run -E/--exttest
#1252
kola: Replace run-ext-bin with kola run -E/--exttest
#1252
Conversation
I updated ostreedev/ostree#2025 to use this as a demo. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Couple items but I really like this design!
5e6f856
to
3c790da
Compare
Pairs with coreos/coreos-assembler#1252 The motivation here is to get the ball rolling for (where it makes sense) to have a place to add tests corresponding to changes in this repository.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Didn't test it but few stylistic comments, otherwise LGTM. Will let someone else take a review stab as well.
/approve
New demonstration: coreos/fedora-coreos-config#305 |
kola run -E/--exttest
kola run -E/--exttest
I discovered the rebooting support was even more aspirational rather than functional. |
What specifically are you seeing? I haven't tried it with |
This code is trying to support having the test code run This avoids a whole lot synchronous round trips with But the current kola code around reboots is much more oriented around having kola drive/monitor the reboot status so I'm reworking things to have the test code request a reboot rather than invoke it directly. |
334da8c
to
f8bb1b3
Compare
OK cool, I fixed up the rebooting support (not in the most elegant way, but eh it works) and also rewrote the |
Oh and this now supports a
|
kola run -E/--exttest
kola run -E/--exttest
Pairs with coreos/coreos-assembler#1252 The motivation here is to get the ball rolling for (where it makes sense) to have a place to add tests corresponding to changes in this repository.
f8bb1b3
to
be271a4
Compare
OK, updated the docs and lifting WIP on this! |
@@ -138,9 +139,16 @@ func dispatchRunExtUnit(unitname string, sdconn *systemddbus.Conn) (bool, error) | |||
} | |||
case CLD_KILLED: | |||
// SIGTERM; we explicitly allow that, expecting we're rebooting. | |||
if mainstatus == 15 { | |||
if mainstatus == int32(15) { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Oh yeah this...SO EVIL.
Pid: os.Getpid(), | ||
} | ||
selfproc.Signal(syscall.SIGTERM) | ||
time.Sleep(time.Hour) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
It feels odd to wait wait up to an hour, but then again it won't hurt to wait that long since it should be interrupted.
Another big topic here is that I think we need to standardize a bit more of the flow for "build project X, overlay it on fcos, run just the tests for that project (plus possibly kola basic or others) as external tests". A bit more discussion of related topics in coreos/rpm-ostree#1958 |
Previously we made an effort to use the [Fedora Standard Test Interface](https://docs.fedoraproject.org/en-US/ci/standard-test-interface/). This effort was not very successful; the primary thing that it really died on is Ansible just didn't support rebooting very well. I think that's since gotten better, but even then, Ansible wasn't the best thing for a test framework for us anyways. In the meantime Fedora CoreOS happened emphasizing Ignition and not "post-hoc reconciliation" models like Ansible over ssh. And, [coreos-assembler](https://github.com/coreos/coreos-assembler) happened too. Furthermore, we really need to test OSTree's interaction with Ignition as we've invented several special things there. Then most recently, I've been working on having cosa/kola support running externally defined tests: coreos/coreos-assembler#1215 There's a lot of things to clean up after this but at least this works for me: ``` $ cd /srv/fcos $ cosa kola run -- --parallel 4 --output-dir tmp/kola -E ~/src/github/ostreedev/ostree/ 'ext.ostree.*' ``` NOTE: This *does not* drop ostree binaries into the target. See: coreos/coreos-assembler#1252 (comment) This drops our dependency on Python in the installed tests, and also fixes a few bugs that came up. I disabled the `itest-bare-user-root.sh` one because it's entangled with the shell script infrastructure for the unit tests.
So we can use `-E` in CI, see coreos/coreos-assembler#1252
So we can use `-E` in CI, see coreos/coreos-assembler#1252
Previously we made an effort to use the [Fedora Standard Test Interface](https://docs.fedoraproject.org/en-US/ci/standard-test-interface/). This effort was not very successful; the primary thing that it really died on is Ansible just didn't support rebooting very well. I think that's since gotten better, but even then, Ansible wasn't the best thing for a test framework for us anyways. In the meantime Fedora CoreOS happened emphasizing Ignition and not "post-hoc reconciliation" models like Ansible over ssh. And, [coreos-assembler](https://github.com/coreos/coreos-assembler) happened too. Furthermore, we really need to test OSTree's interaction with Ignition as we've invented several special things there. Then most recently, I've been working on having cosa/kola support running externally defined tests: coreos/coreos-assembler#1215 There's a lot of things to clean up after this but at least this works for me: ``` $ cd /srv/fcos $ cosa kola run -- --parallel 4 --output-dir tmp/kola -E ~/src/github/ostreedev/ostree/ 'ext.ostree.*' ``` NOTE: This *does not* drop ostree binaries into the target. See: coreos/coreos-assembler#1252 (comment) This drops our dependency on Python in the installed tests, and also fixes a few bugs that came up. I disabled the `itest-bare-user-root.sh` one because it's entangled with the shell script infrastructure for the unit tests.
So we can use `-E` in CI, see coreos/coreos-assembler#1252
So we can use `-E` in CI, see coreos/coreos-assembler#1252
So we can use `-E` in CI, see coreos/coreos-assembler#1252
So we can use `-E` in CI, see coreos/coreos-assembler#1252
Pairs with coreos/coreos-assembler#1252 The motivation here is to get the ball rolling for (where it makes sense) to have a place to add tests corresponding to changes in this repository.
Pairs with coreos/coreos-assembler#1252 The motivation here is to get the ball rolling for (where it makes sense) to have a place to add tests corresponding to changes in this repository.
Pairs with coreos/coreos-assembler#1252 The motivation here is to get the ball rolling for (where it makes sense) to have a place to add tests corresponding to changes in this repository.
There's a lot going on here. First, this is intended to run nicely as part of the new [cosa/kola ext-tests](coreos/coreos-assembler#1252). With Rust we can get one big static binary that we can upload, and include a webserver as part of the binary. This way we don't need to do the hack of running a container with Python or whatever. Now, what's even better about Rust for this is that it has macros, and specifically we are using [commandspec](https://github.com/tcr/commandspec/) which allows us to "inline" shell script. I think the macros could be even better, but this shows how we can intermix pure Rust code along with using shell safely enough. We're using my fork of commandspec because the upstream hasn't merged [a few PRs](https://github.com/tcr/commandspec/pulls?q=is%3Apr+author%3Acgwalters+). This model is intended to replace *both* some of our `make check` tests as well. Oh, and this takes the obvious step of using the Rust OSTree bindings as part of our tests. Currently the "commandspec tests" and "API tests" are separate, but nothing stops us from intermixing them if we wanted. I haven't yet tried to write destructive tests with this but I think it will go well.
There's a lot going on here. First, this is intended to run nicely as part of the new [cosa/kola ext-tests](coreos/coreos-assembler#1252). With Rust we can get one big static binary that we can upload, and include a webserver as part of the binary. This way we don't need to do the hack of running a container with Python or whatever. Now, what's even better about Rust for this is that it has macros, and specifically we are using [commandspec](https://github.com/tcr/commandspec/) which allows us to "inline" shell script. I think the macros could be even better, but this shows how we can intermix pure Rust code along with using shell safely enough. We're using my fork of commandspec because the upstream hasn't merged [a few PRs](https://github.com/tcr/commandspec/pulls?q=is%3Apr+author%3Acgwalters+). This model is intended to replace *both* some of our `make check` tests as well. Oh, and this takes the obvious step of using the Rust OSTree bindings as part of our tests. Currently the "commandspec tests" and "API tests" are separate, but nothing stops us from intermixing them if we wanted. I haven't yet tried to write destructive tests with this but I think it will go well.
Pairs with coreos/coreos-assembler#1252 The motivation here is to get the ball rolling for (where it makes sense) to have a place to add tests corresponding to changes in this repository.
There's a lot going on here. First, this is intended to run nicely as part of the new [cosa/kola ext-tests](coreos/coreos-assembler#1252). With Rust we can get one big static binary that we can upload, and include a webserver as part of the binary. This way we don't need to do the hack of running a container with Python or whatever. Now, what's even better about Rust for this is that it has macros, and specifically we are using [commandspec](https://github.com/tcr/commandspec/) which allows us to "inline" shell script. I think the macros could be even better, but this shows how we can intermix pure Rust code along with using shell safely enough. We're using my fork of commandspec because the upstream hasn't merged [a few PRs](https://github.com/tcr/commandspec/pulls?q=is%3Apr+author%3Acgwalters+). This model is intended to replace *both* some of our `make check` tests as well. Oh, and this takes the obvious step of using the Rust OSTree bindings as part of our tests. Currently the "commandspec tests" and "API tests" are separate, but nothing stops us from intermixing them if we wanted. I haven't yet tried to write destructive tests with this but I think it will go well.
There's a lot going on here. First, this is intended to run nicely as part of the new [cosa/kola ext-tests](coreos/coreos-assembler#1252). With Rust we can get one big static binary that we can upload, and include a webserver as part of the binary. This way we don't need to do the hack of running a container with Python or whatever. Now, what's even better about Rust for this is that it has macros, and specifically we are using [commandspec](https://github.com/tcr/commandspec/) which allows us to "inline" shell script. I think the macros could be even better, but this shows how we can intermix pure Rust code along with using shell safely enough. We're using my fork of commandspec because the upstream hasn't merged [a few PRs](https://github.com/tcr/commandspec/pulls?q=is%3Apr+author%3Acgwalters+). This model is intended to replace *both* some of our `make check` tests as well. Oh, and this takes the obvious step of using the Rust OSTree bindings as part of our tests. Currently the "commandspec tests" and "API tests" are separate, but nothing stops us from intermixing them if we wanted. I haven't yet tried to write destructive tests with this but I think it will go well.
There's a lot going on here. First, this is intended to run nicely as part of the new [cosa/kola ext-tests](coreos/coreos-assembler#1252). With Rust we can get one big static binary that we can upload, and include a webserver as part of the binary. This way we don't need to do the hack of running a container with Python or whatever. Now, what's even better about Rust for this is that it has macros, and specifically we are using [commandspec](https://github.com/tcr/commandspec/) which allows us to "inline" shell script. I think the macros could be even better, but this shows how we can intermix pure Rust code along with using shell safely enough. We're using my fork of commandspec because the upstream hasn't merged [a few PRs](https://github.com/tcr/commandspec/pulls?q=is%3Apr+author%3Acgwalters+). This model is intended to replace *both* some of our `make check` tests as well. Oh, and this takes the obvious step of using the Rust OSTree bindings as part of our tests. Currently the "commandspec tests" and "API tests" are separate, but nothing stops us from intermixing them if we wanted. I haven't yet tried to write destructive tests with this but I think it will go well.
There's a lot going on here. First, this is intended to run nicely as part of the new [cosa/kola ext-tests](coreos/coreos-assembler#1252). With Rust we can get one big static binary that we can upload, and include a webserver as part of the binary. This way we don't need to do the hack of running a container with Python or whatever. Now, what's even better about Rust for this is that it has macros, and specifically we are using [commandspec](https://github.com/tcr/commandspec/) which allows us to "inline" shell script. I think the macros could be even better, but this shows how we can intermix pure Rust code along with using shell safely enough. We're using my fork of commandspec because the upstream hasn't merged [a few PRs](https://github.com/tcr/commandspec/pulls?q=is%3Apr+author%3Acgwalters+). This model is intended to replace *both* some of our `make check` tests as well. Oh, and this takes the obvious step of using the Rust OSTree bindings as part of our tests. Currently the "commandspec tests" and "API tests" are separate, but nothing stops us from intermixing them if we wanted. I haven't yet tried to write destructive tests with this but I think it will go well.
There's a lot going on here. First, this is intended to run nicely as part of the new [cosa/kola ext-tests](coreos/coreos-assembler#1252). With Rust we can get one big static binary that we can upload, and include a webserver as part of the binary. This way we don't need to do the hack of running a container with Python or whatever. Now, what's even better about Rust for this is that it has macros, and specifically we are using [commandspec](https://github.com/tcr/commandspec/) which allows us to "inline" shell script. I think the macros could be even better, but this shows how we can intermix pure Rust code along with using shell safely enough. We're using my fork of commandspec because the upstream hasn't merged [a few PRs](https://github.com/tcr/commandspec/pulls?q=is%3Apr+author%3Acgwalters+). This model is intended to replace *both* some of our `make check` tests as well. Oh, and this takes the obvious step of using the Rust OSTree bindings as part of our tests. Currently the "commandspec tests" and "API tests" are separate, but nothing stops us from intermixing them if we wanted. I haven't yet tried to write destructive tests with this but I think it will go well.
There's a lot going on here. First, this is intended to run nicely as part of the new [cosa/kola ext-tests](coreos/coreos-assembler#1252). With Rust we can get one big static binary that we can upload, and include a webserver as part of the binary. This way we don't need to do the hack of running a container with Python or whatever. Now, what's even better about Rust for this is that it has macros, and specifically we are using [commandspec](https://github.com/tcr/commandspec/) which allows us to "inline" shell script. I think the macros could be even better, but this shows how we can intermix pure Rust code along with using shell safely enough. We're using my fork of commandspec because the upstream hasn't merged [a few PRs](https://github.com/tcr/commandspec/pulls?q=is%3Apr+author%3Acgwalters+). This model is intended to replace *both* some of our `make check` tests as well. Oh, and this takes the obvious step of using the Rust OSTree bindings as part of our tests. Currently the "commandspec tests" and "API tests" are separate, but nothing stops us from intermixing them if we wanted. I haven't yet tried to write destructive tests with this but I think it will go well.
There's a lot going on here. First, this is intended to run nicely as part of the new [cosa/kola ext-tests](coreos/coreos-assembler#1252). With Rust we can get one big static binary that we can upload, and include a webserver as part of the binary. This way we don't need to do the hack of running a container with Python or whatever. Now, what's even better about Rust for this is that it has macros, and specifically we are using [commandspec](https://github.com/tcr/commandspec/) which allows us to "inline" shell script. I think the macros could be even better, but this shows how we can intermix pure Rust code along with using shell safely enough. We're using my fork of commandspec because the upstream hasn't merged [a few PRs](https://github.com/tcr/commandspec/pulls?q=is%3Apr+author%3Acgwalters+). This model is intended to replace *both* some of our `make check` tests as well. Oh, and this takes the obvious step of using the Rust OSTree bindings as part of our tests. Currently the "commandspec tests" and "API tests" are separate, but nothing stops us from intermixing them if we wanted. I haven't yet tried to write destructive tests with this but I think it will go well.
There's a lot going on here. First, this is intended to run nicely as part of the new [cosa/kola ext-tests](coreos/coreos-assembler#1252). With Rust we can get one big static binary that we can upload, and include a webserver as part of the binary. This way we don't need to do the hack of running a container with Python or whatever. Now, what's even better about Rust for this is that it has macros, and specifically we are using [commandspec](https://github.com/tcr/commandspec/) which allows us to "inline" shell script. I think the macros could be even better, but this shows how we can intermix pure Rust code along with using shell safely enough. We're using my fork of commandspec because the upstream hasn't merged [a few PRs](https://github.com/tcr/commandspec/pulls?q=is%3Apr+author%3Acgwalters+). This model is intended to replace *both* some of our `make check` tests as well. Oh, and this takes the obvious step of using the Rust OSTree bindings as part of our tests. Currently the "commandspec tests" and "API tests" are separate, but nothing stops us from intermixing them if we wanted. I haven't yet tried to write destructive tests with this but I think it will go well.
There's a lot going on here. First, this is intended to run nicely as part of the new [cosa/kola ext-tests](coreos/coreos-assembler#1252). With Rust we can get one big static binary that we can upload, and include a webserver as part of the binary. This way we don't need to do the hack of running a container with Python or whatever. Now, what's even better about Rust for this is that it has macros, and specifically we are using [commandspec](https://github.com/tcr/commandspec/) which allows us to "inline" shell script. I think the macros could be even better, but this shows how we can intermix pure Rust code along with using shell safely enough. We're using my fork of commandspec because the upstream hasn't merged [a few PRs](https://github.com/tcr/commandspec/pulls?q=is%3Apr+author%3Acgwalters+). This model is intended to replace *both* some of our `make check` tests as well. Oh, and this takes the obvious step of using the Rust OSTree bindings as part of our tests. Currently the "commandspec tests" and "API tests" are separate, but nothing stops us from intermixing them if we wanted. I haven't yet tried to write destructive tests with this but I think it will go well.
There's a lot going on here. First, this is intended to run nicely as part of the new [cosa/kola ext-tests](coreos/coreos-assembler#1252). With Rust we can get one big static binary that we can upload, and include a webserver as part of the binary. This way we don't need to do the hack of running a container with Python or whatever. Now, what's even better about Rust for this is that it has macros, and specifically we are using [commandspec](https://github.com/tcr/commandspec/) which allows us to "inline" shell script. I think the macros could be even better, but this shows how we can intermix pure Rust code along with using shell safely enough. We're using my fork of commandspec because the upstream hasn't merged [a few PRs](https://github.com/tcr/commandspec/pulls?q=is%3Apr+author%3Acgwalters+). This model is intended to replace *both* some of our `make check` tests as well. Oh, and this takes the obvious step of using the Rust OSTree bindings as part of our tests. Currently the "commandspec tests" and "API tests" are separate, but nothing stops us from intermixing them if we wanted. I haven't yet tried to write destructive tests with this but I think it will go well.
There's a lot going on here. First, this is intended to run nicely as part of the new [cosa/kola ext-tests](coreos/coreos-assembler#1252). With Rust we can get one big static binary that we can upload, and include a webserver as part of the binary. This way we don't need to do the hack of running a container with Python or whatever. Now, what's even better about Rust for this is that it has macros, and specifically we are using [commandspec](https://github.com/tcr/commandspec/) which allows us to "inline" shell script. I think the macros could be even better, but this shows how we can intermix pure Rust code along with using shell safely enough. We're using my fork of commandspec because the upstream hasn't merged [a few PRs](https://github.com/tcr/commandspec/pulls?q=is%3Apr+author%3Acgwalters+). This model is intended to replace *both* some of our `make check` tests as well. Oh, and this takes the obvious step of using the Rust OSTree bindings as part of our tests. Currently the "commandspec tests" and "API tests" are separate, but nothing stops us from intermixing them if we wanted. I haven't yet tried to write destructive tests with this but I think it will go well.
As I tried to port the ostree tests, I hit various issues
with the
run-ext-bin
model; for example,kola run
alreadysupports
--parallel
and cleanly separates tests intooutput directories etc.
This reworks things so that one can do e.g.:
cosa kola run -- --parallel 4 --output-dir tmp/kola -E ~/src/github/ostreedev/ostree/tests/installed/nondestructive/ 'ext.nondestructive.*'
to run the ostree test suite. Or of course, run both the
ostree exttests plus builtin tests
basic
or whatever.And another thing I'd like to do soon is teach cosa how to
automatically add
src/config/tests
so that one can seamlesslye.g. add a change to the manifest and also add a test for it
as one atomic commit.
There's still a lot more to do here; among other things ostree
cleanly separates destructive/nondestructive tests and having
each test run as a new VM is a hit. But this is a good start
I think.