Implement GHA test automation #2631
Conversation
StephenBrown2
force-pushed
the
2630_gha_tests
branch
3 times, most recently
from
752d5c7
to
cfe1bd5
Compare
StephenBrown2
force-pushed
the
2630_gha_tests
branch
from
cfe1bd5
to
3fe42e3
Compare
|
We do not run our tests on Ubuntu - we are not Ubuntu based. For this to not be a distraction can it be modified to run on say what we actually run on. Then there is scope for expansion - we have had, and likely still have -some tests reaching the OS. They obviously shouldn't. But running our tests on at least one OS that we target (ideally all of them) is more sane in my opinion. They have to for our rpms to build in the end - given they are all build on their native OS and architecture (x86_64 and aarch64) so this may end giving false positives, or negatives. Or giving us less that it could if it ran on at least one of our own target OSs: openSUSE Leap 15.4, 15.5, and Tumbelweed, in either architecture we support: but likely we have way more x86_64 users. I'm assuming there is say a Leap 15.5 or Tumbleweed target in GitHub actions no? Thanks for kicking this off by the way. But we are OS specific and do-not and will not likely ever run on Ubuntu. That is a no-show I'm afraid. We are entirely already stretched without having to account for further OS variation in how-stuff-works differently (very differently some times) across distros. I know our tests are meant to be completely isolated - but as stated we have had a number of failures over the years re mocking so targeting a relevant OS is critical for this to be relevant. I know there can be value using differing bases - but for now I don't want to be looking into these just yet. We are only a few years into our new openSUSE base. From our prior CentOS base incidentally - dodged a bullet there no! Glad to have you contributing so don't get the wrong impression - but we are openSUSE and extremely invested there. So I just don't want to be seeing stuff that ends up being down to Ubuntu variations - and there are some as I've seen them in Python particularly as it goes. Remember we build our own venv - that is highly dependant on the host Python arrangement. Less variation for smaller teams is good. Thing problem space reduction :). |
Per the documentation, Github hosted Linux runners are Ubuntu only.
They do support OpenSuSE as a self-hosted runner. I did not expect this to be a turnkey solution, rather a starter that perhaps could get all the python/db tests that didn't require a specific base os running. Thankfully for the most part, Linux is Linux and Python is Python (write once, run everywhere?), so I would expect a majority to pass already.
I thought the point of using a venv was to specifically avoid being dependant on the host Python environment, otherwise, what's the point of being virtual?
Certainly not, I'm just trying to offer a solution, if only a partial one. Some automated tests are better than no tests, IMHO, and the tests that fail specifically because it's not on OpenSuSE could be marked as such and skipped when not in the expected environment.
Very much agreed, and to that point, the more automation the better. Do more with less! |
|
Not sure how the Rockstor team wants to proceed with this, since a self-hosted runner will likely cost some money. Linode does have Leap 15.4 and 15.5 images, as well as support for custom images, with which one could simply upload the generated OS images and have the "closest to prod" testing environment imaginable. Available distributions: Upload custom disk image: Install Rockstor on Linode from interface, then can clone and reuse: They offer a GitHub action step to install their CLI as well, which could be used to spin up and down runners for tests: https://github.com/linode/action-linode-cli |
|
@StephenBrown2 Thanks for the research there - Nice. We can't really add to our current expenses really: given our current Open Collective member count. Plus I'm not that keen on adding to our infrastructure dependencies anyway. We do have self hosting via a bunch of in-house VM's that we build the rpms on - and all tests are run during that process (see %check rpm scriptlet). So this would just be an un-affordable expense. We could hang-up this PR until GitHub actions expands beyond a single linux distro! It is a relatively new feature after all.
Nice find. But yes, all cost that we can't currently shoulder. See our open collective for current costs the project has to cover currently. We are after sustainable development here after all. But on the plus side we can drop that Shopify cost now given our recent axing of their requirement (Few!). So after dropping/reducing cost really - not adding to them. Re custom images: we use generic upstream Minimal-VM (previously called JeOS) for our build machines to try not to deviate too far from up-stream. DO incidentally dropped their native openSUSE support so one has to take the custom image route there as the only option - then use cloud orientated images! So maybe we put this pr on hold until more capable/compatible options arise. There used to be a lot of free tier CI options available to projects such as ours but unfortunately the greed & abuse of Crypo miners on mass put paid to that and they have pretty much all been turned off - quite a set-back to many non-profits such as us. |
Sounds like a plan. I'll close this PR and spin up a forum thread in a little bit to see if the community knows of any open collective friendly options exist. |
|
Forum post for further discussion: https://forum.rockstor.com/t/test-automation-for-rockstor-core-pull-requests/ |
This should be a good start for ensuring tests run on every PR.
Closes #2630