ci: move BSD testing from sourcehut to Cirrus CI

[dundargoc]

dispatch.sr.ht is being deprecated, meaning that using sourcehut CI
won't be possible (see #19609).
Since Github Actions doesn't provide any BSD runners an external service
is required and Cirrus CI seems like a good replacement for sourcehut.

Initially experimented with using FreeBSD and OpenBSD virtual machines
in GitHub Actions, but Cirrus has been a much better fit with better
performance, logs and overall experience.

Failing tests are automatically skipped on FreeBSD regardless it it's on
CI or not. Ideally these tests should only be skipped in CI with the
help of isCI helper function. Unfortunately, the tests don't recognize
the environment variable CIRRUS_CI even if it's set manually. This
workaround is good enough for the time being, but we might want to only
skip tests when using the CI (or even better, fix the failing tests).

Closes: #19609

Reminder for myself

Dispatch will shut down 2022-10-01, this will need to be merged by then.

[justinmk]

Ok, but can we couple this with https://github.com/vmactions (see #19609 )? For BSD we at least want to check that builds work (library headers are available, etc.), if not the full test suite.

[dundargoc]

@justinmk ok, I've gotten freebsd to work mostly. There are some tests somewhere that fail which I'll look into, but if you want to see how it will looks overall: https://github.com/neovim/neovim/runs/7638940566?check_suite_focus=true

Initial impressions of vmactions:

Pros:

• It's easy to set up
• It's on github, which means it plays nicely with github-features such as "restart failed jobs"

Cons:

• The vms have the performance of a potato
• It requires macos as the runner, which github is stingy with. This and because the runs take a long time means that this could create huge CI queues.
• Because this needs to be run in a single step, all of the logs must be loaded at once, making the github UI a no-go. We'll need to download the logs if we'll want to debug this.

[justinmk]

• The vms have the performance of a potato

Let's only build, then. Don't run the tests.

• It requires macos as the runner, which github is stingy with. This and because the runs take a long time means that this could create huge CI queues.

🤦 This is more of a problem.

[clason]

Let's only build, then. Don't run the tests.

Hmmm, we recently (with the LibUV CMake PR) had a situation where the build was successful at the face of it but failed as soon as a functionaltest was run.

Maybe we can define a (really) minimal "core" testsuite that exercises some very basic Lua/LibUV functionality? E.g., pick a single suitable TEST_FILE and just run with that?

(That would also be a candidate to run on draft PRs instead of the full suite, relieving CI pressure.)

[dundargoc]

Maybe we can define a (really) minimal "core" testsuite that exercises some very basic Lua/LibUV functionality? E.g., pick a single suitable TEST_FILE and just run with that?

Hmm, yeah, I like this idea. Just as a quick sanity check.

I'm also wondering if we should pick one BSD distro as well. That is instead of checking if 2 BSDs build successfully and pass a few tests, we test one BSD and see if they pass a lot/most/all tests? I don't have good insight into the differences but both have "BSD" in their name so they can't be that different, right? :)

[clason]

oldtest is a) just as slow and b) doesn't exercise Lua...

Maybe make functionaltest TEST_FILE=test/functional/lua/vim_spec.lua? (One of the bigger tests, but arguably the most important one.) Or api_spec.lua? Or loop_spec.lua (would target libuv/luv directly)?

[clason]

Looks like some (terminal) tests really don't like the VM...

[dundargoc]

@justinmk @jamessan I'd say this is pretty much finished.

Opted to only use openbsd as freebsd takes way too long for some reason. Only run the tests in functional/lua since 1. they don't fail and 2. so we don't risk CI queues if we start small due to the macos restrictions.

We can always adjust/add more tests if we notice that our CI budget allows it.

[jamessan]

https://github.com/cross-platform-actions/action uses a different hypervisor. I wonder how its performance compares to vmactions.

[clason]

https://github.com/cross-platform-actions/action uses a different hypervisor. I wonder how its performance compares to vmactions.

Also runs on Linux, which is a definite plus in itself.

[dundargoc]

I can't get the cirrus caching to work. Let's try without any caching since it's sufficiently fast enough anyway.

[clason]

@jamessan @justinmk green light?

[justinmk]

TODO / off topic: could make is_os('...bsd') work

[dundargoc]

Hmm, yeah could work. I'm not against it, but let's save it for a future discussion like you said.

[justinmk]

looks great. thank you for this @dundargoc