CI: Move non-coverage FreeBSD Cirrus CI jobs to GitHub Actions

[kinke]

By running them in a qemu VM on Linux runners.

[dlang-bot]

Thanks for your pull request and interest in making D better, @kinke! We are looking forward to reviewing it, and you should be hearing from a maintainer soon.
Please verify that your PR follows this checklist:

• My PR is fully covered with tests (you can see the coverage diff by visiting the details link of the codecov check)
• My PR is as minimal as possible (smaller, focused PRs are easier to review than big ones)
• I have provided a detailed rationale explaining my changes
• New or modified functions have Ddoc comments (with Params: and Returns:)

Please see CONTRIBUTING.md for more information.

---

If you have addressed all reviews or aren't sure how to proceed, don't hesitate to ping us with a simple comment.

Bugzilla references

Your PR doesn't reference any Bugzilla issue.

If your PR contains non-trivial changes, please reference a Bugzilla issue or create a manual changelog.

Testing this PR locally

If you don't have a local development environment setup, you can use Digger to test this PR:

dub run digger -- build "master + dmd#15618"

[kinke]

This primarily shows

• the ugly required YAML duplication / no way to reuse the native job steps (YAML anchors support wouldn't help here either),
• the poor visual separation of substeps in the log, and
• the surprisingly bad performance, at least with qemu on Linux here - just managing to build DMD and start the DMD testsuite in an hour, which hints at something like a 10x slow-down vs. a native job. Never seen such a slowdown with qemu on my local Linux box, where the performance is near-native.

[kinke]

Okay, the performance on Mac runners is much better - the ci/run.sh build step takes 5-6 minutes in the FreeBSD VM (derivable by enabling the log timestamps in the UI - this took something like 35 minutes in the qemu VM on Linux), vs. 2-3 minutes for the native Linux/Mac jobs. The 'unit' tests fail (also did in the qemu VMs on Linux), otherwise the DMD testsuite at least passes.

[ibuclaw]

23-09-19T16:49:41.2233380Z 1) global.deinitialize
2023-09-19T16:49:41.2235820Z core.exception.AssertError@test/unit/deinitialization.d(152): unittest failure

This appeared 7x - once for each deinitialize member function.

dmd/compiler/test/unit/deinitialization.d

Line 152 in 5b8e829

assert(objc is null);

Does FreeBSD even have objc support in dmd?

[kinke]

Right interesting, I didn't realize it's complaining about a single assertion. I find it bizarre that this unittest is apparently invoked multiple times, makes absolutely no sense (Objc.deinitialize, the unittest with that assertion, should be the only failure). The failing assertion hints at that the frontend state hasn't been deinitialized at the time the test is run - dmd.objc.objc is null after dmd.objc.Objc.deinitialize(), and set in dmd.objc.Objc._init() (to a class instance of either dmd.objc.{Supported,Unsupported}).

So my gut feeling is that we only see follow-up errors, maybe a missing dmd.frontend.deinitializeDMD() after a real-failing test - there are lots of

@beforeEach initializeFrontend()
{
    import dmd.frontend : initDMD;
    initDMD();
}

@afterEach deinitializeFrontend()
{
    import dmd.frontend : deinitializeDMD;
    deinitializeDMD();
}

for these tests.

[ibuclaw]

Only just got round to running these in my own FBSD VMs, all unit tests pass. Rebase and re-run?

[kinke]

Well IIRC, the tests were just fine in the Cirrus FreeBSD VMs before too. So I very much doubt a rebase changes anything, but I'll try. Edit: Ah okay, you did, but no changes apparently.

[ibuclaw]

Shame, I've ran the github steps as a script directly on my VM, and it doesn't trigger.

[ibuclaw]

@kinke looks like all I can do is fudge the test/run.d file to skip the unittest target on FreeBSD.

However, it looks like the std.concurrency unittest has hung. 🤷

[kinke]

Oh well, thanks a lot for trying though! 👍 - I've just seen that the Action has been updated to v0.21.0 very recently; bumping it here (currently using v0.19.0) might be worth a shot: https://github.com/cross-platform-actions/action/releases

[ibuclaw]

Oh well, thanks a lot for trying though! 👍 - I've just seen that the Action has been updated to v0.21.0 very recently; bumping it here (currently using v0.19.0) might be worth a shot: https://github.com/cross-platform-actions/action/releases

Unless most of the changes in the VMs available are made behind the scenes, I doubt it.

Having a look at the alternative vmactions/freebsd-vm - it looks like since 2 weeks that VM doesn't boot anymore.

[ibuclaw]

Updated to v0.21.0 - looks like both have hung again. The logs aren't rendering for me atm, but appears to be this time in different places to before. So maybe the VMs themselves are unstable to be used for 15 minutes at a time? GItHub says its hosted OSX runners have 1 CPU, and 14G memory. The cross-platform action says it gives its FreeBSD VMs 6GB memory by default. I'm not convinced hitting some resource limit is occurring, but I have seen hanging before because of lack of memory.

[kinke]

GItHub says its hosted OSX runners have 1 CPU, and 14G memory. The cross-platform action says it gives its FreeBSD VMs 6GB memory by default.

The hosted Mac runners have 3 cores (https://docs.github.com/en/actions/using-github-hosted-runners/about-github-hosted-runners/about-github-hosted-runners#supported-runners-and-hardware-resources). And cross-platform uses these 3 by default, and 13G of memory (https://github.com/cross-platform-actions/action#inputs). Maybe the 1GB for the host is a tad too low? We surely don't need 13G in the VM, 8G should be more than enough.

[ibuclaw]

Spoken to Jacob, and freezing is probably related to cross-platform-actions/action#29 and/or cross-platform-actions/action#61.

Going to try MacOS+QEMU combination.

Last test results: FBSD13 passed, FBSD12 hung.

[ibuclaw]

@kinke looks like both succeed with qemu hypervisor? Just going to undo some of the other stuff to see if that trend continues...

https://github.com/dlang/dmd/actions/runs/6671677680/job/18134093839?pr=15618

[kinke]

Oh wow - and the performance seems to at least match the default other hypervisor's, way better than on the Linux host! I must admit I didn't even consider giving the Mac+qemu combo a try after the first results.

I guess the 'unit' tests still fail - it did with qemu on Linux for the first experiments.

[ibuclaw]

With N=3 it's down to 20 minutes.
https://github.com/dlang/dmd/actions/runs/6672127402/job/18135449280?pr=15618

[kinke]

Looks like there's room to shave off an additional ~1m20s (from https://github.com/dlang/dmd/actions/runs/6672331301/job/18136041945?pr=15618):

Fri, 27 Oct 2023 21:46:01 GMT Tearing down VM
Fri, 27 Oct 2023 21:46:01 GMT   Syncing back files
Fri, 27 Oct 2023 21:46:01 GMT   /usr/bin/rsync -uzrtopg runner@localhost:work/ /Users/runner/work
Fri, 27 Oct 2023 21:46:03 GMT   skipping non-regular file "dmd/dmd/dmd"
Fri, 27 Oct 2023 21:46:03 GMT   skipping non-regular file "dmd/phobos/generated/freebsd/release/64/libphobos2.so"
Fri, 27 Oct 2023 21:46:03 GMT   skipping non-regular file "dmd/phobos/generated/freebsd/release/64/libphobos2.so.0.105"
Fri, 27 Oct 2023 21:47:20 GMT   Shuting down VM

We don't need any syncing, we don't produce any artifacts etc.

Edit: Should be sync_files: runner-to-vm AFAICT (copy the DMD repo into the VM, but don't copy any build artifacts back to the host).

[ibuclaw]

Using the default memory (13G) and it's back up to 27-30 minutes again. I guess still within reasonable times given it's a VM on a shared runner. Reducing memory back down to 8G anyway.

https://github.com/dlang/dmd/actions/runs/6672331301/job/18136041945?pr=15618

I also noticed it takes about a minute and half to tear down the VM, looks to be consumed by rsync'ing the files in the VM back to the runner. Trying sync_files: runner-to-vm to shave that time off.

[ibuclaw]

We don't need any syncing, we don't produce any artifacts etc.

Edit: Should be sync_files: runner-to-vm AFAICT (copy the DMD repo into the VM, but don't copy any build artifacts back to the host).

Ah! I see we're talking over each other. I already had the change primed, was just waiting for the run to finish to count just how long it takes. :-)

[kinke]

Using the default memory (13G) and it's back up to 27-30 minutes again.

Oh, if that's consistent, we should probably tell Jacob to change the default value, at least on macOS, giving the host more room to breathe.

[kinke]

LVGTM.

[ibuclaw]

I say this is for no good reason - however when commenting out deinitialization.d:152 explicitly the same module then just failed at deinitialization.d:131.

This could suggest that al globals have an initial value other than null. Which begs the question: Why?, but actually also Why not? - afterall as a security feature, you can tell a compiler to default initialize all variables with a bit mask value (as opposed to zeros or garbage). Though I am struggling to justify this for global data. But equally surely you're testing it wrong if you're checking the value before it's been initialized. 🤷

[ibuclaw]

Need to really get all pipelines green again, and this is progress in getting there.

[jacob-carlborg]

@kinke FYI, the Linux runners don't support hardware accelerated nested virtualization (KVM) while the macOS runners do (Hypervisor framework). That's why the big difference in performance.

[jacob-carlborg]

no way to reuse the native job steps

It depends on what the steps do. I try to do as little as possible inside the VM. Because of performance and to be able to use the GitHub action steps. That's also why files are synced in both directions. For example, you can do a checkout before the VM step and then use a step to create GitHub release after the VM step.

[kinke]

I try to do as little as possible inside the VM.

I fully understand that it'd be total overkill to be able to run arbitrary actions in the VM, with all that node.js crap etc. I was just comparing this to the Cirrus CI approach, where the VM setup is done as built-in job properties (e.g., https://github.com/ldc-developers/ldc/blob/9928895c548af3789c7fcc7947d5e9b3ac152ae9/.cirrus.yml#L307-L314), and the steps are basically limited to bash steps AFAIK, so the native and VM steps can be trivially shared via YAML anchors (https://github.com/ldc-developers/ldc/blob/9928895c548af3789c7fcc7947d5e9b3ac152ae9/.cirrus.yml#L376 - another feature that GHA doesn't support).

Considering the GHA limitations/flexibility, I can't think of anything your action could do better to improve our use case here. :) Thanks a lot for your hard work!