Support FreeBSD as sccache-dist server

[grembo]

FreeBSD support makes use of pot for sandboxing.

Implementation aimed to be minimally invasive, almost all
FreeBSD specific parts happen in segregated files.

We've been using this successfully in CI for over a month now, so it
appears to be stable enough to be potentially useful to others.

[sylvestre]

Could you please add a job in the ci to test this on freebsd? Thanks

[codecov-commenter]

Codecov Report

Base: 30.03% // Head: 30.02% // Decreases project coverage by -0.01% ⚠️

Coverage data is based on head (b4e85fb) compared to base (635b2a1).
Patch has no changes to coverable lines.

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #1184      +/-   ##
==========================================
- Coverage   30.03%   30.02%   -0.02%     
==========================================
  Files          47       47              
  Lines       16642    16642              
  Branches     7870     7877       +7     
==========================================
- Hits         4999     4996       -3     
+ Misses       6334     6331       -3     
- Partials     5309     5315       +6     

Impacted Files	Coverage Δ
src/compiler/compiler.rs	35.91% <ø> (+0.14%)	⬆️
src/config.rs	35.64% <ø> (-0.18%)	⬇️
src/jobserver.rs	47.91% <0.00%> (-2.09%)	⬇️
src/compiler/gcc.rs	54.33% <0.00%> (-1.04%)	⬇️
src/util.rs	33.44% <0.00%> (-0.33%)	⬇️
src/lru_disk_cache/mod.rs	41.03% <0.00%> (-0.31%)	⬇️
src/compiler/c.rs	38.26% <0.00%> (-0.23%)	⬇️
src/lib.rs	9.04% <0.00%> (-0.06%)	⬇️
src/test/utils.rs	37.37% <0.00%> (ø)
... and 7 more

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

☔ View full report at Codecov.
📢 Do you have feedback about the report comment? Let us know in this issue.

[sylvestre]

a way to do it:
https://github.com/uutils/coreutils/blob/main/.github/workflows/CICD.yml#L902-L919=

and please run rustfmt on your change

[grembo]

a way to do it: https://github.com/uutils/coreutils/blob/main/.github/workflows/CICD.yml#L902-L919=

and please run rustfmt on your change

Thanks for the pointer, I'll look into it.

[sylvestre]

@grembo do you have an update on this? thanks

[grembo]

@grembo do you have an update on this? thanks

@sylvestre Thanks for asking, no updates yet, as I’ve been too busy with other things, but it will definitely happen soonish.

[grembo]

@sylvestre I rebased the patch and added a series of CI tests for FreeBSD. With those I also found a small issue with creating server tokens which I fixed in the process. Tests for the latest push worked out ok in my account's actions, see https://github.com/grembo/sccache/actions/runs/2619631534 (assuming it is visible to you).

They won't run on your account though, as the required actions aren't allowlisted:

Error: .github#L1
swatinem/rust-cache@v1 and vmactions/freebsd-vm@v0.1.6 are not allowed to be used in mozilla/sccache. Actions in this workflow must be: within a repository that belongs to your Enterprise account, created by GitHub, or matching the following: !/mozilla/, !mozilla/, ./**, 10up/wpcs-action@, aws-actions/, codecov/codecov-action@v2, docker/, pypa/gh-action-pypi-publish@v1.4.2, slackapi/slack-github-action@, google-github-actions/*, erlef/setup-beam@v1.

(see https://github.com/mozilla/sccache/actions/runs/2619631572)

[grembo]

@sylvestre I removed the dependency rust-cache and updated the testsuite to be based on macos-12, as it appears to be more stable.

See https://github.com/grembo/sccache/runs/7212100672?check_suite_focus=true#logs for successful CI run in my repo (you would have to allow vmactions/freebsd-vm@b528c16ee726e448eda5b1a78a96b7dc81899b02 or a wildcard version of that in your actions).

[grembo]

@sylvestre Rebased again to keep track with @dependabot

Is there anything else from my side that's required at this point?

[drahnr]

No, at this point it requires some verification on my end, but I won't have time to setup a test env until end of next week.

[grembo]

No, at this point it requires some verification on my end, but I won't have time to setup a test env until end of next week.

Cool, thanks for spending time on this. It's certainly not perfect, but works reliably for us.

[sylvestre]

Sorry again.
I was looking at merging this but the patch needs some rebasing :(

[grembo]

Sorry again.
I was looking at merging this but the patch needs some rebasing :(

Rebased and merged some redundant changes (e.g., adding exp: to JWT, which my patch included).

Actions ran successfully in my fork:

[sylvestre]

Thanks.
I opened this bug on the Mozilla side to get this github action enabled:
https://bugzilla.mozilla.org/show_bug.cgi?id=1796182

[drahnr]

Note I just setup freebsd, so I am not at the stage to confirm functionality.

[sylvestre]

Yeah, the github action has been activated:
https://github.com/mozilla/sccache/actions/runs/3290878677/jobs/5424313325

[grembo]

@sylvestre Moving parts of the action to a script (which then could also be used for local testing) makes a lot of sense. It will also allow to runs tests in a loop on beefier hardware to check test stability.

It's a bit unclear what happened with the previous runs (timing out after 6 hours, while a build should take < 1 hour):

I'll also add "copyback: false" to the CI action, as there are no artifacts we want to copy out (maybe that's part of the issue).

[grembo]

@sylvestre Like you suggested, I moved most of the CI actions into a separate script, which now allow to run the test locally. I ran this in a loop and after some tinkering it seems to be stable. Also did a few other minor improvements like outlined in the commit log.

[sylvestre]

Much better, thanks

[sylvestre]

Seems that they regressed on

  error[E0308]: mismatched types
    --> /opt/tester/.cargo/registry/src/github.com-1ecc6299db9ec823/core_affinity-0.7.5/src/lib.rs:51:5
     |
  50 | pub fn set_for_current(core_id: CoreId) -> bool {
     |                                            ---- expected `bool` because of return type
  51 |     set_for_current_helper(core_id)
     |     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ expected `bool`, found `()`
  

Rings a bell ?
thanks

[grembo]

core_affinity

I downloaded the repo, debugged it and was about to open a pull request, just to find that there is one already:

Elzair/core_affinity_rs#15

[drahnr]

Thank you, I was about to hit the button 👍

[sylvestre]

@grembo it is now in the 4.0.0 pre1 release
freebsd binaries aren't available here:
https://github.com/mozilla/sccache/releases
would you like to update the CI to generate them?

[grembo]

@grembo it is now in the 4.0.0 pre1 release freebsd binaries aren't available here: https://github.com/mozilla/sccache/releases would you like to update the CI to generate them?

@sylvestre Sounds like a good idea. How would you approach it though? Right now we build and test in one target (which does not build a release version). Creating a separate build target for FreeBSD would be an option, but the mac-wrapping is already quite expensive - maybe build the release after build/test worked out ok (and then upload the artifacts)? Otherwise, if this consumes too many resources, sccache-dist will certainly become part of the FreeBSD sccache port, so it will also be available to users using pkg install eventually.