Broken since February 2023

[Ambrevar]

The Install Guix phase has recently started failing with

[1677520229.916]: [ INFO ] Linking the root user's profile
[1677520230.118]: [ PASS ] activated root profile at /root/.config/guix/current
[1677520230.169]: [ PASS ] group <guixbuild> created
[1677520230.173]: [ INFO ] group kvm exists and build users will be added to it
[1677520230.246]: [ PASS ] user added <guixbuilder01>
[1677520230.270]: [ PASS ] user added <guixbuilder02>
[1677520230.297]: [ PASS ] user added <guixbuilder03>
[1677520230.339]: [ PASS ] user added <guixbuilder04>
[1677520230.375]: [ PASS ] user added <guixbuilder05>
[1677520230.420]: [ PASS ] user added <guixbuilder06>
[1677520230.452]: [ PASS ] user added <guixbuilder07>
[1677520230.479]: [ PASS ] user added <guixbuilder08>
[1677520230.512]: [ PASS ] user added <guixbuilder09>
[1677520230.542]: [ PASS ] user added <guixbuilder10>
Created symlink /etc/systemd/system/guix-daemon.service.wants/gnu-store.mount → /etc/systemd/system/gnu-store.mount.
Unit /etc/systemd/system/gnu-store.mount is added as a dependency to a non-existent unit guix-daemon.service.
Created symlink /etc/systemd/system/multi-user.target.wants/guix-daemon.service → /etc/systemd/system/guix-daemon.service.
[1677520232.146]: [ PASS ] enabled Guix daemon via systemd
[1677520232.149]: [ INFO ] making the guix command available to other users
[1677520232.474]: [ PASS ] Authorized public key for ci.guix.gnu.org
[1677520232.632]: [ PASS ] Authorized public key for bordeaux.guix.gnu.org
[1677520232.647]: [ PASS ] installed shell completion
[1677520232.701]: [ PASS ] Bash shell prompt successfully customized for Guix
[1677520232.703]: [ INFO ] cleaning up /tmp/guix.KiwyEv
[1677520232.705]: [ PASS ] Guix has successfully been installed!
[1677520232.707]: [ INFO ] Run 'info guix' to read the manual.
[1677520232.709]: [ INFO ] Please log out and back in to complete the installation.
yes: standard output: Broken pipe
Error: Unable to process file command 'env' successfully.
Error: Access to the path '/home/runner/work/_temp/_runner_file_commands/set_env_ce49b965-d6e4-4c91-8f30-d2ab253d8be5' is denied.
Error: Unable to process file command 'state' successfully.
Error: Access to the path '/home/runner/work/_temp/_runner_file_commands/save_state_ce49b965-d6e4-4c91-8f30-d2ab253d8be5' is denied.
Error: Unable to process file command 'output' successfully.
Error: Access to the path '/home/runner/work/_temp/_runner_file_commands/set_output_ce49b965-d6e4-4c91-8f30-d2ab253d8be5' is denied.
Error: Unable to process file command 'env' successfully.
Error: Access to the path '/home/runner/work/_temp/_runner_file_commands/set_env_00feac3d-a6fe-4153-b14f-0323783bb262' is denied.
Error: Unable to process file command 'state' successfully.
Error: Access to the path '/home/runner/work/_temp/_runner_file_commands/save_state_00feac3d-a6fe-4153-b14f-0323783bb262' is denied.
Error: Unable to process file command 'output' successfully.
Error: Access to the path '/home/runner/work/_temp/_runner_file_commands/set_output_00feac3d-a6fe-4153-b14f-0323783bb262' is denied.

I presume it's due to a change in the GitHub CI API. @PromyLOPh Any idea?

[PromyLOPh]

My guess would have been a change to Guix’ guix-install.sh, but seeing that new jobs[1] currently simply get stuck it could be a GitHub issue. [1] https://github.com/PromyLOPh/guix-install-action/actions/runs/4285821379

[Ambrevar]

Yes, also notice how the "Install Guix" phase is not even clickable. Looks like a GitHub breakage.

[jmercouris]

Can we verify that it is a GitHub breakage? A way to isolate the changes?

[PromyLOPh]

I’m seeing warnings/errors like these in the “summary” tab of the action:

The hosted runner: Hosted Agent lost communication with the server. Anything in your workflow that terminates the runner process, starves it for CPU/Memory, or blocks its network access can cause this error.
Unable to process file command 'env' successfully.
Access to the path '/home/runner/work/_temp/_runner_file_commands/set_env_11576070-b03e-4973-a286-2478e86a4cd6' is denied.
Unable to process file command 'state' successfully.

No idea what that means though.

[jmercouris]

Well, there is a temporary GitHub CI outage, however I do not think this is related to that. I guess we will find out when the service is restored if something was awry.

[ph]

I've been experimenting with the action and I have the same results as you above, looking a the github status page everything should be green. Probably something change in how they run their working node for the workflow.

[jmercouris]

I concur, I tested again recently, and same result :-(

[ph]

I did some messing around with yes | bash based on another issue reported on macOS action runner actions/runner#884.

No luck so far.

[PromyLOPh]

I’m also seeing extremely weird behaviour when trying to run any command after guix-install.sh. A simple ls /home/runner/work/_temp/ in the same step will just fail with

/home/runner/work/_temp/19176e17-ffd7-4e56-97cc-3559cb2950d3.sh: line 4: /usr/bin/ls: Permission denied

[jmercouris]

Interesting that it will fail for ls, that I did not expect. Are you able to drop into a shell on the machine after the job fails? Is there a way to run a Github Action as root? Perhaps there is something we can tweak in the action to run as root.

[PromyLOPh]

I’m running the actions on GitHub’s runners (i.e. push and wait), so there’s no way to drop into a shell.

[jmercouris]

That's unfortunate. I did check the GitHub changelog, and couldn't find anything of note. Perhaps I didn't look hard enough.

[TxGVNN]

yes: standard output: Broken pipe

Hi, I think this is time we should pin guix-install.sh version into this repo instead get latest from upstream (Reproducible).
And we modify guix-install.sh to run directly without pipe yes | bash guix-install.sh

[TxGVNN]

I created a PR to try resolving this issue. Could you guys have a look? And we can improve better. Thanks

[PromyLOPh]

It looks like yes and pipes are not at fault here. Even just running tar -xf guix-binary.tar.xz -C / breaks the runner. Turns out the archive has a member

drwx------ root/root         0 1970-01-01 01:00 ./

which explains the “permission denied” issues. And GNU tar has no option to ignore permissions in the supplied file.

[Ambrevar]

Thanks!