.cache/nix/gitv3/ folder deleted

[alvintang]

Describe the bug
We started getting the evaluation errors on active jobsets in our hydra server:

Jan 20 05:02:57 server-name hydra-evaluator[1136464]: 'nix flake metadata' returned exit code 1:
Jan 20 05:02:57 server-name hydra-evaluator[1136464]: warning: could not update mtime for file '/home/hydra/.cache/nix/gitv3/1gbz1m3n2zq9z50ll1xb8x63dscq1s5p975xf7cgaxvn7w82bnr3/refs/heads/<branch-name-here>': No such file or directory
Jan 20 05:02:57 server-name hydra-evaluator[1136464]: error: opening file '/home/hydra/.cache/nix/gitv3/1gbz1m3n2zq9z50ll1xb8x63dscq1s5p975xf7cgaxvn7w82bnr3/refs/heads/<branch-name-here>': No such file or directory

Checked out the actual path inside the server and /home/hydra/.cache/nix/gitv3/1gbz1m3n2zq9z50ll1xb8x63dscq1s5p975xf7cgaxvn7w82bnr3/refs/heads does not have any of the branches with the evaluation errors. We have the Github Pulls plugin setup and a new push to a branch checks out the branch again so the cache is re-populated and then the evaluation errors go away.

Our theory is something deleted the /home/hydra/.cache/nix/gitv3/1gbz1m3n2zq9z50ll1xb8x63dscq1s5p975xf7cgaxvn7w82bnr3/ directory but we couldn't figure out what. None of our cronjobs touch the .cache folder and nothing else is running in this server. Is there any nix or hydra process that could've deleted this directory?

The hydra master server (no slaves) has been running for almost 3 months now and encountered this just last Friday.

To Reproduce
Unable to reproduce since we are investigating the cause of the issue.

Expected behavior
the hydra-eval-jobset should check if the directory exist before running nix flake metadata

Screenshots
N/A

Hydra Server:

Please fill out this data as well as you can, but don't worry if you can't -- just do your best.

• OS and version: Ubuntu 20.04.5 LTS
• Version of Hydra: Hydra 2022-09-08
• Version of Nix Hydra is built against: nix-2.11.0
• Version of the Nix daemon: 2.7.0

Additional context
Add any other context about the problem here.

[JackKelly-Bellroy]

I have seen this error appear on a jobset tracking a fairly fast-moving master branch, which means that whatever's removing these cache directories is not doing so based on a timeout.

[bouk]

I think this is the same issue as NixOS/nix#8822