-
-
Notifications
You must be signed in to change notification settings - Fork 13.5k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Fetching paths from cache.nixos.org can get 'stuck' #160289
Comments
I'm seeing the same just now. For me lots of builds worked today, but I get stuck at
|
It seems to be related to remote builds. I have to https://discourse.nixos.org/t/remote-building-experienence-massively-degraded-since-2-3-months/16950 |
Hmm, I don't use remote builds though
Which processes? |
if you run that as a normal user it'll ignore you |
I ran it with After the reboot and failed attempt I tried to |
(I sudo'ed, and verified in |
ohh, did kill them. I misread. Well after I ran into this problem a couple times and removed remote builders I've not encountered it again since. |
This issue has been mentioned on NixOS Discourse. There might be relevant details there: |
How? Here's a fix for a deadlock involving CurlDownloader and the interrupt handlers. May or may not be related. Does it repro with this fix applied? |
Sorry I should have said that I have the same issue. I'll apply that pr and see if I get this issue anymore |
i've been running nix with this overlay since 5 days ago and i haven't gotten the same problem
updated #158455 to the newest nix commit to include the pr |
@Artturin Just as a warning that you might just be lucky: I also ran into the stuck downloads once and after following the instructions by @kenranunderscore ( Did anyone figure out how to reproduce this bug? |
I also haven't seen stuck downloads for the past week or so... |
I'm still getting this on NixOS unstable with Nix 2.6.1. Is there a workaround that doesn't require collecting garbage? I have a couple compute-expensive derivations on my hard drive I'd rather not have to rebuild.... |
reboot and optionally add gcroots for derivs |
I haven't seen this for a while now. Let's close for now and re-open when we see evidence it's still a problem. |
I've hit this issue a few times in the past week. It's always the first package that is attempted to be copied from cache.nixos.org, which for me is |
Interesting. The problem hasn't returned for me, and I think in my case it wasn't the first package, so it might not be the same problem. Perhaps it would be better to add a new issue (and link to this one for possible background) rather than re-open this? |
I was hit by a similar problem (but copying files from host to a vds via remote nixos-rebuild). nix-collect-garbage on both host and vds did nothing but upgrading host's system did solved it |
In my case, this was due to a bad routing configuration between T-mobile and Fastly CDN (which hosts cache.nixos.org), which was causing network traffic to cache.nixos.org to route from LAX to Argentina (traceroute indicated this). Fastly/tmobile fixed this specific routing table misconfiguration, but since then I've moved to my own substituter on cloudfront so I can't say whether the issue has occurred again. |
Encountered a similar issue: nix operations would just hang when trying to download (the perl package in my case). Was able to resolve with:
plus killing all active/hung Running |
… + switch to nixos-stable channel Fix from NixOS/nixpkgs#160289
I also have seen this a few times. Killing the daemon did the trick. |
I've been having this issue in my Docker containers. After losing the docker cache, I have been unable to rebuild my images at all. Stuck at Pinning my tags to |
Saw this on a system today. I think in this particular instance it was triggered through some odd interaction with one remote builder being unavailable; removing it from the builders list, |
I just ran into this. The fix for me was to kill all stuck |
Describe the bug
Sometimes, nix gets stuck fetching paths:
Stalling without any network traffic.
Interestingly this seems to be somehow tied to the specific path: nix can download other paths just fine, but always gets stuck on (in this case)
/nix/store/0kg8zm25k6av8hgf177kwnp9q5j6yrf3-openjdk-17.0.1+12
.It has something to do with the system state: rebooting the machine makes it possible to fetch the path again, but killing all
nix-daemon
processes without rebooting doesn't seem to help.edit: it does seem to be something around nix-daemon though, because when rebooting I see systemd-shutdown waiting for several nix-daemon processes to shut down which takes quite a while.
If anyone can suggest ways to further diagnose the problem I'd be happy to try things out!
Metadata
Please run
nix-shell -p nix-info --run "nix-info -m"
and paste the result.(current profile is at 9f697d6, nixpkgs is at 48d63e9)
The text was updated successfully, but these errors were encountered: