Configurable substituter disable time

[cidkidnix]

Allow to change how long substituters are disabled for based on a nix.conf config option
Still need to add a better explanation to globals.hh

[Ericson2314]

Yeah I think most magic numbers like this should be config options. There is no globally optimal choice.

[edolstra]

Just because there is no globally optimal choice, doesn't mean adding a configuration option is a good idea. The existence of options like these is generally a sign of bad design, since it punts the problem to the user. We should try to make things work out of the box, without user configuration.

What's the use case for a different interval?

[cidkidnix]

We only disable a substituter temporarily (assuming fallback and others are enabled) at the moment (from what I can see), and it's hard set at 60sec. I'd like to be able to configure this so we don't re-trigger a substituter being checked "too-soon" for my use-case. Basically I want to be able to change how long in-between that happens.

[thufschmitt]

I agree that adding an option doesn't look like a nice solution here. Ideally Nix should just behave in a way that's satisfying for everyone in this kind of scenario. Would some kind of exponential back off work better for your use-case?

[Ericson2314]

The current code is absolutely atrocious for our use-case because if one substituter is inaccessible, it won't try any other.

I agree good design means one shouldn't need to change configuration options. E.g. if it will try the substitutors concurrently a 60 second timeout isn't so bad.

But just cause it isn't necessary to change the magic numbers doesn't mean having a config object is bad. IMO there is a big gap between on one hand obscurist endlessly configurable consultantware, and on the other patronizingly configuration-free Apple-style software. A middle ground of "there are settings but you shouldn't need to change them" feels fine to me.

[cidkidnix]

Though yes what @Ericson2314 said is 100% a problem (although workaround able with #7188 or enabling fallback currently), this PR further allows you to configure how long substituters that are down/missing will stay disabled locally.

[Ericson2314]

#7188 still tries them sequentially, so a 60 second timeout can be annoying. We would need to them concurrently for it to not be annoying.

[domenkozar]

See #7188 (comment)

[nixos-discourse]

This pull request has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/2023-07-24-nix-team-meeting-minutes-75/31112/1