Make number of parallel substituter queries configurable #5324
Conversation
| @@ -250,6 +250,17 @@ public: | |||
| )", | |||
| {"build-use-substitutes"}}; | |||
|
|
|||
| Setting<unsigned int> maxQuerySubstitutersJobs{ | |||
There was a problem hiding this comment.
Wouldn't it be better to just check if the substituter supports http/2 and then significantly increase this setting?
There was a problem hiding this comment.
The better solution is not to use threads at all here, which would obviate the need for any parallelism guessing by making that the sole responsibility of libcurl. This here is the trivial fix hopefully in time for Nix 2.4 that at least gives people an option to max out the connections. Any more complex solution should come later.
Kha
force-pushed
the
max-query-substituters-jobs
branch
from
1ef3b45
to
b3104c2
Compare
|
@edolstra I'd really love to get this into 2.4 as slow querying of binary caches makes it a really bad experience for bigger projects. |
|
This pull request has been mentioned on NixOS Discourse. There might be relevant details there: https://discourse.nixos.org/t/ignore-offline-substituters/15450/2 |
|
This pull request has been mentioned on NixOS Discourse. There might be relevant details there: https://discourse.nixos.org/t/ignore-offline-substituters/15450/3 |
|
@edolstra Any chance you can take a look? |
|
This pull request has been mentioned on NixOS Discourse. There might be relevant details there: https://discourse.nixos.org/t/nix-2-4-release-candidate/15393/2 |
|
@edolstra any reason to hold this? |
|
It's probably better to rewrite |
|
@edolstra that would be great. Merging this will fix query performance for low-core counts until we get the async version done. |
This is the minimal fix for #5118. The default is taken from
http-connections, though as stated, multiplexing can mean that a far higher number is even more effective. Still, the new default should be a sizeable improvement over the old one ofnprocfor most machines while still low enough that time and memory overhead from the additional threads should be insignificant.