-
-
Notifications
You must be signed in to change notification settings - Fork 13.6k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Complete build on "wide" public cloud? #30525
Comments
Even with 1-second billing, it's still advantageous to reuse instances to prevent having to download the same dependencies over and over again. Hydra doesn't currently schedule builds in a way to minimize downloads, but there will be a lot of overlap between dependencies anyway (e.g. stdenv will be the same for most builds).
|
If we're really talking about thousands of machines, then Hydra is not really appropriate due to its centralized architecture. This would require a different architecture to be scalable. E.g. each worker node independently fetches work items from a queue, where a work item is something like "build the |
Thanks for playing along on the thought experiment. Yes it seems like a different architecture would be suitable. One machine evaluates a Nix expression to find the set of required derivations, this dependency tree is stored in a common location with the status of each one, and a set of build slaves do "work stealing" to build derivations whose dependencies are done. Right? |
Here is a second iteration of this idea. This is based on a braindump over at lukego/live#22. Goal: Build a set of benchmarks in parallel using cloud resources. The benchmarks are plain Nix derivations but they are expected to be numerous (thousands), to be independent (no build dependencies from one benchmark to another), and to have similar dependencies. Solution:
This way the software dependencies are built locally, the GCE instances boot with a pre-populated Nix store, and the Google infrastructure is responsible for starting and stopping VM instances based on e.g. goal of a 2:1 ratio of pending builds to VMs. The results of the benchmark derivations would need to be scavenged back into our own Nix store e.g. via a Pub/Sub stream of results. This would hopefully execute the full benchmark campaign in O(log n) time and it may be possible to improve this to O(1) depending on the auto-scaling policy. This could be attractive with a basic price of $0.02 per hour for 2-vCPU instances (suitable for single-core benchmarks) and per-second billing increments. Risks:
I also considered the new Hetzner Cloud. This is about half the cost of GCE but does not seem suitable due to coarse per-hour billing increments. The whole idea here is to reduce hours/days/weeks of CPU-time down to minutes of wall-clock time and the billing model would need to support this. Whaddayareckon? |
Here is a third iteration that's intended to break the dependency on Google Compute Engine:
That should work on any cloud provider supported by NixOps and it should be O(1) rather than O(log N) because the number of VMs is not scaled down prematurely as the queue shrinks. |
Thank you for your contributions. This has been automatically marked as stale because it has had no activity for 180 days. If this is still important to you, we ask that you leave a comment below. Your comment can be as simple as "still important to me". This lets people see that at least one person still cares about this. Someone will have to do this at most twice a year if there is no other activity. Here are suggestions that might help resolve this more quickly:
|
Is there a well-known way to build large nix expressions (e.g. whole nixpkgs) on "wide" public cloud compute resources? For example, to run each individual build in a separate Google Compute Engine preemptible VM instance, with thousands of VMs in total. Parallelism would be bounded only by the number of preemptible instances that are available on the cloud at the time and the dependencies between nix derivations.
I am curious about whether solution(s) for this already exist and how heavyweight they are (e.g. uses only basic nix commands, or requiring nixops, or requiring Hydra, etc.)
Generally musing that with preemptible VMs now being billed in one-second increments it may be time to consider a 1:1 scheduler from builds to machines both for simplicity (outsource scheduling to cloud provider) and for performance (more elastic capacity for lower build latency.)
The text was updated successfully, but these errors were encountered: