Support Nixery deployments inside of Kubernetes clusters

[tazjin]

Several open questions around caching, internal addressing etc. remain here - more information coming soon.

[tazjin]

There does not seem to be a clean way of doing this that works across all Kubernetes clusters using something like NodePort.

Some discussions with people revealed that there's also no good definition of what "clean" means in this context.

Here's a very raw list of issues:

• using a Kubernetes-internal Service does not work in most cases because kube-dns and cluster-internal routing is not available to nodes
• NodePort works, but it has a limited port range (30000-31000 by default) and Docker requires TLS certificates on registries - in a simple internal case, one might end up pulling images from a registry named something like localhost:30822 which is not pleasant
• GCP-specific tools that make this cleanly achievable (e.g. internal zones in Cloud DNS) aren't necessarily available elsewhere

I will set up guides and examples for how to do this that focus specifically on GKE. Other users might want to contribute equivalent guides for other Kubernetes hosters.

[tazjin]

Exciting times!

[tazjin]

My personal infrastructure repository (tazjin/depot) now features a Nixery deployment inside of Kubernetes (see here). A similar setup to this should be documented in the Nixery docs for people to experiment with.

The network setup basically involves a private DNS zone for the GCP VPC (in which the cluster pool(s) run) that points nixery.local towards an internal LB which directs traffic to Nixery. There's some room for improvement here (using "real" domains with certificates for instance) still.

[adrian-gierakowski]

Exciting times!

The link seems to be broken. I’d be really interested in learning how to get nixery running on GKE. Thanks!

[blaggacao]

@tazjin Maybe you could give loose input to:

• https://discourse.nixos.org/t/bring-nix-build-to-moby-buildkit/8680 and
• Feature Request: Bring nix builds to buildkit moby/buildkit#1650 ?