Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support Nixery deployments inside of Kubernetes clusters #4

Open
tazjin opened this issue Jul 29, 2019 · 5 comments
Open

Support Nixery deployments inside of Kubernetes clusters #4

tazjin opened this issue Jul 29, 2019 · 5 comments

Comments

@tazjin
Copy link
Owner

@tazjin tazjin commented Jul 29, 2019

Several open questions around caching, internal addressing etc. remain here - more information coming soon.

@tazjin
Copy link
Owner Author

@tazjin tazjin commented Aug 2, 2019

There does not seem to be a clean way of doing this that works across all Kubernetes clusters using something like NodePort.

Some discussions with people revealed that there's also no good definition of what "clean" means in this context.

Here's a very raw list of issues:

  • using a Kubernetes-internal Service does not work in most cases because kube-dns and cluster-internal routing is not available to nodes
  • NodePort works, but it has a limited port range (30000-31000 by default) and Docker requires TLS certificates on registries - in a simple internal case, one might end up pulling images from a registry named something like localhost:30822 which is not pleasant
  • GCP-specific tools that make this cleanly achievable (e.g. internal zones in Cloud DNS) aren't necessarily available elsewhere

I will set up guides and examples for how to do this that focus specifically on GKE. Other users might want to contribute equivalent guides for other Kubernetes hosters.

@tazjin
Copy link
Owner Author

@tazjin tazjin commented Aug 2, 2019

Exciting times!

Nixery in a GKE cluster

@tazjin tazjin self-assigned this Aug 2, 2019
@tazjin
Copy link
Owner Author

@tazjin tazjin commented Sep 4, 2019

My personal infrastructure repository (tazjin/depot) now features a Nixery deployment inside of Kubernetes (see here). A similar setup to this should be documented in the Nixery docs for people to experiment with.

The network setup basically involves a private DNS zone for the GCP VPC (in which the cluster pool(s) run) that points nixery.local towards an internal LB which directs traffic to Nixery. There's some room for improvement here (using "real" domains with certificates for instance) still.

@adrian-gierakowski
Copy link

@adrian-gierakowski adrian-gierakowski commented Jul 28, 2020

Exciting times!

Nixery in a GKE cluster

The link seems to be broken. I’d be really interested in learning how to get nixery running on GKE. Thanks!

@blaggacao
Copy link

@blaggacao blaggacao commented Aug 27, 2020

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

None yet
3 participants