Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support Nixery deployments inside of Kubernetes clusters #4

Open
tazjin opened this issue Jul 29, 2019 · 3 comments
Assignees

Comments

@tazjin
Copy link
Member

@tazjin tazjin commented Jul 29, 2019

Several open questions around caching, internal addressing etc. remain here - more information coming soon.

@tazjin

This comment has been minimized.

Copy link
Member Author

@tazjin tazjin commented Aug 2, 2019

There does not seem to be a clean way of doing this that works across all Kubernetes clusters using something like NodePort.

Some discussions with people revealed that there's also no good definition of what "clean" means in this context.

Here's a very raw list of issues:

  • using a Kubernetes-internal Service does not work in most cases because kube-dns and cluster-internal routing is not available to nodes
  • NodePort works, but it has a limited port range (30000-31000 by default) and Docker requires TLS certificates on registries - in a simple internal case, one might end up pulling images from a registry named something like localhost:30822 which is not pleasant
  • GCP-specific tools that make this cleanly achievable (e.g. internal zones in Cloud DNS) aren't necessarily available elsewhere

I will set up guides and examples for how to do this that focus specifically on GKE. Other users might want to contribute equivalent guides for other Kubernetes hosters.

@tazjin

This comment has been minimized.

Copy link
Member Author

@tazjin tazjin commented Aug 2, 2019

Exciting times!

Nixery in a GKE cluster

@tazjin tazjin self-assigned this Aug 2, 2019
@tazjin

This comment has been minimized.

Copy link
Member Author

@tazjin tazjin commented Sep 4, 2019

My personal infrastructure repository (tazjin/depot) now features a Nixery deployment inside of Kubernetes (see here). A similar setup to this should be documented in the Nixery docs for people to experiment with.

The network setup basically involves a private DNS zone for the GCP VPC (in which the cluster pool(s) run) that points nixery.local towards an internal LB which directs traffic to Nixery. There's some room for improvement here (using "real" domains with certificates for instance) still.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
1 participant
You can’t perform that action at this time.