Headless k8 service instead of relying on K8 APIs

[seivan]

One of the ways we solved this was creating a headless Service for all the Deployments or Daemonset that should join the cluster.
That way when the application launches using the <app@ip> it can do a lookup agains the service name and get the addresses for the rest of the cluster to join.

I am not sure, but my guess is (and correct me if I am wrong) using the Kubernetes API might break requirements for some people who don't want to expose those things on the pod/containers.

[bitwalker]

So far I haven't had any pushback on it, since permissions are pretty fine-grained, exposing the ability to list endpoints is pretty low-risk. That said, if there is an alternative method that I can either make part of the documentation, or support in the k8s strategy itself, I'm certainly open to that!

[seivan]

Yeah I am not so concerned over security more than I am over relying on an API that is often subject to change compared to a regular nslookup .

---

apiVersion: v1
kind: Service
metadata:
  creationTimestamp: null
  labels:
    name: my-backend
  name: my-backend-list
  namespace: default
spec:
  clusterIP: None 
  ports:
    - port: 80
  selector:
    name: my-backend

This would let you do a lookup via :inet_res.getbyname('my-backend-list', :a) and get all the nodes.
I would guess This is absolutely less code than using the Kubernetes API directly.
The less code (e.g working with the K8 API) the better, when you can rely on the internal DNS.

If you dislike the shortname, you can also use the fully qualified name by
combining

value = :inet_db.get_searchlist  |> extract_domain
:inet_res.lookup(name <> value, :in, :a)

Another benefit is, you can let Kubernetes do the selector matching and just use the single name it offers from the service you create e.g my-backend-list

If I can attach my own Strategies to try it out, I'd do a PR..
I don't mind adding in this myself, but I'd want to make sure you're ok with it.

[bitwalker]

I'm a fan, if you want to open a PR, I'll merge it once we've worked through review. My only issue is that I'm not sure we should remove the old strategy just yet, instead I'm thinking we make this an alternative k8s strategy and call it Cluster.Strategy.Kubernetes.Service, or whatever you think makes the most sense as a name, the point being that it's namespaced under Kubernetes. I'll have to sort out a deprecation plan for the old one.

[seivan]

Yup, that's what I'd like. Keep the previous one, mostly because I am not sure if this is something that works everywhere (e.g AWS China, or AWS Gov) or with particular installation or version of K8 for instance. I've only tested >= 1.8.0 and on AWS though I have a strong suspicion it's the same on most vendors, I haven't confirmed it yet.

Am I able to inject the strategy behaviours/interfaces within my own project, or should I make a local clone of libcluster and set the new strategy there?

Also how about Cluster.Strategy.Kubernetes.DNS?
I do want to point out that there seems to be a pattern in K8 to cluster nodes under a "headless" service, and this is what I'm currently doing (check the spec I pasted above), however
it isn't really related to Kubernetes itself. It just assumes it can find other nodes through a lookup. I've only tested with K8 but I think the same applies to anything that uses DNS for service discover, perhaps even Docker Swarm.

[bitwalker]

That name works for me! You can create the strategy in your own codebase, you just change the configuration to reference the module of the strategy to use; once it's ready, you can just copy it into libcluster.

[seivan]

@bitwalker 👍 🏩

[seivan]

Alright, we're game: #36
It works, no hiccups and nothing. Definitely needs a gentle touch on naming and docs.

[bitwalker]

Thanks! I'm ready to merge the PR, just need to add a mention to the README, and we're good to go. I'll close this for now, since merging that will resolve this.

[seivan]

@bitwalker I just want to point out I'm not comfortable with the quality and more specifically the naming of things, I'd love some feedback on that part.
I can update the README during the weekend 🌮