Akrobateo registry (https://registry.pharos.sh) is out of service making it non-operable

[tmeltser]

Hi,
I'm trying to operate Akrobateo as follows:

• Cloned the repo

• Moved to the "deploy" folder

• Run the following command: "kubectl apply -k ."

According to the error I see, it seems the private Docker registry hosting the image for the Akrobateo Pod is out of service as I get the following error: Get https://registry.pharos.sh/v2/: dial tcp 198.54.117.199:443: connect: connection refused"

Any advice would be very much appreciated.

[tmeltser]

Well, I was able to change deploy/04_operator.yaml by replacing "image: registry.pharos.sh/kontenapharos/akrobateo:latest" with "image: kontenapharos/akrobateo:latest"

This enabled the Akrobateo to run.

But when adding a service of type LoadBalancer I fell back into the original problem (and I don't see how to control it from the deployment files) happening now on the newly created Akrobateo DS' pods::
$ kubectl get pods -n demo4echo
NAME READY STATUS RESTARTS AGE
akrobateo-echofe-service-9hkfn 0/1 Init:ImagePullBackOff 0 8m44s
akrobateo-echofe-service-qfkcq 0/1 Init:ImagePullBackOff 0 8m44s
akrobateo-echofe-service-vf59r 0/1 Init:ImagePullBackOff 0 8m44s
echobe-deployment-78967cf876-8vbmz 1/1 Running 0 41m
echofe-deployment-84874bcdd8-vr95p 1/1 Running 0 26m

And the specific problem (for all akrobateo lb pods):
"Warning Failed 8m22s (x3 over 9m1s) kubelet, k8s-worker-1 Failed to pull image "registry.pharos.sh/kontenapharos/akrobateo-lb:latest": rpc error: code = Unknown desc = Error response from daemon: Get https://registry.pharos.sh/v2/: dial tcp 198.54.117.198:443: connect: connection refused"

Is anyone able to operate current version of Akrobateo?

[toonsevrin]

Seems like this project is stale. I gave the source code a quick peek and setting the controller's LB_IMAGE environment variable to your akrobateo-lb image will solve this issue.

The Dockerfile for the akrobateo-lb can be found here.

[tmeltser]

Hi @toonsevrin - just saw your message.
First - thanks for taking the time to comment
Second - I'll try to give it a try, although not sure I should use a stale project (although it looks very promising)

In case I progress in some way - I'll post back a comment to close the loop.

Thanks again.

[toonsevrin]

Yeah, I've also stopped using this project because of it being stale, within our project, we've just moved our ingress deployment to a daemonset and made all the ports hostPorts. Works like magic.

[tmeltser]

Well - we thought about using this tool with our TCP based services (actually SIP, so Ingress isn't suitable here) alongside with the ExternalDNS add-on to dynamically publish the designated IPs into a DNS server - it seems we'll need to find another alternative (this project should have done the job quite nicely :-()

[toonsevrin]

How about spending a few $ on an actual load balancer? If you're on-prem you can also consider metallb. Otherwise, I still think running externalDNS + a daemonset (you could put your external facing app in the set) can work out.

[tmeltser]

Actually the metallb was our previous solution but it turns out it collides with our CNI -> Calico.
So, while looking at what Calico can offer, this solution would have seen very nice (I'll try to give your suggestion a try tomorrow and see how it goes (and report back)).
As for using a daemonset, the problem is that it dictates the number of replicas according to the number of workers, and that isn't always suitable (might be too many or not enough pods)

[toonsevrin]

That is very true indeed, as you are using hostPort, you are constraint to 1 ingress per machine.

If this is a concern, you can have an autoscaling nodepool and deploy the daemonset only on those nodes. This is pretty similar to having a second proxy cluster that forwards to your nodePort if you would have a nodeport service.

But I think if these are your concerns, I would consider finding a way to make metallb not collide with calico, that sounds like the cleaner solution at your scale.

[tmeltser]

What you are saying is true, but I wanted to give the Akrobateo another try. Alas, I got stuck with a same problem like last time, which is fully described in: ip_forward is not enabled #31. Despite of what is written there, I have K8S with version 1.16.1 (which is higher than the 15.5 mentioned there) and I'm getting this error...

Any thought on how to bypass this continuing obstacle?

[toonsevrin]

No idea, but note that behind the scenes, this project takes the same Daemonset approach as you said was not applicable.

[tmeltser]

I don't mind the Akrobateo works as a Daemonset, I don't want to deploy my micro-service(s) as Daemonset from the reasons mentioned above.
I will try to crack it anyway, and if succeeded, will come back and update.

[toonsevrin]

oooh of course, yeah you just need your ingress to be a daemonset, within my company, we simply patch our istio ingress-gateway deployment to a daemonset (with hostPorts) and it works like gold. All our services still work and route as they would by default.

In case you don't have an ingress, you can setup a proxy pretty easily yourself, all though using a kubernetes framework is a lot nicer: We use istio gateways and virtualservices to route all our traffic, makes it ridiculously easy to add new endpoints and services and routes.

[pabloa]

What about:

https://hub.docker.com/r/kontenapharos/akrobateo

[tmeltser]

Thanks, but I've already switched to these images - currently I'm blocked by Akrobateo claiming IP Forwarding is disabled on our Cluster and thus it is unable to run (couldn't solve this yet)

[toonsevrin]

I got blocked by some weird Akrobateo behavior as well. Go back to DaemonSets. All I had to do for this was apply a few small patches to my ingress deployment, literally 8 lines.

The idea of a LoadBalancer operator to do this automatically for you is really awesome, but sadly enough it doesn't seem to exist outside of this project, the solution above is your best bet. Good luck!

[tmeltser]

I'm not sure your suggestion will work for me. as in my case I'm externalizing a service dealing with UDP based SIP (on port 5060), and not http/https.

[toonsevrin]

Well as long as you can setup a proxy for that your golden! Note that this is also how Akrobateo works :)

[tmeltser]

This is what I'm currently doing as an alternative to Akrobateo.

[jawabuu]

Hey @toonsevrin I'm also looking to achieve HA with embedded DB in k3s.
In my setup, I use Traefik V2 as a daemonset with hostports.
MetalLB is used to populate LoadBalancer IPs for use with services like ExternalDNS.
I need a mechanism to update the LoadBalancer IP if a node goes down. Do you have any ideas?

[toonsevrin]

Well this is an easy problem!

Just add a healthcheck endpoint to your daemonset service (eg. /healthz) 🎉

If you have a DNS with health checks (and you don't mind manually adding new ips), you are done here. If either of these is not the case, you would want to write a small controller that syncs you dns (using your DNS API).

This is considered very easy! Note that DNS updates take a while to propagate. I would not consider this truely HA until you use floating IPs (perhaps you are doing this, I'm not sure what you meant with the metallb part).

[jawabuu]

Hey @toonsevrin Glad to know that this is considered easy 😄
For my use case, I am thinking of using external-dns. External DNS responds to changes in LoadBalancer IP and updates A records.
MetalLB or Akrobateo is used to populate the LoadBalncerIP for my Traefik Service.
All I want is the LoadBalancer IP to have the value of the current cluster leader. So suppose I have;

k3s-master-1 192.168.0.1
k3s-master-2 192.168.0.2
k3s-master-3 192.168.0.3

The initial LoadBalancer IP will be 192.168.0.1 . If k3s-master-1 goes down, and the leader changes to k3s-master-2 I would like 192.168.0.2 to be the new LoadBalancer IP.
Do you have any idea on how I can prefer a pod be scheduled in the current cluster leader?

[tmeltser]

Hi @jawabuu,
From my experience with Metallb/Akrobateo (and ExternalDns)) the IPs at question are not the ones of the masters nodes. In Metallb they are from a pre-defined pool (not related to the masters/workers IPs), and in Akrobateo they are the IPs of the workers.
So not sure why the IPs of the masters (leader or not) is relevant here.

[jawabuu]

Hey @tmeltser The IPs of the master is important because this is where the ingress(in this case Traefik) is deployed.
If we can have that the LoadBalancer IP is always the IP of the master(leader) node then we can have a (quasi) HA setup for cases where one cannot provision an external LoadBalancer.

[tmeltser]

Hey @jawabuu - we are using the Metallb for the same purpose, but we don't deploy anything on the master nodes (Ingress controller or anything else), but only on our worker nodes, so we don't need (nor do we want) to expose any of our master(s) IP(s); but it's your choice of course.

[toonsevrin]

Hi @jawabuu,
From my experience with Metallb/Akrobateo (and ExternalDns)) the IPs at question are not the ones of the masters nodes. In Metallb they are from a pre-defined pool (not related to the masters/workers IPs), and in Akrobateo they are the IPs of the workers.
So not sure why the IPs of the masters (leader or not) is relevant here.

I didn't know about externalDns, seems like that does exactly what I was proposing (pretty sweet, I'll definitely use it in the future) 😅 I'll leave it to the other people to help you as I don't have any experience with metallb. Good luck!

[jawabuu]

Thanks @toonsevrin One question about akrobateo. How can I get it to preserve source IP?

[toonsevrin]

@jawabuu If you're able to add it as metadata (eg. header), you'll be the happiest. That way you don't have any networking challenges. hostPort Daemonsets (eg. akrobateo) should receive packets with the hostIP. I'm not certain what proxy akrobateo uses but you should check if it adds the SOURCE IP to some metadata field (or if that is configurable).

But as with almost all reverse proxies, akrobateo will override the IP packet source. If you want your service to receive the raw packets, use your service as a hostPorted DaemonSet directly.

Anyways, getting your source IP in the metadata is usually your best bet if you need the IP for something.

P.S: I'm quite bad at network related stuff so please correct me if I'm wrong.

[jawabuu]

No worries. I was actually thinking maybe this is why we can't retrieve the hostIP when using metalLB with an ingress like Traefik.
The MASQUERADE bit specifically.

akrobateo/lb-image/entrypoint.sh

Lines 12 to 16 in f858444

	echo "Setting up forwarding from port $SRC_PORT to $DEST_IP:$DEST_PORT/$DEST_PROTO"
	# Setup the actual forwarding
	sudo iptables -t nat -I PREROUTING ! -s ${DEST_IP}/32 -p ${DEST_PROTO} --dport ${SRC_PORT} -j DNAT --to ${DEST_IP}:${DEST_PORT}
	sudo iptables -t nat -I POSTROUTING -d ${DEST_IP}/32 -p ${DEST_PROTO} -j MASQUERADE

[toonsevrin]

For everyones interest, I've noticed inlets, a project that seems similar to akrobateo, but is maintained.

[jawabuu]

Hey @toonsevrin
I was finally able to achieve this. I retrieve the Leader Node IP as follows

LEADER_NODE=`kubectl get endpoints kube-scheduler -n kube-system -o jsonpath='{.metadata.annotations.control-plane\.alpha\.kubernetes\.io/leader}' | jq -r .holderIdentity | sed "s/_[^_]*$//"`
LEADER_IP=`kubectl get nodes $LEADER_NODE -o jsonpath={.status.addresses[?\(@.type==\"ExternalIP\"\)].address}`

I can then apply it to my service or metallb and have external DNS update the records.
This done from a deployment in my cluster.

[jocutajar]

Would be nice if someone picked this up. There are some forks, https://github.com/alexfouche/akrobateo looking the most updated. Or is klipper now somehow available to k8s? https://github.com/k3s-io/klipper-lb