Support for native coordination on Kubernetes

[malthe]

Search before asking

• I searched in the issues and found nothing similar.

Motivation

Pulsar now has a pluggable interface for coordination and metadata services, see #572 which was resolved through PIP-45.

In Apache NiFi, they've done something similar but thus far targeting the services offered already by Kubernetes, namely the Lease API and ConfigMaps:

https://exceptionfactory.com/posts/2024/08/10/bringing-kubernetes-clustering-to-apache-nifi/

Being internally based currently on etcd, this should perform similarly.

The motivation presented at the Pulsar Summit in 2022 applies even more so here:

Small clusters → remove overhead

• Less components to deploy
• Easier operations

Solution

Include a coordination and metadata backend that uses native Kubernetes services.

Alternatives

In the past, people have written proxies that surface for example the ZooKeeper API on top of etcd, see zetcd. It could be argued that an entirely separated service should be written that standardizes the use of Kubernetes services for leader election and metadata needs.

Anything else?

No response

Are you willing to submit a PR?

• I'm willing to submit a PR!

[lhotari]

There's etcd support already in Pulsar and BookKeeper. However there's currently an open issue #23513 which I'll come back to in the next few weeks unless someone addresses before me.
There might not be a lot of end user documentation for using other than Zookeeper at the moment, although there's also other alternatives (etcd, Oxia) available in Pulsar which also get used for BookKeeper metadata. /cc @Apurva007

[lhotari]

However, I don't think that there's a recommendation to use Kubernetes internal etcd for Pulsar & BookKeeper metadata although it could be technically possible.

[liangyuanpeng]

However, I don't think that there's a recommendation to use Kubernetes internal etcd for Pulsar & BookKeeper metadata although it could be technically possible.

I think it's talking about to using kubernetes API and not etcd.

[malthe]

It's using the Kubernetes API so in principle this could be etcd underneath or something else.

[liangyuanpeng]

and i think using the kubernetes API for metadata backend is a not good idea, This is frequently written data.