Add K8sEndpointGroup #4497
Comments
|
May I work on this issue? |
|
Of course. It's all yours. 😄 |
|
Is it ok that we create a new Gradle module like |
|
|
|
If it does not need third-party dependencies, I'm OK to add |
|
I personally prefer adding a dedicated module for
|
|
I haven't started implementation yet, but it seems that it doesn't need any third-party dependencies and it needs a small number of classes. |
|
I think that there are 2 options.
I understand from the above communication that, the team is not planning to use any third-party dependencies. Before the decision is made, I would kindly recommend to consider the above mentioned SDK too. |
|
Hi @adiseshan! |
|
I think we intended to query kubernetes API and retrieve the list of pods etc., The mentioned library contains many prebuilt classes to take care of authentication, watching a pod etc., Though I have not used the below mentioned packages, |
|
Thanks for the explanation. |
|
I agree that it is not practical to reinvent the wheel. But https://github.com/kubernetes-client/java depends on OkHttp client which would be weird since we already have our own HTTP client. Additionally, as @minwoox said, most Armeria users do not expect to see those transitive dependencies from Armeria modules. If the protocol provided by Kubernetes is straightforward and solid what do you think of implementing the specification without relying on the kubernatetes java client? |
|
I agree. Let us please consider developing with lesser dependencies. If there is too much of orchestration, kindly reconsider. |
Motivation: Fabric8 is one of the popular Kubernetes client implementations. It has an abstract layer for HTTP and WebSocket protocols. If Armeria provides a Kubernetes client to better support the cloud infrastructure, other useful functions such as #4497 will be able to be implemented based on it. Modifications: - Add `kubernetes` module to provide Fabric `StandardHttpClient`. - `ArmeriaHttpClientFactory` is automatically activated via Java SPI. - Both HTTP and WebSocket clients have been implemented in compliance with Reactive Streams specification. - WebSocket is working over HTTP/1. WebSocket over HTTP/2 is disabled for compatibility. - Forked test suites from the upstream repo. - Miscellaneous) - Allow `ProxyConfig` to configure proxy headers. - Allow `WebSocketClient` to configure `HttpHeaders` and `RequestOptions` when starting a WebSocket session. Result: You can use the Fabric Kubernetes client on top of the Armeria client.
Motivation: It is tricky to send requests to a Kubernetes cluster from outside servers without ingress. There is no way to send traffic directly to the pod, but we can send traffic to the port of nodes (NodePort) where the pods are located. This PR proposes a new EndpointGroup that can send requests with CSLB using NodeIP and NodePort to pods in Kubernetes. This way is not an ideal CSLB where servers and clients communicate directly, but it will be a safer way to send traffic without going through ingress which can be SPOF. Modifications: - Add `KubernetesEndpointGroup` on top of `KubernetesClient` to dynamically obtain Kubernetes resources. - [Permission](https://kubernetes.io/docs/reference/access-authn-authz/rbac) to watch `services`, `nodes`, `pods` is required to fetch endpoints. - `service.ports[*].nodePort` is used to create the port of `Endpoint`. - [Watch API](https://kubernetes.io/docs/reference/using-api/api-concepts/#efficient-detection-of-changes) is used to track changes in Kubernetes with a minimal delay. - `ADDED` and `MODIFIED` events are used to update resouces. - `DELETED` is used to remove the resouce. - `BOOKMARK` event is not used and `ERROR` may be ignorable. - Test `KubernetesEndpointGroup` with both a real Kubernetes cluster and a mock Kubernetes server. Result: - You can use `KubernetesEndpointGroup` to perform client-side load-balancing when sending requests. - Fixes #4497 ```java // Create a KubernetesEndpointGroup that fetches the endpoints of the 'my-service' service in the 'default' // namespace. The Kubernetes client will be created with the default configuration in the $HOME/.kube/config. KubernetesClient kubernetesClient = new KubernetesClientBuilder().build(); KubernetesEndpointGroup .builder(kubernetesClient) .namespace("default") .serviceName("my-service") .build(); // If you want to use a custom configuration, you can create a KubernetesEndpointGroup as follows: // The custom configuration would be useful when you want to access Kubernetes from outside the cluster. Config config = new ConfigBuilder() .withMasterUrl("https://my-k8s-master") .withOauthToken("my-token") .build(); KubernetesEndpointGroup .builder(config) .namespace("my-namespace") .serviceName("my-service") .build(); ```
So that the client connects to the pods directly instead of through service proxy.
Of course, we can set up "Headless" Services and use
DnsEndpointGroupfor CSLB.https://kubernetes.io/docs/concepts/services-networking/dns-pod-service/#a-aaaa-records
However, if so, users cannot use the service proxy anymore which means that other components that are not using Armeria have no way to balance the load unless they have the well-made equivalent of
DnsEndpointGroup.Related links:
https://kubernetes.io/docs/reference/kubernetes-api/service-resources/service-v1/#get-read-the-specified-service
https://kubernetes.io/docs/reference/kubernetes-api/workload-resources/pod-v1/#list-list-or-watch-objects-of-kind-pod
https://kubernetes.io/docs/reference/kubernetes-api/workload-resources/pod-v1/#PodStatus
The text was updated successfully, but these errors were encountered: