[#3537] Get K8s container id via K8s API. #3546

calohmn · 2023-09-22T10:32:15Z

This fixes #3537 .

calohmn · 2023-09-22T10:36:10Z

Can be tested with these adaptations to the Hono Helm chart: eclipse/packages#509 (K8s API will only be used if Cgroups v2 is used).

calohmn · 2023-09-22T18:17:14Z

Note that the K8s API request handling is synchronous here. On application startup, the getContainerId method is first invoked from the main thread.

Call stack

org.eclipse.hono.client.command.KubernetesContainerUtil.getContainerId(KubernetesContainerUtil.java:81)
org.eclipse.hono.client.command.CommandRoutingUtil.getNewAdapterInstanceId(CommandRoutingUtil.java:57)
org.eclipse.hono.client.command.ProtocolAdapterCommandConsumerFactoryImpl.(ProtocolAdapterCommandConsumerFactoryImpl.java:77)
org.eclipse.hono.adapter.AbstractProtocolAdapterApplication.commandConsumerFactory(AbstractProtocolAdapterApplication.java:609)
org.eclipse.hono.adapter.AbstractProtocolAdapterApplication.setCollaborators(AbstractProtocolAdapterApplication.java:408)
org.eclipse.hono.adapter.amqp.app.Application.adapter(Application.java:51)
org.eclipse.hono.adapter.amqp.app.Application.adapter(Application.java:26)
io.vertx.core.impl.DeploymentManager.doDeploy(DeploymentManager.java:154)
io.vertx.core.impl.DeploymentManager.deployVerticle(DeploymentManager.java:72)
io.vertx.core.impl.VertxImpl.deployVerticle(VertxImpl.java:718)
io.vertx.core.impl.VertxImpl.deployVerticle(VertxImpl.java:698)
org.eclipse.hono.adapter.AbstractProtocolAdapterApplication.doStart(AbstractProtocolAdapterApplication.java:303)
org.eclipse.hono.service.AbstractServiceApplication.onStart(AbstractServiceApplication.java:186)
org.eclipse.hono.service.AbstractServiceApplication_Observer_onStart_9175e72ff8337562e079de521daf6d4270c94c01.notify(Unknown Source)

When deploying Hono on Minikube on my local machine, I see that the getContainerId method takes 7-8s.

A further optimization could be to defer the getContainerId invocation, letting it be done in a Vertx worker thread in the Future<Void> start() method of ProtocolAdapterCommandConsumerFactoryImpl. The corresponding refactoring doesn't look that straightforward though. I would therefore rather keep the first implementation here in this synchronous manner.
On could also argue that in terms of the time needed for all the Hono components to get ready on Helm chart deployment, the initial time spent in the protocol adapter pods isn't that relevant because the bottle neck is the startup time of Kafka and the device-registry, and the overall load on the K8s components. When restarting an adapter pod at a later point in time, when all other components are ready, the time needed for the getContainerId method was 1.6s on my machine.

EDIT: PR got changed now, determining the id on a vert.x worker thread.

sophokles73 · 2023-09-24T08:19:05Z

A further optimization could be to defer the getContainerId invocation, letting it be done in a Vertx worker thread in the Future start() method of ProtocolAdapterCommandConsumerFactoryImpl.

I would very much prefer this. My concern is that if you don't refactor this now, it will never get refactored accordingly ...

calohmn · 2023-09-24T19:30:10Z

Ok, I'll have a closer look here.
Btw, since Quarkus 3.0, the quarkus-kubernetes-client dependency uses the fabric8 kubernetes-client version >= 6.4, which includes support for using the vert.x HTTP client under the hood instead of the OkHttp client. quarkus-kubernetes-client lets this vert.x HTTP client be used by default (see quarkusio/quarkus#30480).
This doesn't mean there is support for getting a Future<Pod> result from the fabric8 client (see this discussion thread), but nonetheless it seems to be preferable letting these HTTP requests be managed via the same Vert.x instance that is already used in Hono, also getting rid of the extra OkHttp dependency.
Have there been any attempts already to use Quarkus 3.x within Hono?

sophokles73 · 2023-09-25T05:47:52Z

Have there been any attempts already to use Quarkus 3.x within Hono?

Not yet. I have created #3547

calohmn · 2023-09-28T20:36:56Z

@sophokles73 I've updated the PR, letting the container id be determined on a vert.x worker thread now.

sophokles73 · 2023-10-04T08:46:05Z

...s/command/src/main/java/org/eclipse/hono/client/command/KubernetesContainerInfoProvider.java

+/**
+ * Provides information about the Kubernetes container if this application is running in Kubernetes.
+ */
+public class KubernetesContainerInfoProvider {


That would mean it can't be mocked anymore (in ProtocolAdapterCommandConsumerFactoryTest).

sophokles73 · 2023-10-04T08:50:16Z

...s/command/src/main/java/org/eclipse/hono/client/command/KubernetesContainerInfoProvider.java

+            return Future.succeededFuture(containerId);
+        }
+        final Promise<String> containerIdPromise = Promise.promise();
+        if (!containerIdPromiseRef.compareAndSet(null, containerIdPromise)) {


if this is such an expensive operation and you anticipate this operation to be invoked concurrently/multiple times, wouldn't it make sense to cache the result?

The result is already cached via the containerId field value.

sophokles73 · 2023-10-04T08:51:40Z

...s/command/src/main/java/org/eclipse/hono/client/command/KubernetesContainerInfoProvider.java

+        if (containerId != null) {
+            return Future.succeededFuture(containerId);
+        }
+        final Context currentContext = Vertx.currentContext();


How about passing a Context into the method instead of relying on the static method to obtain the context? That way, it would also be much easier to test this method, right?

Or maybe it would make even more sense to pass a Vertx instance to the getInstance method?

I've now added a context parameter to the getContainerId() method.

sophokles73 · 2023-10-07T11:06:48Z

@calohmn IMHO we also should backport this to 2.4.x and 2.3.x

calohmn · 2023-10-08T09:03:52Z

IMHO we also should backport this to 2.4.x and 2.3.x

Yes, we can do that.

sophokles73 · 2023-10-08T10:27:05Z

...s/command/src/main/java/org/eclipse/hono/client/command/KubernetesContainerInfoProvider.java

+            return Future.succeededFuture(containerId);
+        }
+        final Promise<String> containerIdPromise = Promise.promise();
+        if (!containerIdPromiseRef.compareAndSet(null, containerIdPromise)) {


sophokles73 · 2023-10-08T10:30:05Z

...src/main/java/org/eclipse/hono/client/command/ProtocolAdapterCommandConsumerFactoryImpl.java

+     * @param context The context.
+     * @throws NullPointerException if context is {@code null}.
+     */
+    void setContext(final Context context) {


is there a particular reason why you do not want to pass in a Vertx instance to the constructor as we do in most other services?

Right, that is the better solution. I've changed it now.

sophokles73

LGTM

calohmn added the C&C Command and Control label Sep 22, 2023

calohmn added this to In progress in 2.5.0 via automation Sep 22, 2023

calohmn mentioned this pull request Sep 22, 2023

[#512] Add service account for Hono adapter pods; update tracing image versions. eclipse/packages#509

Merged

calohmn force-pushed the PR/container_id_via_k8s_api branch from 5274852 to ad5b26b Compare September 28, 2023 20:32

sophokles73 reviewed Oct 4, 2023

View reviewed changes

sophokles73 added this to the 2.5.0 milestone Oct 4, 2023

sophokles73 added this to In progress in 2.4.1 via automation Oct 7, 2023

sophokles73 added this to In progress in 2.3.2 via automation Oct 7, 2023

sophokles73 removed this from In progress in 2.3.2 Oct 7, 2023

sophokles73 removed this from In progress in 2.4.1 Oct 7, 2023

calohmn force-pushed the PR/container_id_via_k8s_api branch from ad5b26b to 191fda0 Compare October 8, 2023 09:00

sophokles73 reviewed Oct 8, 2023

View reviewed changes

[eclipse-hono#3537] Get K8s container id via K8s API.

4d2bcaf

calohmn force-pushed the PR/container_id_via_k8s_api branch from 191fda0 to 4d2bcaf Compare October 9, 2023 05:34

calohmn requested a review from dejanb as a code owner October 9, 2023 05:34

sophokles73 approved these changes Oct 9, 2023

View reviewed changes

calohmn merged commit dc13f1b into eclipse-hono:master Oct 9, 2023
7 checks passed

2.5.0 automation moved this from In progress to Done Oct 9, 2023

calohmn deleted the PR/container_id_via_k8s_api branch October 9, 2023 17:59

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[#3537] Get K8s container id via K8s API. #3546

[#3537] Get K8s container id via K8s API. #3546

calohmn commented Sep 22, 2023

calohmn commented Sep 22, 2023 •

edited

calohmn commented Sep 22, 2023 •

edited

sophokles73 commented Sep 24, 2023

calohmn commented Sep 24, 2023

sophokles73 commented Sep 25, 2023

calohmn commented Sep 28, 2023

sophokles73 Oct 4, 2023

calohmn Oct 8, 2023

sophokles73 Oct 4, 2023

calohmn Oct 8, 2023

sophokles73 Oct 8, 2023

sophokles73 Oct 4, 2023

calohmn Oct 8, 2023

sophokles73 commented Oct 7, 2023

calohmn commented Oct 8, 2023

sophokles73 Oct 8, 2023

sophokles73 Oct 8, 2023

calohmn Oct 9, 2023

sophokles73 left a comment

[#3537] Get K8s container id via K8s API. #3546

[#3537] Get K8s container id via K8s API. #3546

Conversation

calohmn commented Sep 22, 2023

calohmn commented Sep 22, 2023 • edited

calohmn commented Sep 22, 2023 • edited

sophokles73 commented Sep 24, 2023

calohmn commented Sep 24, 2023

sophokles73 commented Sep 25, 2023

calohmn commented Sep 28, 2023

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

sophokles73 commented Oct 7, 2023

calohmn commented Oct 8, 2023

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

sophokles73 left a comment

Choose a reason for hiding this comment

calohmn commented Sep 22, 2023 •

edited

calohmn commented Sep 22, 2023 •

edited