Initial Observability extension - devservices, devresources, LGTM #38448

alesj · 2024-01-29T10:21:43Z

This is a simpler version of the previous (see below) PR.
This one just includes LGTM (https://github.com/grafana/docker-otel-lgtm) support with the new devresource mechanism.
The stuff mentioned below will be added in the next PR(s).

Fixes #26445

This adds dynamic lookup of possible observability dev resources / services.
It also adds REST clients for PromQL (Prometheus query language) and VictoriaMetrics REST api support; push, etc

The dev resources / services can be run in 3 different ways

re-usable dev services (default)
simple start/stop dev services
single / plain per QuarkusTestResource dev resource

(1) can be configured
(2) depends what's available on the classpath
(3) needs to be explicitly used in QuarkusTestResource

Currently supported dev services

Grafana
Jaeger
OTel
VictoriaMetrics DB
VictoriaMetrics Agent / scraper

It should be easy to add new / other observability dev services in the future,
e.g. currently missing Prometheus, etc
by adding a new DevResourceLifecycleManager implementation.

Current TODO list

test with 2 app running at the same time, propagating tracing to same dev service(s)
Grafana-all-in-one Docker image support
Prometheus dev service
reload and other (combo) tests

alesj · 2024-01-29T10:23:47Z

Just remembered I can add back the devresource vs devservice (autodiscovery) tests -- but need to adjust them to LGTM usage.

alesj · 2024-01-29T10:25:03Z

Documentation is WIP -- first lets discuss if the concept is suitable enough to handle the things we want with observability and then hopefully beyond.
e.g. the issue @cescoffier just mention previous week, with multi-devservices

github-actions · 2024-01-29T10:56:41Z

🙈 The PR is closed and the preview is expired.

integration-tests/observability-lgtm/src/main/resources/application.properties

extensions/observability-devservices/testlibs/devresource/pom.xml

extensions/observability-devservices/sub/lgtm/pom.xml

extensions/observability-devservices/common/pom.xml

integration-tests/observability-lgtm/pom.xml

cescoffier

.idea files should be removed from git.

cescoffier · 2024-01-29T13:52:27Z

extensions/devservices/common/src/main/java/io/quarkus/devservices/common/ContainerLocator.java

+    /**
+     * @return container id, if exists
+     */
+    public Optional<String> locateContainer(String serviceName, boolean shared, LaunchMode launchMode,


what about "tests"?

Tests for an existing container usage?
Do we have some already?

I guess I need to force the redeploy / restart of the test somehow,
and see if the container is still up-n-running?

cescoffier · 2024-01-29T13:53:12Z

...ity-devservices/common/src/main/java/io/quarkus/observability/common/ContainerConstants.java

+
+    // Images
+
+    public static final String LGTM = "grafana/otel-lgtm:0.2.0";


prefix it with docker.io

cescoffier · 2024-01-29T13:54:41Z

...ity-devservices/common/src/main/java/io/quarkus/observability/common/config/ConfigUtils.java

+public class ConfigUtils {
+
+    public static boolean isEnabled(ContainerConfig config) {
+        if (config != null && config.enabled()) {


do you want to check the Quarkus runtime classloader or build classloader?

I actually forgot to delete this class, as it's not used anymore.

cescoffier · 2024-01-29T13:59:10Z

...ces/deployment/src/main/java/io/quarkus/observability/deployment/ObservabilityProcessor.java

+
+@BuildSteps(onlyIfNot = IsNormal.class, onlyIf = { GlobalDevServicesConfig.Enabled.class,
+        ObservabilityProcessor.IsEnabled.class })
+class ObservabilityProcessor {


Maybe you should add DevService in the name?

cescoffier · 2024-01-29T13:59:33Z

...ces/deployment/src/main/java/io/quarkus/observability/deployment/ObservabilityProcessor.java

+
+    @BuildStep
+    FeatureBuildItem feature() {
+        return new FeatureBuildItem(Feature.OBSERVABILITY);


Do we need this feature?

I guess it doesn't hurt, if you want to reference it in some other extension?
Or what's the use case then?
(I saw that all of the extension processors had this ...)

alesj · 2024-01-29T14:09:32Z

.idea files should be removed from git.

Yeah, no idea how they ended up in in the first place.
Or how they were deleted ...

brunobat · 2024-01-30T13:48:34Z

@alesj Can you please add a guide with documentation on how to use the extension?
A new telemetry-observability-devservices.adoc filename could work. The telemetry prefix is a convention for OTel, Micrometer related guides that we started to use recently.

alesj · 2024-01-30T13:52:09Z

@alesj Can you please add a guide with documentation on how to use the extension? A new telemetry-observability-devservices.adoc filename could work. The telemetry prefix is a convention for OTel, Micrometer related guides that we started to use recently.

Well, it won't be just telemetry, as this guide will be a generic one, with explicit examples for all sorts of tech - that we add later on (from the old PR).
So I think we need a better name - what about just observability-devservices?

Grafana OTel LGTM additional Documentation Fix enforcer configuration and make names consistent Update descriptors with latest dependency state Abstract Testcontainers usage, own Container SPI.

quarkus-bot · 2024-04-19T07:15:42Z

Status for workflow `Quarkus Documentation CI`

This is the status report for running Quarkus Documentation CI on commit 796348d.

✅ The latest workflow run for the pull request has completed successfully.

It should be safe to merge provided you have a look at the other checks in the summary.

⚠️ There are other workflow runs running, you probably need to wait for their status before merging.

quarkus-bot · 2024-04-19T11:26:48Z

Status for workflow `Quarkus CI`

This is the status report for running Quarkus CI on commit 796348d.

Failing Jobs

Status	Name	Step	Failures	Logs	Raw logs	Build scan
✖	MicroProfile TCKs Tests	`Verify`	Failures	Logs	Raw logs	🔍

Full information is available in the Build summary check run.
You can consult the Develocity build scans.

Failures

⚙️ MicroProfile TCKs Tests #

- Failing: tcks/microprofile-opentelemetry

📦 tcks/microprofile-opentelemetry

✖ org.eclipse.microprofile.telemetry.tracing.tck.async.JaxRsClientAsyncTest.testIntegrationWithJaxRsClientAsync - History - More details - Source on GitHub

java.util.concurrent.RejectedExecutionException: event executor terminated
	at io.netty.util.concurrent.SingleThreadEventExecutor.reject(SingleThreadEventExecutor.java:934)
	at io.netty.util.concurrent.SingleThreadEventExecutor.offerTask(SingleThreadEventExecutor.java:351)
	at io.netty.util.concurrent.SingleThreadEventExecutor.addTask(SingleThreadEventExecutor.java:344)
	at io.netty.util.concurrent.SingleThreadEventExecutor.execute(SingleThreadEventExecutor.java:836)
	at io.netty.util.concurrent.SingleThreadEventExecutor.execute0(SingleThreadEventExecutor.java:827)
	at io.netty.util.concurrent.SingleThreadEventExecutor.execute(SingleThreadEventExecutor.java:817)
	at io.vertx.core.impl.EventLoopExecutor.execute(EventLoopExecutor.java:35)

brunobat · 2024-04-19T16:03:08Z

Thanks very much @alesj for the effort!

alesj requested review from brunobat and ozangunalp January 29, 2024 10:22

quarkus-bot bot added area/core area/dependencies Pull requests that update a dependency file area/devtools Issues/PR related to maven, gradle, platform and cli tooling/plugins area/documentation labels Jan 29, 2024

brunobat reviewed Jan 29, 2024

View reviewed changes

integration-tests/observability-lgtm/src/main/resources/application.properties Outdated Show resolved Hide resolved

brunobat reviewed Jan 29, 2024

View reviewed changes

extensions/observability-devservices/testlibs/devresource/pom.xml Outdated Show resolved Hide resolved

brunobat reviewed Jan 29, 2024

View reviewed changes

extensions/observability-devservices/sub/lgtm/pom.xml Outdated Show resolved Hide resolved

brunobat reviewed Jan 29, 2024

View reviewed changes

extensions/observability-devservices/common/pom.xml Outdated Show resolved Hide resolved

alesj force-pushed the obs_lgtm branch from 3a2686f to f7cac61 Compare January 29, 2024 11:58

brunobat reviewed Jan 29, 2024

View reviewed changes

integration-tests/observability-lgtm/pom.xml Show resolved Hide resolved

alesj force-pushed the obs_lgtm branch from f7cac61 to b99b4df Compare January 29, 2024 12:41

cescoffier reviewed Jan 29, 2024

View reviewed changes

cescoffier requested changes Jan 29, 2024

View reviewed changes

alesj force-pushed the obs_lgtm branch 3 times, most recently from 002fbba to 6f43cf8 Compare January 29, 2024 16:11