traceId, spanId not set in logs when using quartz and opentelemetry #28552

syr · 2022-10-12T22:36:08Z

Describe the bug

I use quarkus-quartz and quarkus-opentelemetry

Have definded a custom log format like in https://quarkus.io/guides/opentelemetry#create-the-configuration

quarkus.log.console.format=%d{HH:mm:ss} %-5p traceId=%X{traceId}, parentId=%X{parentId}, spanId=%X{spanId}, sampled=%X{sampled} [%c{2.}] (%t) %s%e%n

Expected behavior

Expected that traceId etc. are filled, (this only works for rest resource), see:

  00:16:43 INFO  traceId=7d79594584b49a7db88f47c75a0cd185, parentId=, spanId=cb9d17786f3ed0c0, sampled=true [or.ac.qu.TaskResource] (executor-thread-0) log test

Actual behavior

But for log output in quartz scheduled method everything is empty

00:16:45 INFO  traceId=, parentId=, spanId=, sampled= [or.ac.qu.TaskBean] (QuarkusQuartzScheduler_Worker-3) log test

How to Reproduce?

https://github.com/syr/quartz-otel-logs is based on latest quartz-quickstart code.

Only needs to be started, scheduler runs every 10 seconds printing log lines without traceId etc.

Calling http://localhost:8080/tasks shows that it works generally, but only for rest resources

Output of `uname -a` or `ver`

No response

Output of `java -version`

java -version openjdk version "17.0.2" 2022-01-18 OpenJDK Runtime Environment Temurin-17.0.2+8 (build 17.0.2+8) OpenJDK 64-Bit Server VM Temurin-17.0.2+8 (build 17.0.2+8, mixed mode, sharing)

GraalVM version (if different from Java)

No response

Quarkus version or git rev

2.13.2.Final

Build tool (ie. output of `mvnw --version` or `gradlew --version`)

Maven 3.8.1

Additional information

No response

The text was updated successfully, but these errors were encountered:

quarkus-bot · 2022-10-12T22:36:10Z

/cc @Ladicek, @brunobat, @machi1990, @mkouba, @radcortez

mkouba · 2022-10-13T06:21:05Z

I believe that this works as designed. You can't expect a traceId to be set during execution of a scheduled method that has no HTTP request associated. In fact, a scheduled method cannot have an HTTP request associated because it represents a kind of entry point in your application - there is no external event that would trigger the execution of a scheduled method.

I'm going to close this issue. Feel free to reopen if the paragraph above does not make any sense.

Ladicek · 2022-10-13T08:36:07Z

Technically, a scheduled method can call other external services etc., and so tracing may make sense. As you mention, a scheduled method is an entrypoint, so it deserves a fresh new trace/span, I'd say.

mkouba · 2022-10-13T09:04:49Z

Technically, a scheduled method can call other external services etc., and so tracing may make sense. As you mention, a scheduled method is an entrypoint, so it deserves a fresh new trace/span, I'd say.

I see. My point was that the tracing info should not be propagated but that was probably just a misunderstanding of the problem.

brunobat · 2022-10-13T09:20:07Z

Tracing does make sense on Quartz Jobs. This is not just for HTTP.
It will be great to understand the outcome of each job.
This requires 2 changes, though. One thing is to instrument jobs, another one is curating the MDC context with the right data.

syr · 2022-10-14T10:10:59Z

Thanks for adding the issue to the roadmap.
As a workaround I use otel-agent (via JAVA_TOOL_OPTIONS) and set the MDC fields by myself by puttying MDC entries for traceId, spanId, parentId from Span.current().getSpanContext()
Thus, all logs have the same trace id for each scheduled run.

However get two problem:

no idea where to get parentId from. As far as I read it is not possible to get it manually
manual instrumentation by @WithSpan is not working/ignored. Should it generally be possible to mix auto-instrumentation (by otel agent) and manual instrumenation? Updatde reproducer project https://github.com/syr/quartz-otel-logs/commit/4ccd464e8e216576070acfd63ff6bdf3084d1019

brunobat · 2022-10-14T10:25:28Z

You don't really need the parent. The SpanId and the TraceId are enough to search on Jaeger or an equivalent tool.
In Quarkus we don't recommend using the agent because of context propagation problems. The '@WithSpan' on a CDI bean should work with imperative code. If you can provide a reproducer it would help.

syr · 2022-10-14T10:28:10Z

https://github.com/syr/quartz-otel-logs/commit/4ccd464e8e216576070acfd63ff6bdf3084d1019

Without agent I don't get valid traceId, only initial values 0000000000000. So atm it's also only meant as a temp. workaround.

brunobat · 2022-10-14T10:33:18Z

We need to instrument Quartz with OpenTelemetry. Not done yet.
But if you call a CDI bean from quartz and use '@WithSpan' it should work.

syr · 2022-10-14T10:42:54Z

moved the method with @span to a seperate service class in https://github.com/syr/quartz-otel-logs/commit/57aef0684ec3cd944d3067b4d245203b2b299e67 , but still the same spanId

12:39:39 INFO  traceId=1646cfc30a48e24dfaa8a5e3f4cc4a75, parentId=unknown, spanId=5058966601badfb7, sampled= [or.ac.qu.TaskBean] (QuarkusQuartzScheduler_Worker-5) log test
  12:39:39 INFO  traceId=1646cfc30a48e24dfaa8a5e3f4cc4a75, parentId=unknown, spanId=5058966601badfb7, sampled= [or.ac.qu.TaskService] (QuarkusQuartzScheduler_Worker-5) log test from testMethod
´´´

brunobat · 2022-10-14T13:51:07Z

I think we'll need a reproducer project

syr · 2022-10-14T16:06:26Z

See link in my previous reply.

I also sent all traces to a local jaeger-aio where I saw in the jaeger UI that actually all spans were created. So it seems to me that maybe only MDC is not kept consistent. Will extend the reproducer project to verify..likely tomorrow.

syr · 2022-10-15T12:10:12Z

I added some log output that shows that spanId from spanContext vs. MDC is not consistent
https://github.com/syr/quartz-otel-logs

correct: logs from scheduled method (spanId in log entry is correct, but is also directly before set in MDC)

14:01:26 INFO  traceId=59ec1645b4243fdf69e4dc8a61377bf3, parentId=unknown, spanId=46e90f4f12994226, sampled= [or.ac.qu.TaskBean] (QuarkusQuartzScheduler_Worker-2) traceId: 59ec1645b4243fdf69e4dc8a61377bf3
14:01:26 INFO  traceId=59ec1645b4243fdf69e4dc8a61377bf3, parentId=unknown, spanId=46e90f4f12994226, sampled= [or.ac.qu.TaskBean] (QuarkusQuartzScheduler_Worker-2) spanId: 46e90f4f12994226
14:01:26 INFO  traceId=59ec1645b4243fdf69e4dc8a61377bf3, parentId=unknown, spanId=46e90f4f12994226, sampled= [or.ac.qu.TaskBean] (QuarkusQuartzScheduler_Worker-2) log test

IMO not correct: logs from service method, called by scheduled method (spanId in log entries should be b6b70be685472933 as logged, because method is annotated with @WithSpan)

14:01:26 INFO  traceId=59ec1645b4243fdf69e4dc8a61377bf3, parentId=unknown, spanId=46e90f4f12994226, sampled= [or.ac.qu.TaskService] (QuarkusQuartzScheduler_Worker-2) traceId in testMethod: 59ec1645b4243fdf69e4dc8a61377bf3
14:01:26 INFO  traceId=59ec1645b4243fdf69e4dc8a61377bf3, parentId=unknown, spanId=46e90f4f12994226, sampled= [or.ac.qu.TaskService] (QuarkusQuartzScheduler_Worker-2) spanId in testMethod: b6b70be685472933
14:01:26 INFO  traceId=59ec1645b4243fdf69e4dc8a61377bf3, parentId=unknown, spanId=46e90f4f12994226, sampled= [or.ac.qu.TaskService] (QuarkusQuartzScheduler_Worker-2) log test in testMethod

Maybe also another issue due to using quarkus-opentelemetry together with opentelemetry agent.

brunobat · 2022-10-17T09:24:43Z

The MDC context needs to be cleared after the job was done, with MDC.remove(...) after:
https://github.com/syr/quartz-otel-logs/blob/2ad12e7efbbb1557a17c57dc305280a50a92a3d0/src/main/java/org/acme/quartz/TaskBean.java#L34

syr · 2022-10-17T16:20:13Z

spanId from log from testMethod keeps unchanged when I do it like this..

  18:22:48 INFO  traceId=cbd5541755282dcac116f8dec98ac033, parentId=unknown, spanId=84d275bf0774f598, sampled= [or.ac.qu.TaskBean] (QuarkusQuartzScheduler_Worker-2) spanId: 84d275bf0774f598
  18:22:48 INFO  traceId=cbd5541755282dcac116f8dec98ac033, parentId=unknown, spanId=84d275bf0774f598, sampled= [or.ac.qu.TaskBean] (QuarkusQuartzScheduler_Worker-2) log test
  18:22:48 INFO  traceId=cbd5541755282dcac116f8dec98ac033, parentId=unknown, spanId=84d275bf0774f598, sampled= [or.ac.qu.TaskService] (QuarkusQuartzScheduler_Worker-2) traceId in testMethod: cbd5541755282dcac116f8dec98ac033
  18:22:48 INFO  traceId=cbd5541755282dcac116f8dec98ac033, parentId=unknown, spanId=84d275bf0774f598, sampled= [or.ac.qu.TaskService] (QuarkusQuartzScheduler_Worker-2) spanId in testMethod: f43a3652290c1da4

Only thing that worked but is cumbersome is to set also the MDC explicitly in the org.acme.quartz.TaskService#testMethod, so again like:

@WithSpan("custom span")
    public void testMethod(){
        SpanContext spanContext = Span.current().getSpanContext();
        MDC.put("traceId", spanContext.getTraceId());
        MDC.put("spanId", spanContext.getSpanId());
...

Which leads to correct/expected log field with new spanId and same traceId

  18:13:19 INFO  traceId=c99870cee4e1da346d9d73535db2ec5a, parentId=unknown, spanId=58e43c8f29b1f67b, sampled= [or.ac.qu.TaskBean] (QuarkusQuartzScheduler_Worker-2) spanId: 58e43c8f29b1f67b
  18:13:19 INFO  traceId=c99870cee4e1da346d9d73535db2ec5a, parentId=unknown, spanId=58e43c8f29b1f67b, sampled= [or.ac.qu.TaskBean] (QuarkusQuartzScheduler_Worker-2) log test
  18:13:19 INFO  traceId=c99870cee4e1da346d9d73535db2ec5a, parentId=unknown, spanId=8524d787f125a66b, sampled= [or.ac.qu.TaskService] (QuarkusQuartzScheduler_Worker-2) traceId in testMethod: c99870cee4e1da346d9d73535db2ec5a
  18:13:19 INFO  traceId=c99870cee4e1da346d9d73535db2ec5a, parentId=unknown, spanId=8524d787f125a66b, sampled= [or.ac.qu.TaskService] (QuarkusQuartzScheduler_Worker-2) spanId in testMethod: 8524d787f125a66b

Any idea why opentelemetry is not doing this automatically? Or is it not its responsibility?
Sorry, maybe I should ask this in opentelemetry github or SO

brunobat · 2022-10-18T09:04:34Z

There is no connection between the MDC context and OpenTelemetry.
On Quarkus we curate the MDC context with that data (it's a Quarkus specific feature), however, because we haven't instrumented Quartz yet, that's why the info is not being shown in the logs as well.
Setting the MDC context manually in the method works even if it's not clean. Please don't forget to call MDC.remove in the end of the method, otherwise you will get wrong log entries afterwards.
Once we implement the OTel Quartz instrumentation on Quarkus, you will be able to remove that code.

syr · 2022-10-18T09:55:09Z

@brunobat thank you for the clarification which brings some light in the dark and the hint about MDC.remove. Actually from my basic understanding would be easier if OTel would also set the MDC, maybe configurable by a flag.
Is there an overview of quarkus extensions thta are OTel-ready (except REST resources)?
Was working through https://quarkus.io/guides/opentelemetry which is good to start but does not have a list of OTel ready quarkus extensions listed.

syr · 2022-10-18T13:49:48Z

Or is it maybe possible to plug jboss logging to one of the logging frameworks (or vice versa) that supports automatic config of MDC (Log4J, Logback), see https://github.com/open-telemetry/opentelemetry-java-instrumentation/blob/main/docs/logger-mdc-instrumentation.md ?

syr · 2022-11-13T10:19:53Z

@Ladicek @brunobat @brunobat @radcortez
Is there some list of libraries that are instrumented by quarkus-opentelemetry?
I guess it must be a subset of https://github.com/open-telemetry/opentelemetry-java-instrumentation/blob/main/docs/supported-libraries.md, e.g. quartz not yet instrumented.

Also having problems, as context propagation with AWS SDK seems to be not working, neither with quarkus-opentelemetry nor with the opentelemetry agent...created issue with reproducer on optentelemetry github: open-telemetry/opentelemetry-java-instrumentation#7120 without any feedback,

In the meantime I wonder if I should really follow opentelemetry atm, as there are so many issues, maybe it is too new/experimental to be used.

brunobat · 2022-11-14T10:12:09Z

Hi Syr,
There is no such list for the time being and we currently don't follow that list.
We prefer to instrument Quarkus ourselves and make sure context propagation and native mode behave correctly.
I can tell you that inside Quarkus, today we instrument:

Vertx, including the HTTP server
REST clients
GRPC server
But also if you use:
Reactive messaging from SmallRye
SQL queries if you use the dedicated driver
Provide CDI aware beans where you can create your own spans

OpenTelemetry is constantly evolving but also needs the community contributions...

syr · 2022-11-16T10:27:06Z

Hi @brunobat, thanks for the info. Could you please add this list to the opentelemetry guide? IMO it's crucial to decide if the extension fits or if otel agent must be used to get more auto-instrumenation, like in our case using quartz and aws sdk libs.

Might be a silly question, but is there a recommended way atm for production applications?
OpenTracing is deprecated and OpenTelemetry experimental, so sitting between chairs atm, using custom stuff.
Would love to contribute, need to fight to get some time slots, which is super hard during a dev project, while otel is the most complex/foggy lib, especially about understanding responsibilities IMO.

brunobat · 2022-11-16T10:53:14Z

OpenTelemetry tracing is not experimental anymore. We are preparing a few changes, soon, that will wide the instrumentation support.
The OTel agent instrumentation is too generic. It seems to work but, as I mentioned, it has context propagation issues within the service, specially if you mix imperative and reactive code. The built in Quarkus instrumentation fixes that.

syr · 2022-11-17T16:59:09Z

OpenTelemetry tracing is not experimental anymore. We are preparing a few changes, soon, that will wide the instrumentation support.

OK sounds good. I was referring to the status shown at code.quarkus.io.
Is there a rough roadmap weeks/months/2023 when to expect these extended otel support beeing released?
This info would be very helpful to decide how much effort we put in workarounds using otel agent.

The OTel agent instrumentation is too generic. It seems to work but, as I mentioned, it has context propagation issues within the service, specially if you mix imperative and reactive code. The built in Quarkus instrumentation fixes that.

Yes I noticed quite some issues (mainly three points in open-telemetry/opentelemetry-java-instrumentation#7120 (comment)), however it was the only way I see to get quartz and aws sdk auto-instrumented atm.

We have quite few reactive code in the project, I would rather consider to change it back to impierative because there are also issues when Mutiny is combined with @transactional for example...likely mixing imperative and reactive is not a good idea. But that's another topic...

syr · 2022-11-28T11:46:28Z

@brunobat @Ladicek @mkouba
Can you please have a look at the last comment in open-telemetry/opentelemetry-java-instrumentation#7120 (comment)
Maybe there is not so much to change to get the MDC beeing updated when ussing jboss logger via io.quarkus.logging.Log

famod · 2022-11-28T18:17:37Z

I ran into the same problem, but with the simple scheduler (not quartz).
IMO, #28552 (comment) also applies to the simple scheduler.

Shall I open a separate issue?

brunobat · 2022-11-30T18:00:18Z

@famod yes please.

famod · 2022-11-30T18:05:30Z

@brunobat will do.

For the simple scheduler, I've realized that a workaround seems as easy as adding @WithSpan to my @Scheduled method. Am I missing something?

syr · 2022-11-30T18:38:47Z

@famod do you use otel agent or is this already working with quarkus-opentelemetry + quarkus-scheduler dependencies?

Btw. I would expect this to work with simple scheduler by default opposite to the described issue here which is scoped to quartz not beeing instrumented atm.
I would advice you to test to add another @WithSpan on a subsequent method, and log something, just to check that MDCs are correctly updated by Otel span context (traceId should stay the same, spanId should change).

famod · 2022-12-02T17:40:24Z

@syr I'm not using the otel agent explicitly, only quarkus-opentelemetry + quarkus-scheduler dependencies and additionally io.opentelemetry.instrumentation:opentelemetry-jdbc.

@WithSpan just works, I see JDBC sub-spans for my job executions.

I created a separate enhancement issue: #29647

Kasokaso · 2023-06-13T12:34:46Z

Is there any plan to do something similar when consuming events from Kafka? I believe it would be useful to understand the outcome of processed events.

brunobat · 2023-06-13T12:47:08Z

Not sure when this is going to be scheduled.

geoand · 2023-07-04T09:08:51Z

IIUC, everything just works with @WithSpan correct?

If so, maybe we just need to add a note in the scheduler documentation instead of doing all the work to enable tracing by default for scheduled methods?

Ladicek · 2023-07-04T09:18:54Z

If everything works with @WithSpan, which we treat as an interceptor binding, enabling tracing by default for scheduled methods would amount to a simple annotation transformation I'd say.

geoand · 2023-07-04T09:21:08Z

Good point

syr · 2023-07-04T16:22:14Z

IIUC, everything just works with @WithSpan correct?

If so, maybe we just need to add a note in the scheduler documentation instead of doing all the work to enable tracing by default for scheduled methods?

AFAIU it does not work when using custom job classes (similar to https://quarkus.io/guides/quartz#scheduling-jobs-programmatically).
Just tested with version 2.16.7.Final: https://github.com/syr/quarkus-resteasy-postgres/tree/otel_quartz_2.16

Prints from job execute method (org.acme.quartz.TaskCreateJob#execute):

18:12:16 INFO  traceId=, parentId=, spanId=, sampled= [or.ac.qu.TaskCreateJob] (DefaultQuartzScheduler_Worker-1) created task
18:12:17 INFO  traceId=, parentId=, spanId=, sampled= [or.ac.qu.TaskCreateJob] (DefaultQuartzScheduler_Worker-2) created task
18:12:18 INFO  traceId=, parentId=, spanId=, sampled= [or.ac.qu.TaskCreateJob] (DefaultQuartzScheduler_Worker-3) created task
18:12:19 INFO  traceId=, parentId=, spanId=, sampled= [or.ac.qu.TaskCreateJob] (DefaultQuartzScheduler_Worker-4) created task

However, it's interesting that it works (even without @WithSpan) when using @Scheduled e.g.:

    @Scheduled(every = "10s", identity = "task-job")
    void schedule() {
        Log.info("created task");
    }

with quarkus-quartz extension.

However, we need the custom job classes, so cannot use @Scheduled on methods :/

brunobat · 2023-07-04T16:58:33Z

@syr I see that you are implementing org.quartz.Job and annotating the execute method with @WithSpan.
I don't think this will be on a CDI context, therefore the interceptor will not kick in.
If you implement the job in a bean and then, you call it from inside the execute, it should work.

syr · 2023-07-04T17:24:02Z

@brunobat Yes, like this it's working

public class TaskCreateJob implements Job {
    @Override
    public void execute(JobExecutionContext jobExecutionContext) {
        Arc.container().select(TaskCreateJobBean.class).get().execute(jobExecutionContext);
    }
}

@ApplicationScoped
@Unremovable
public class TaskCreateJobBean  {
    @WithSpan
    public void execute(JobExecutionContext jobExecutionContext) {
        Log.info("created task");
    }
}

..but it's damn ugly. Was hoping there is a more elegant way.
Please let me know if you see a better/less redundant way.
I would not like to create bean classes for every job class we have.

radcortez · 2023-07-04T22:53:55Z

It would require adding proper OTel integration in a JobListener. This can be used as a basis: https://github.com/open-telemetry/opentelemetry-java-instrumentation/tree/main/instrumentation/quartz-2.0/library/src/main/java/io/opentelemetry/instrumentation/quartz/v2_0

Would you like to try submitting a PR with the proper support? :)

syr · 2023-07-10T14:04:47Z

It would require adding proper OTel integration in a JobListener. This can be used as a basis: https://github.com/open-telemetry/opentelemetry-java-instrumentation/tree/main/instrumentation/quartz-2.0/library/src/main/java/io/opentelemetry/instrumentation/quartz/v2_0

Would you like to try submitting a PR with the proper support? :)

In which repo would this have to be added? opentelemetry-java-instrumentation or quarkus-opentelemetry?

geoand · 2023-07-10T14:13:29Z

quarkus-opentelemetry

mskacelik · 2023-09-07T09:31:14Z

Gonna have a look at this

syr added the kind/bug Something isn't working label Oct 12, 2022

quarkus-bot bot added area/scheduler area/smallrye area/tracing labels Oct 12, 2022

mkouba closed this as not planned Won't fix, can't repro, duplicate, stale Oct 13, 2022

mkouba reopened this Oct 13, 2022

brunobat added this to Backlog in Observability roadmap [Deprecated] via automation Oct 13, 2022

This was referenced Dec 2, 2022

OpenTelemetry instrumentation stops working for resteasy-reactive resources and JDBC after dev mode reload #29645

Closed

Auto-create trace/span context via OTEL for simple @Scheduled methods #29647

Closed

brunobat moved this from In progress to Backlog in Observability roadmap [Deprecated] Dec 12, 2022

syr pushed a commit to syr/quarkus-resteasy-postgres that referenced this issue Jul 4, 2023

trying to reproduce quarkusio/quarkus#28552

84fe22d

syr added a commit to syr/quarkus-resteasy-postgres that referenced this issue Jul 4, 2023

trying to reproduce quarkusio/quarkus#28552

7ea4aff

syr added a commit to syr/quarkus-resteasy-postgres that referenced this issue Jul 4, 2023

reproduced quarkusio/quarkus#28552

fdfc1f5

syr added a commit to syr/quarkus-resteasy-postgres that referenced this issue Jul 4, 2023

reproduced quarkusio/quarkus#28552

b944bf9

mskacelik mentioned this issue Sep 18, 2023

Scheduler: OTel Instrumentation feature improvements and refactoring #35989

Merged

brunobat assigned mskacelik Sep 19, 2023

mkouba closed this as completed in #35989 Nov 29, 2023

Observability roadmap [Deprecated] automation moved this from Backlog to Done Nov 29, 2023

quarkus-bot bot added this to the 3.7 - main milestone Nov 29, 2023

traceId, spanId not set in logs when using quartz and opentelemetry #28552

traceId, spanId not set in logs when using quartz and opentelemetry #28552

Comments

syr commented Oct 12, 2022

Describe the bug

Expected behavior

Actual behavior

How to Reproduce?

Output of uname -a or ver

Output of java -version

GraalVM version (if different from Java)

Quarkus version or git rev

Build tool (ie. output of mvnw --version or gradlew --version)

Additional information

quarkus-bot bot commented Oct 12, 2022

mkouba commented Oct 13, 2022

Ladicek commented Oct 13, 2022

mkouba commented Oct 13, 2022

brunobat commented Oct 13, 2022 • edited

syr commented Oct 14, 2022 • edited

brunobat commented Oct 14, 2022

syr commented Oct 14, 2022

brunobat commented Oct 14, 2022

syr commented Oct 14, 2022

brunobat commented Oct 14, 2022

syr commented Oct 14, 2022 • edited

syr commented Oct 15, 2022 • edited

brunobat commented Oct 17, 2022

syr commented Oct 17, 2022 • edited

brunobat commented Oct 18, 2022

syr commented Oct 18, 2022 • edited

syr commented Oct 18, 2022 • edited

syr commented Nov 13, 2022 • edited

brunobat commented Nov 14, 2022

syr commented Nov 16, 2022 • edited

brunobat commented Nov 16, 2022

syr commented Nov 17, 2022 • edited

syr commented Nov 28, 2022

famod commented Nov 28, 2022

brunobat commented Nov 30, 2022

famod commented Nov 30, 2022

syr commented Nov 30, 2022 • edited

famod commented Dec 2, 2022

Kasokaso commented Jun 13, 2023

brunobat commented Jun 13, 2023

geoand commented Jul 4, 2023

Ladicek commented Jul 4, 2023

geoand commented Jul 4, 2023

syr commented Jul 4, 2023 • edited

brunobat commented Jul 4, 2023 • edited

syr commented Jul 4, 2023 • edited

radcortez commented Jul 4, 2023

syr commented Jul 10, 2023

geoand commented Jul 10, 2023

mskacelik commented Sep 7, 2023

Output of `uname -a` or `ver`

Output of `java -version`

Build tool (ie. output of `mvnw --version` or `gradlew --version`)

brunobat commented Oct 13, 2022 •

edited

syr commented Oct 14, 2022 •

edited

syr commented Oct 14, 2022 •

edited

syr commented Oct 15, 2022 •

edited

syr commented Oct 17, 2022 •

edited

syr commented Oct 18, 2022 •

edited

syr commented Oct 18, 2022 •

edited

syr commented Nov 13, 2022 •

edited

syr commented Nov 16, 2022 •

edited

syr commented Nov 17, 2022 •

edited

syr commented Nov 30, 2022 •

edited

syr commented Jul 4, 2023 •

edited

brunobat commented Jul 4, 2023 •

edited

syr commented Jul 4, 2023 •

edited