Build time analytics #33555

brunobat · 2023-05-23T13:51:53Z

This PR enables an opt-in feature to allow gathering of build time analytics to get info/insights about how Quarkus is being used during development/builds (only build time NOT runtime).

TO TEST THIS

build the PR first using mvn -Dquickly, then create a project using latest snapshot and with a non quarkus grouped, for example: quarkus create app -P io.quarkus:quarkus-bom:999-SNAPSHOT something.random:myapp.

Please try and use your own groupid/name so we get more diverse reports.

Then run quarkus dev in the project and say y to report data.
Running in dev mode and mvn quarkus:dev, and gradle quarkusDev should report then submit data on each run.

Check target/build-analytics-event.json for content of what was sent.

Implementation

It is implemented using Segment which is used by other Red Hat projects to gather anonymous usage data.

Intent is that on first quarkus dev mode user will be asked if they wish to share anonymously build time info to Quarkus team.

If answer is no, marker file is created in ~/.redhat and no recording/analytics is made before that markerfile is removed.

If answer is yes, markerfile is made + ~/.redhat/anonymousId is created which is just to be able to group data per installation without any identifiable info in them. From that point on when ever quarkus build plugin runs it will send a json packet with info. This is sent async and non-blocking. Meaning build won't block because of this.

One-way SHA-265 hashes representing the application:
- groupId+artifactId
- artifactId+version
Name/Version info for Java, GraalVM, Operating System, Build system
Whether or not Kubernetes is used on the build
Known Extensions enabled
i.e. group-ids "io.quarkus", "io.quarkiverse", “org.apache.camel.quarkus” or listed in Quarkus platform.
The IP is set to 0.0.0.0 on purpose to disable any possible inference, if not present.

An example of data collected is shown below - all non-identifiable info.

{
  "context": {
    "app": {
      "name": "ZzOlt3q0VgNK0oyUd6NDAhPqJjTSxNOAnqHb5dHfS7M=",
      "version": "FUDdh16tTKtuJkcoY2Wp57IKKwxZ/Hx42tTvi3Izpmc="
    },
    "build": {
      "gradle_version": "N/A",
      "maven_version": "3.8.7"
    },
    "ci": {
      "name": "unknown"
    },
    "graalvm": {
      "java_version": "N/A",
      "vendor": "N/A",
      "version": "N/A"
    },
    "ip": "0.0.0.0",
    "java": {
      "vendor": "Eclipse Adoptium",
      "version": "17.0.4.1"
    },
    "kubernetes": {
      "detected": "false"
    },
    "location": {
      "country": "PT"
    },
    "os": {
      "name": "Mac OS X",
      "os_arch": "aarch64",
      "version": "13.3.1"
    },
    "quarkus": {
      "version": "999-SNAPSHOT"
    },
    "timezone": "Europe/Lisbon"
  },
  "event": "PROD_BUILD",
  "integrations": {},
  "messageId": "api-2QCEFlcxqJeuKjX4Uf0m1aWoKED",
  "originalTimestamp": "2023-05-23T13:37:39.913591Z",
  "properties": {
    "app_extensions": [
      {
        "artifact_id": "quarkus-resteasy-jackson",
        "group_id": "io.quarkus",
        "version": "999-SNAPSHOT"
      },
      {
        "artifact_id": "quarkus-smallrye-openapi",
        "group_id": "io.quarkus",
        "version": "999-SNAPSHOT"
      },
      {
        "artifact_id": "quarkus-hibernate-orm",
        "group_id": "io.quarkus",
        "version": "999-SNAPSHOT"
      },
      {
        "artifact_id": "quarkus-opentelemetry",
        "group_id": "io.quarkus",
        "version": "999-SNAPSHOT"
      },
      {
        "artifact_id": "quarkus-jdbc-postgresql",
        "group_id": "io.quarkus",
        "version": "999-SNAPSHOT"
      },
      {
        "artifact_id": "quarkus-arc",
        "group_id": "io.quarkus",
        "version": "999-SNAPSHOT"
      }
    ]
  },
  "receivedAt": "2023-05-23T13:37:40.856Z",
  "timestamp": "2023-05-23T13:37:39.913Z",
  "type": "track",
  "userId": "58991333-5f25-4401-ae5b-e3eb09ceb4de"
}

github-actions · 2023-05-24T18:39:43Z

🙈 The PR is closed and the preview is expired.

brunobat · 2023-05-25T13:57:22Z

@snazy would you be able to double check my changes to the gradle plugin?

snazy · 2023-05-26T05:20:41Z

@snazy would you be able to double check my changes to the gradle plugin?

Sure thing.

snazy

Took a look, mostly from the Gradle plugin. I recommend to call the AnalyticsService directly from the BuildWorker, which should have everything that's needed. Doing so would also remove the need for the added tasks and the new extension. Having some Gradle IT(s) would be nice.

...gradle/gradle-application-plugin/src/main/java/io/quarkus/gradle/tasks/QuarkusAnalytics.java

.../gradle-application-plugin/src/main/java/io/quarkus/gradle/extension/AnalyticsExtension.java

...dent-projects/tools/analytics-common/src/main/java/io/quarkus/analytics/rest/RestClient.java

...jects/tools/analytics-common/src/test/java/io/quarkus/analytics/rest/RestClientFailTest.java

...ent-projects/tools/analytics-common/src/main/java/io/quarkus/analytics/AnalyticsService.java

brunobat · 2023-05-29T14:51:45Z

Thanks @snazy. The review was much appreciated.

...projects/tools/analytics-common/src/main/java/io/quarkus/analytics/config/GroupIdFilter.java

...ts/tools/analytics-common/src/main/java/io/quarkus/analytics/dto/segment/TrackEventType.java

...ent-projects/tools/analytics-common/src/main/java/io/quarkus/analytics/AnalyticsService.java

maxandersen

logs when they show up in mvn/gradle output does not tell what is making the log/error - ie. quarkus usage analytics.

quarkus-bot · 2023-06-05T06:29:11Z

/cc @mjurc (qe)

...endent-projects/tools/analytics-common/src/main/java/io/quarkus/analytics/ConfigService.java

brunobat · 2023-06-16T09:23:21Z

Updated the prompt and it's text and added a section to the documentation about How to disable analytics @rolfedh.

brunobat · 2023-06-16T09:45:43Z

rsvobodawould you also be able to test in gradle and also behind a proxy?

...endent-projects/tools/analytics-common/src/main/java/io/quarkus/analytics/ConfigService.java

snazy

Looked through the Gradle changes and some related pieces of the code. The Gradle changes mostly LGTM, a couple mostly minor comments, except that there's no unit/integration test for Gradle.

One concern about the use of the fork-join-common-pool in ConfigService.

Also added some suggestions (maybe not doable at this stage?) about the contents of the analytics data.

...adle/gradle-application-plugin/src/main/java/io/quarkus/gradle/tasks/worker/BuildWorker.java

...ools/gradle/gradle-application-plugin/src/test/java/io/quarkus/gradle/QuarkusPluginTest.java

devtools/gradle/gradle-application-plugin/src/main/java/io/quarkus/gradle/QuarkusPlugin.java

snazy · 2023-06-19T08:02:04Z

...ent-projects/tools/analytics-common/src/main/java/io/quarkus/analytics/AnalyticsService.java

+                .pair(PROP_OS_ARCH, getProperty("os.arch", "N/A"))
+                .build()
+                .mapPair(PROP_CI)
+                .pair(PROP_CI_NAME, getBuildSystemName())


Does it make sense to have a property that tells whether the CI environment is set to indicate that it's running in CI (in case getBuildSystemName returns unknown for some exotic CI)?

There are a limited number of CIs we can detect. It returns unknown when he couldn't detect any CI.

snazy · 2023-06-19T08:03:41Z

...ent-projects/tools/analytics-common/src/main/java/io/quarkus/analytics/AnalyticsService.java

+                .pair(PROP_VERSION, getProperty("java.version", "N/A"))
+                .build()
+                .mapPair(PROP_GRAALVM)
+                .pair(PROP_VENDOR, ofNullable(buildInfo.get(GRAALVM_VERSION_DISTRIBUTION)).orElse("N/A"))


Nit: wonder why not just Map.getOrDefault?

It would be less verbose but I'll not change that now to reduce the amount of changes.

...ent-projects/tools/analytics-common/src/main/java/io/quarkus/analytics/AnalyticsService.java

snazy · 2023-06-19T08:20:43Z

Forgot to mention: the Gradle changes look MUCH simpler now - thanks for that!

brunobat · 2023-06-19T10:29:22Z

Forgot to mention: the Gradle changes look MUCH simpler now - thanks for that!

Thanks for your guidance @snazy!
Still working on an IT test... The setup is tricky.

brunobat · 2023-06-19T16:28:04Z

Updated a field name on the remote payload

quarkus-bot · 2023-06-20T04:09:17Z

Failing Jobs - Building `9061f37`

Status	Name	Step	Failures	Logs	Raw logs
✔️	JVM Tests - JDK 11
✔️	JVM Tests - JDK 17
✖	JVM Tests - JDK 17 Windows	`Build`	Failures	Logs	Raw logs
✔️	JVM Tests - JDK 19
✖	Native Tests - Windows - RESTEasy Jackson	`Setup GraalVM`	⚠️ Check →	Logs	Raw logs

Full information is available in the Build summary check run.

Failures

⚙️ JVM Tests - JDK 17 Windows #

- Failing: integration-tests/smallrye-context-propagation

📦 integration-tests/smallrye-context-propagation

✖ io.quarkus.context.test.SimpleContextPropagationTest.testArcTCContextPropagationDisabled line 92 - More details - Source on GitHub

org.opentest4j.AssertionFailedError: expected: <2> but was: <3>
	at org.junit.jupiter.api.AssertionFailureBuilder.build(AssertionFailureBuilder.java:151)
	at org.junit.jupiter.api.AssertionFailureBuilder.buildAndThrow(AssertionFailureBuilder.java:132)

brunobat · 2023-06-20T08:12:36Z

@paulrobinson @maxandersen @rolfedh @rsvoboda @snazy @jsmrcka Do you have any concerns with the PR? If not, can you please approve?

rsvoboda · 2023-06-20T09:16:58Z

would you also be able to test in gradle and also behind a proxy?

Sorry, was on pto. I don't capacity to look into this today. Not aware of any proxy I could use right now.

gsmet

I added a few comments and questions but I see nothing blocking.

Let's make sure we fix these issues (when they are actually issues) before the Final though.

Let me test it locally again and I'll give my approval.

core/runtime/src/main/java/io/quarkus/runtime/BuildAnalyticsConfig.java

devtools/gradle/gradle-application-plugin/src/main/java/io/quarkus/gradle/Prompter.java

devtools/gradle/gradle-application-plugin/src/main/java/io/quarkus/gradle/tasks/QuarkusDev.java

...adle/gradle-application-plugin/src/main/java/io/quarkus/gradle/tasks/worker/BuildWorker.java

gsmet · 2023-06-20T09:18:44Z

devtools/maven/src/main/java/io/quarkus/maven/BuildMojo.java

+    @Component
+    BuildAnalyticsProvider analyticsProvider;


The component is automatically closed? I'm surprised it is given it doesn't implement any marker interface.

Closing is performed in the BootstrapSessionListener

docs/src/main/asciidoc/build-analytics.adoc

gsmet · 2023-06-20T09:21:47Z

docs/src/main/asciidoc/build-analytics.adoc

+* Build system details, such as Maven, Gradle, and so on.
+* If a CI system was detected
+* Country and timezone
+* Extensions enabled. Only extensions whose `groupIds` start with `io.quarkus`, `io.quarkiverse` or are included in the quarkus platform are collected.


will rephrase to:
Only extensions whose groupIds start with io.quarkus, io.quarkiverse and others included in the quarkus platform are collected.

docs/src/main/asciidoc/build-analytics.adoc

gsmet · 2023-06-20T09:25:31Z

...endent-projects/tools/analytics-common/src/main/java/io/quarkus/analytics/ConfigService.java

+        } else {
+            try {
+                CompletableFuture<String> userInputFuture = CompletableFuture
+                        .supplyAsync(() -> analyticsEnabledSupplier.apply(ACCEPTANCE_PROMPT));


@geoand @cescoffier could you have a look at this?

gsmet · 2023-06-20T09:26:18Z

...jects/tools/analytics-common/src/main/java/io/quarkus/analytics/config/ExtensionsFilter.java

+            "io.quarkus",
+            "io.quarkiverse",
+            "org.apache.camel.quarkus",
+            "io.debezium",
+            "org.drools",
+            "org.optaplanner",
+            "org.amqphub.quarkus",
+            "com.hazelcast",
+            "com.datastax.oss.quarkus");


OK, so you need to clarify the list in the documentation because we are not doing what we documented.

These are group Ids of extensions bundled on the platform.

brunobat · 2023-06-20T10:21:44Z

On a follow up PR:

The config files namespace will change to io.quarkus.analytics.*
Support environment properties
localconfig should have disabled:true
Improve maven logging

rsvoboda · 2023-06-20T12:23:29Z

I don't capacity to look into this today.

I managed to shift some stuff to Wed and Thu so I have a bit of time to look into this.

Windows
There are no dev mode logs once y is answered to sharing build time analytics

See bottom left green icon that indicates the app is running, but there are no logs from the app boot

Proxy
We do not really have a proper production proxy setup in RH that would simulate restrictions some companies introduce on their employees.

I tried company internal squid instance and configured it in macOS network settings for http and https. No warnings in the application logs, even if I used bad port number.

I checked the code and it uses

    private HttpClient createHttpClient() {
        return HttpClient.newBuilder()
                .followRedirects(HttpClient.Redirect.ALWAYS)
                .connectTimeout(Duration.ofMillis(timeoutMs))
                .build();
    }

I don't think the client is checking or setting anything for proxy
To use proxy there is this construct for HttpClient

    HttpClient.newBuilder()
        .proxy(ProxySelector.of(new InetSocketAddress(...)))

brunobat · 2023-06-20T12:54:43Z

@rsvoboda Thanks very much.

Will debug and fix the windows issue in a new PR.

In relation to the proxy. My main concern was if it would hang the build or blow up in some unforeseen way. Proxy handling should be a concern for later.

rsvoboda · 2023-06-20T13:04:33Z

I tried the builds with disconnected network, nothing was delayed or hanging.

gsmet

Let's merge and iterate from there.

We decided to merge it.

quarkus-bot bot added this to To do in Quarkus Documentation May 23, 2023

This comment has been minimized.

Sign in to view

brunobat force-pushed the build-analytics branch from 385a377 to f8dd0d6 Compare May 24, 2023 18:05

quarkus-bot bot added the area/testing label May 24, 2023

brunobat marked this pull request as ready for review May 25, 2023 13:19

This comment has been minimized.

Sign in to view

brunobat force-pushed the build-analytics branch from f8dd0d6 to 95c5b51 Compare May 25, 2023 13:52

This comment has been minimized.

Sign in to view

snazy reviewed May 29, 2023

View reviewed changes

brunobat commented Jun 2, 2023

View reviewed changes

...projects/tools/analytics-common/src/main/java/io/quarkus/analytics/config/GroupIdFilter.java Show resolved Hide resolved

brunobat commented Jun 2, 2023

View reviewed changes

...ts/tools/analytics-common/src/main/java/io/quarkus/analytics/dto/segment/TrackEventType.java Outdated Show resolved Hide resolved

maxandersen reviewed Jun 3, 2023

View reviewed changes

...ent-projects/tools/analytics-common/src/main/java/io/quarkus/analytics/AnalyticsService.java Outdated Show resolved Hide resolved

maxandersen previously requested changes Jun 3, 2023

View reviewed changes

Quarkus Documentation automation moved this from To do to Review pending Jun 3, 2023

rsvoboda added the triage/qe? Issue could use a quality focused review to further harden it label Jun 5, 2023

paulrobinson reviewed Jun 5, 2023

View reviewed changes

...endent-projects/tools/analytics-common/src/main/java/io/quarkus/analytics/ConfigService.java Outdated Show resolved Hide resolved

paulrobinson reviewed Jun 5, 2023

View reviewed changes

...endent-projects/tools/analytics-common/src/main/java/io/quarkus/analytics/ConfigService.java Outdated Show resolved Hide resolved

paulrobinson reviewed Jun 5, 2023

View reviewed changes

...endent-projects/tools/analytics-common/src/main/java/io/quarkus/analytics/ConfigService.java Outdated Show resolved Hide resolved

brunobat requested a review from maxandersen June 16, 2023 09:31

maxandersen reviewed Jun 16, 2023

View reviewed changes

...endent-projects/tools/analytics-common/src/main/java/io/quarkus/analytics/ConfigService.java Show resolved Hide resolved

This comment has been minimized.

Sign in to view

snazy reviewed Jun 19, 2023

View reviewed changes

brunobat force-pushed the build-analytics branch from a8e053f to eff7783 Compare June 19, 2023 10:56

Build time analytics

9061f37

brunobat force-pushed the build-analytics branch from eff7783 to 9061f37 Compare June 19, 2023 16:34

This comment has been minimized.

Sign in to view

gsmet reviewed Jun 20, 2023

View reviewed changes

gsmet approved these changes Jun 20, 2023

View reviewed changes

Quarkus Documentation automation moved this from Review pending to Reviewer approved Jun 20, 2023

gsmet merged commit df7445d into quarkusio:main Jun 20, 2023
47 of 49 checks passed

Quarkus Documentation automation moved this from Reviewer approved to Done Jun 20, 2023

quarkus-bot bot added this to the 3.2 - main milestone Jun 20, 2023

gsmet mentioned this pull request Jun 20, 2023

Adjust usage page link per Max's complaint #34167

Merged

brunobat mentioned this pull request Jun 20, 2023

Build analytics improvements #34168

Merged

jsmrcka mentioned this pull request Jul 7, 2023

Amend QUARKUS-2812 test plan quarkus-qe/quarkus-test-plans#145

Merged

1 task

Build time analytics #33555

Build time analytics #33555

Conversation

brunobat commented May 23, 2023 • edited by maxandersen

TO TEST THIS

Implementation

This comment has been minimized.

github-actions bot commented May 24, 2023 • edited

This comment has been minimized.

brunobat commented May 25, 2023

This comment has been minimized.

snazy commented May 26, 2023

snazy left a comment

Choose a reason for hiding this comment

brunobat commented May 29, 2023

maxandersen left a comment

Choose a reason for hiding this comment

quarkus-bot bot commented Jun 5, 2023

brunobat commented Jun 16, 2023

brunobat commented Jun 16, 2023

This comment has been minimized.

snazy left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

brunobat Jun 19, 2023 • edited

Choose a reason for hiding this comment

snazy commented Jun 19, 2023

brunobat commented Jun 19, 2023

brunobat commented Jun 19, 2023

This comment has been minimized.

quarkus-bot bot commented Jun 20, 2023

Failing Jobs - Building 9061f37

Failures

⚙️ JVM Tests - JDK 17 Windows #

📦 integration-tests/smallrye-context-propagation

brunobat commented Jun 20, 2023

rsvoboda commented Jun 20, 2023

gsmet left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

brunobat commented Jun 20, 2023 • edited

rsvoboda commented Jun 20, 2023

brunobat commented Jun 20, 2023

rsvoboda commented Jun 20, 2023

gsmet left a comment

Choose a reason for hiding this comment

brunobat commented May 23, 2023 •

edited by maxandersen

github-actions bot commented May 24, 2023 •

edited

brunobat Jun 19, 2023 •

edited

Failing Jobs - Building `9061f37`

brunobat commented Jun 20, 2023 •

edited