Fix native compilation for Netty and Mutiny #39830

jponge · 2024-04-02T10:41:41Z

jponge · 2024-04-02T11:44:37Z

Note: updated with a substitution for UnsafeRefArrayAccess / ignoring it could yield to segfaults as the field offset couldn't be computed (it uses a non-classic Unsafe access pattern).

zakkak · 2024-04-02T12:25:08Z

Note: updated with a substitution for UnsafeRefArrayAccess / ignoring it could yield to segfaults as the field offset couldn't be computed (it uses a non-classic Unsafe access pattern).

@jponge I had a look at org.graalvm.nativeimage.hosted.Feature.BeforeAnalysisAccess#registerAsUnsafeAccessed and it looks like it's mostly being used by Truffle.

Unsafe accesses seem to be automatically detected and registered by GraalVM, e.g. https://github.com/oracle/graal/blob/c95781d03099ebd1dad495ac1fefca863178b358/substratevm/src/com.oracle.graal.pointsto/src/com/oracle/graal/pointsto/flow/MethodTypeFlowBuilder.java#L304

Does this mean we can follow this pattern and completely remove UnsafeAccessedFieldBuildItems in favor of RecomputeFieldValue?

zakkak · 2024-04-02T12:43:09Z

@jponge please have a look at #39831 as well before merging this. ~~It makes the mutiny changes redundant.~~

jponge · 2024-04-02T13:01:53Z

Does this mean we can follow this pattern and completely remove UnsafeAccessedFieldBuildItems in favor of RecomputeFieldValue?

AFAIK most Unsafe patterns indeed get detected now, but one such as UnsafeRefArrayAccess.REF_ELEMENT_SHIFT is not, hence the need for either a substitution or a UnsafeAccessedFieldBuildItem that does not involved build-time init

quarkus-bot · 2024-04-02T16:14:23Z

Status for workflow `Quarkus CI`

This is the status report for running Quarkus CI on commit b35ae00.

Failing Jobs

Status	Name	Step	Failures	Logs	Raw logs	Build scan
✔️	JVM Tests - JDK 17			Logs	Raw logs	🚧
✖	JVM Tests - JDK 17 Windows	`Build`	Failures	Logs	Raw logs	🚧
✔️	JVM Tests - JDK 21			Logs	Raw logs	🚧

Failures

⚙️ JVM Tests - JDK 17 Windows #

- Failing: integration-tests/grpc-hibernate-reactive

📦 integration-tests/grpc-hibernate-reactive

✖ Failed to execute goal io.quarkus:quarkus-maven-plugin:999-SNAPSHOT:build (default) on project quarkus-integration-test-grpc-hibernate-reactive: Failed to build quarkus application

Flaky tests - Develocity

⚙️ Maven Tests - JDK 17 Windows

📦 integration-tests/maven

✖ io.quarkus.maven.it.TestMojoIT.testThatTheTestsAreReRunMultiModule - History

Condition with Lambda expression in io.quarkus.maven.it.continuoustesting.TestModeContinuousTestingMavenTestUtils was not fulfilled within 3 minutes. - org.awaitility.core.ConditionTimeoutException

org.awaitility.core.ConditionTimeoutException: Condition with Lambda expression in io.quarkus.maven.it.continuoustesting.TestModeContinuousTestingMavenTestUtils was not fulfilled within 3 minutes.
	at org.awaitility.core.ConditionAwaiter.await(ConditionAwaiter.java:167)
	at org.awaitility.core.CallableCondition.await(CallableCondition.java:78)
	at org.awaitility.core.CallableCondition.await(CallableCondition.java:26)
	at org.awaitility.core.ConditionFactory.until(ConditionFactory.java:1006)
	at org.awaitility.core.ConditionFactory.until(ConditionFactory.java:975)
	at io.quarkus.maven.it.continuoustesting.TestModeContinuousTestingMavenTestUtils.waitForNextCompletion(TestModeContinuousTestingMavenTestUtils.java:49)
	at io.quarkus.maven.it.LaunchMojoTestBase.testThatTheTestsAreReRunMultiModule(LaunchMojoTestBase.java:56)

✖ io.quarkus.maven.it.TestMojoIT.testThatTheTestsAreReRunMultiModule - History

Condition with Lambda expression in io.quarkus.maven.it.continuoustesting.TestModeContinuousTestingMavenTestUtils was not fulfilled within 3 minutes. - org.awaitility.core.ConditionTimeoutException

org.awaitility.core.ConditionTimeoutException: Condition with Lambda expression in io.quarkus.maven.it.continuoustesting.TestModeContinuousTestingMavenTestUtils was not fulfilled within 3 minutes.
	at org.awaitility.core.ConditionAwaiter.await(ConditionAwaiter.java:167)
	at org.awaitility.core.CallableCondition.await(CallableCondition.java:78)
	at org.awaitility.core.CallableCondition.await(CallableCondition.java:26)
	at org.awaitility.core.ConditionFactory.until(ConditionFactory.java:1006)
	at org.awaitility.core.ConditionFactory.until(ConditionFactory.java:975)
	at io.quarkus.maven.it.continuoustesting.TestModeContinuousTestingMavenTestUtils.waitForNextCompletion(TestModeContinuousTestingMavenTestUtils.java:49)
	at io.quarkus.maven.it.LaunchMojoTestBase.testThatTheTestsAreReRunMultiModule(LaunchMojoTestBase.java:56)

cescoffier · 2024-04-03T06:15:55Z

Does this mean we can follow this pattern and completely remove UnsafeAccessedFieldBuildItems in favor of RecomputeFieldValue?

I would be in favor of this. The UnsafeAccessedFieldBuildItems leads to loading the class at build time (and so run the initialization of the class) which can lead to issues if this class must be runtime only. I wonder if we can workaround this using "re-initialize at runtime".

zakkak · 2024-04-03T06:34:16Z

The UnsafeAccessedFieldBuildItems leads to loading the class at build time (and so run the initialization of the class) which can lead to issues if this class must be runtime only. I wonder if we can workaround this using "re-initialize at runtime".

FYI I have a fix for this behavior in #39831, but it's still WIP since it triggers other issues.

zakkak

LGTM

jponge · 2024-04-03T07:50:53Z

One down! 🤣

geoand · 2024-04-03T07:56:15Z

🎉

luneo7 · 2024-04-03T15:15:11Z

should this be backported to 3.8.4 as well? since vertx and netty were also upgraded there

geoand · 2024-04-03T15:16:35Z

Makes sense

jponge · 2024-04-03T15:17:17Z

Yes please!

gsmet · 2024-04-09T14:56:33Z

@jponge I don't think this can be backported as is to 3.9 and 3.8.

At least the commit with JCTools is causing issues as I think you are only using this with latest Mutiny?

Error: Substitution target for io.quarkus.mutiny.runtime.Target_org_jctools_util_UnsafeRefArrayAccess is not loaded. Use field `onlyWith` in the `TargetClass` annotation to make substitution only active when needed.
com.oracle.svm.core.util.UserError$UserException: Substitution target for io.quarkus.mutiny.runtime.Target_org_jctools_util_UnsafeRefArrayAccess is not loaded. Use field `onlyWith` in the `TargetClass` annotation to make substitution only active when needed.
	at org.graalvm.nativeimage.builder/com.oracle.svm.core.util.UserError.abort(UserError.java:73)
	at org.graalvm.nativeimage.builder/com.oracle.svm.hosted.substitute.AnnotationSubstitutionProcessor.findTargetClass(AnnotationSubstitutionProcessor.java:1074)
	at org.graalvm.nativeimage.builder/com.oracle.svm.hosted.substitute.AnnotationSubstitutionProcessor.handleClass(AnnotationSubstitutionProcessor.java:373)
	at org.graalvm.nativeimage.builder/com.oracle.svm.hosted.substitute.AnnotationSubstitutionProcessor.init(AnnotationSubstitutionProcessor.java:351)
	at org.graalvm.nativeimage.builder/com.oracle.svm.hosted.NativeImageGenerator.createAnnotationSubstitutionProcessor(NativeImageGenerator.java:1029)
	at org.graalvm.nativeimage.builder/com.oracle.svm.hosted.NativeImageGenerator.setupNativeImage(NativeImageGenerator.java:907)
	at org.graalvm.nativeimage.builder/com.oracle.svm.hosted.NativeImageGenerator.doRun(NativeImageGenerator.java:590)
	at org.graalvm.nativeimage.builder/com.oracle.svm.hosted.NativeImageGenerator.run(NativeImageGenerator.java:550)
	at org.graalvm.nativeimage.builder/com.oracle.svm.hosted.NativeImageGeneratorRunner.buildImage(NativeImageGeneratorRunner.java:539)
	at org.graalvm.nativeimage.builder/com.oracle.svm.hosted.NativeImageGeneratorRunner.build(NativeImageGeneratorRunner.java:721)
	at org.graalvm.nativeimage.builder/com.oracle.svm.hosted.NativeImageGeneratorRunner.start(NativeImageGeneratorRunner.java:143)
	at org.graalvm.nativeimage.builder/com.oracle.svm.hosted.NativeImageGeneratorRunner.main(NativeImageGeneratorRunner.java:98)

jponge · 2024-04-09T15:00:52Z

(discussed offline - yes 👍 )

gsmet · 2024-04-09T15:05:36Z

For 3.9 and 3.8, only the Netty changes should be backported.

See this commit: d2673c8

quarkus-bot bot added area/mutiny area/smallrye labels Apr 2, 2024

jponge linked an issue Apr 2, 2024 that may be closed by this pull request

Netty update caused failure when building image with GraalVM for JDK 17 #39819

Closed

Fix native compilation for Netty and Mutiny

b35ae00

See quarkusio#39819 and quarkusio#39788

jponge force-pushed the fix/netty-mutiny-native-39819 branch from b90033b to b35ae00 Compare April 2, 2024 11:43

jponge added the triage/backport label Apr 2, 2024

jponge mentioned this pull request Apr 2, 2024

No build time init of classes used in UnsafeAccessedFieldBuildItem #39831

Merged

quarkus-bot bot added the triage/flaky-test label Apr 2, 2024

jponge requested review from cescoffier, gsmet and geoand April 2, 2024 16:32

cescoffier approved these changes Apr 3, 2024

View reviewed changes

zakkak approved these changes Apr 3, 2024

View reviewed changes

geoand approved these changes Apr 3, 2024

View reviewed changes

geoand merged commit e1cc9a2 into quarkusio:main Apr 3, 2024
48 of 49 checks passed

quarkus-bot bot added the kind/bugfix label Apr 3, 2024

quarkus-bot bot added this to the 3.10 - main milestone Apr 3, 2024

geoand added the triage/backport-3.8 label Apr 3, 2024

gsmet modified the milestones: 3.10 - main, 3.9.3 Apr 9, 2024

gsmet removed the triage/backport label Apr 9, 2024

gsmet modified the milestones: 3.9.3, 3.8.4 Apr 9, 2024

gsmet removed the triage/backport-3.8 label Apr 9, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Fix native compilation for Netty and Mutiny #39830

Fix native compilation for Netty and Mutiny #39830

jponge commented Apr 2, 2024

jponge commented Apr 2, 2024

zakkak commented Apr 2, 2024

zakkak commented Apr 2, 2024 •

edited

jponge commented Apr 2, 2024

quarkus-bot bot commented Apr 2, 2024

cescoffier commented Apr 3, 2024

zakkak commented Apr 3, 2024

zakkak left a comment

jponge commented Apr 3, 2024

geoand commented Apr 3, 2024

luneo7 commented Apr 3, 2024 •

edited

geoand commented Apr 3, 2024

jponge commented Apr 3, 2024

gsmet commented Apr 9, 2024

jponge commented Apr 9, 2024

gsmet commented Apr 9, 2024

Fix native compilation for Netty and Mutiny #39830

Fix native compilation for Netty and Mutiny #39830

Conversation

jponge commented Apr 2, 2024

jponge commented Apr 2, 2024

zakkak commented Apr 2, 2024

zakkak commented Apr 2, 2024 • edited

jponge commented Apr 2, 2024

quarkus-bot bot commented Apr 2, 2024

Status for workflow Quarkus CI

Failing Jobs

Failures

⚙️ JVM Tests - JDK 17 Windows #

📦 integration-tests/grpc-hibernate-reactive

Flaky tests - Develocity

⚙️ Maven Tests - JDK 17 Windows

📦 integration-tests/maven

cescoffier commented Apr 3, 2024

zakkak commented Apr 3, 2024

zakkak left a comment

Choose a reason for hiding this comment

jponge commented Apr 3, 2024

geoand commented Apr 3, 2024

luneo7 commented Apr 3, 2024 • edited

geoand commented Apr 3, 2024

jponge commented Apr 3, 2024

gsmet commented Apr 9, 2024

jponge commented Apr 9, 2024

gsmet commented Apr 9, 2024

zakkak commented Apr 2, 2024 •

edited

Status for workflow `Quarkus CI`

luneo7 commented Apr 3, 2024 •

edited