Different behavior in native build using env var in application.properties when build with gradle vs. maven (Quarkus 3.x)

[dagrammy]

Describe the bug

Setting a environment variable which is used like quarkus.some.var=${MY_ENV_VAR:defaultValue} in application.properties has no effect when the native image is build with gradle.

The important part is the replacement of MY_ENV_VAR with the set value in the property quarkus.some.var.

This can lead to unexpected behaviour during execution in production because a property does not have the expected value.
The behavior is only reproducible in Quarkus 3.x, but not in Quarkus 2.x.

To describe and reproduce this behavior the following configuration example is used:
quarkus.banner.enabled=${BANNER:false}

Reproducers:
maven -> https://github.com/dagrammy/quarkus-env-reproducer-maven
gradle -> https://github.com/dagrammy/quarkus-env-reproducer-gradle

Expected behavior

Setting the env var BANNER to true before starting the application should show the banner regardless of the build tool used.

Actual behavior

Using maven:
Expected behavior is met, the banner is shown.

Using gradle:
Expected behavior is not met, the banner is NOT shown.

How to Reproduce?

maven -> https://github.com/dagrammy/quarkus-env-reproducer-maven

build it using maven:

unset BANNER 
./mvnw clean package -Pnative

run it without env var:

./target/command-mode-quickstart-1.0.0-SNAPSHOT-runner 

-> header is NOT shown

run it with env var:

export BANNER=true 
./target/command-mode-quickstart-1.0.0-SNAPSHOT-runner 

-> header is shown

---

gradle -> https://github.com/dagrammy/quarkus-env-reproducer-gradle

build it using gradle:

unset BANNER 
./gradlew clean build -Dquarkus.package.type=native --info

run it without env var:

./build/command-mode-quickstart-gradle-1.0.0-SNAPSHOT-runner 

-> header is NOT shown

run it with env var:

export BANNER=true 
./build/command-mode-quickstart-gradle-1.0.0-SNAPSHOT-runner 

-> header is NOT shown although env var is set

Output of uname -a or ver

Darwin grammy.local 22.3.0 Darwin Kernel Version 22.3.0: Thu Jan 5 20:49:43 PST 2023; root:xnu-8792.81.2~2/RELEASE_ARM64_T8103 arm64

Output of java -version

openjdk version "17.0.5" 2022-10-18 OpenJDK Runtime Environment GraalVM CE 22.3.0 (build 17.0.5+8-jvmci-22.3-b08) OpenJDK 64-Bit Server VM GraalVM CE 22.3.0 (build 17.0.5+8-jvmci-22.3-b08, mixed mode, sharing)

GraalVM version (if different from Java)

No response

Quarkus version or git rev

3.0.3

Build tool (ie. output of mvnw --version or gradlew --version)

Apache Maven 3.8.6 / Gradle 7.5.1

Additional information

No response

[quarkus-bot]

/cc @glefloch, @quarkusio/devtools (maven)

[maxandersen]

I don't know for sure if this PR is the cause but I recall it not looking right as priorities of config seem to have changed.

#32662 (comment)

@aloubyansky @Eng-Fouad wdyt?

[ThoSap]

I experience the very same issue with for example the Agroal environment variable QUARKUS_DATASOURCE__DATASOURCE_NAME__PASSWORD with Quarkus 3.0.3.Final, but this happens for all env vars and I'm also using Gradle as the build tool in my project.

In the QUARKUS_DATASOURCE__DATASOURCE_NAME__PASSWORD I have a $ inside the password (yes I know this really sucks), which I correctly escaped with \$ in bash and $$ in YAML files (application.yml, values.yaml of the HELM chart or the final generated deployment.yml of the Helm chart), which worked perfectly fine before with Quarkus 2.16.7.Final

As a workaround, I currently use the system properties as a command in the K8s Deployment e.g.

          command:
            - ./application
          args:
            - '-Dquarkus.http.host=0.0.0.0'
            # no $$ escape is needed here as the YAML string is enclosed in ' '
            - '-Dquarkus.datasource.customdatasource.password=SECRET$PW'

Output of java -version

My GraalVM native-image builds are done on Windows 11 22H2 22621.1702 using https://github.com/graalvm/graalvm-ce-builds/releases/tag/vm-22.3.2 and in the CI with builder Docker image quay.io/quarkus/ubi-quarkus-graalvmce-builder-image:22.3.2-java17

openjdk version "17.0.7" 2023-04-18
OpenJDK Runtime Environment GraalVM CE 22.3.2 (build 17.0.7+7-jvmci-22.3-b18)
OpenJDK 64-Bit Server VM GraalVM CE 22.3.2 (build 17.0.7+7-jvmci-22.3-b18, mixed mode, sharing)

Quarkus version or git rev

3.0.3.Final

Build tool (ie. output of mvnw --version or gradlew --version)

The build is started with ./gradlew build -Dquarkus.package.type=native both on Windows and the multi-stage Dockerfile with the quay.io/quarkus/ubi-quarkus-graalvmce-builder-image:22.3.2-java17 Docker builder image

Gradle 8.1.1
Build time:   2023-04-21 12:31:26 UTC
Revision:     1cf537a851c635c364a4214885f8b9798051175b

Extensions used in build.gradle

dependencies {
    /*
     * Quarkus
     */
    implementation enforcedPlatform("${quarkusPlatformGroupId}:${quarkusPlatformArtifactId}:${quarkusPlatformVersion}")
    implementation 'io.quarkus:quarkus-agroal'
    implementation 'io.quarkus:quarkus-arc'
    implementation 'io.quarkus:quarkus-config-yaml'
    implementation 'io.quarkus:quarkus-grpc'
    implementation 'io.quarkus:quarkus-jdbc-oracle'
    implementation 'io.quarkus:quarkus-kubernetes'
    implementation 'io.quarkus:quarkus-micrometer-registry-prometheus'
    implementation 'io.quarkus:quarkus-oidc'
    implementation 'io.quarkus:quarkus-resteasy-reactive-jackson'
    implementation 'io.quarkus:quarkus-smallrye-health'
    implementation 'io.quarkus:quarkus-smallrye-openapi'

    testImplementation 'io.quarkus:quarkus-junit5'
    testImplementation 'io.rest-assured:rest-assured'

    /*
     * Lombok
     */
    compileOnly "org.projectlombok:lombok:1.18.26"
    annotationProcessor "org.projectlombok:lombok:1.18.26"

    /*
     * Guava
     */
    implementation "com.google.guava:guava:31.1-jre"

    /*
     * Protobuf Utils
     */
    implementation "com.google.protobuf:protobuf-java-util:3.22.0"

    /*
     * Quarkus Helm Runtime
     */
    implementation "io.quarkiverse.helm:quarkus-helm:1.0.7"

    /*
     * GraalVM SubstrateVM image builder components
     */
    compileOnly 'org.graalvm.sdk:graal-sdk'
}

[MarkusKramer]

I'm also seeing a similar issue. I have in my application.properties

db.host=localhost
db.port=5432
db.name=postgres
db.schema=public
quarkus.datasource.jdbc.url=jdbc:postgresql://${db.host}:${db.port}/${db.name}
quarkus.datasource.reactive.url=postgresql://${db.host}:${db.port}/${db.name}?search_path=${db.schema}

Previously (Quarkus 2.x) I could set the db host by setting an environment variable "DB_HOST=something".

On Quarkus 3.0.3 when running as native image this no longer works (jvm mode works). Instead it always tried to connect to localhost.

[dagrammy]

@MarkusKramer That is exactly the problem. In native builds with gradle, the environment variables are not resolved correctly with Quarkus 3.0.x.

@maxandersen I took a look at the PR you mentioned. Doesn't this change only affect quarkus dev mode? Are there perhaps other parts of the code that I could take a look at?

[ThoSap]

This issue may be a regression of the following PRs:

• 3.0.0.Alpha1
  • No changelog available, did not check all commits
  • Empty milestone https://github.com/quarkusio/quarkus/milestone/213
• 3.0.0.Alpha2
  • No changelog available, did not check all commits
  • Empty milestone https://github.com/quarkusio/quarkus/milestone/222
• 3.0.0.Alpha3
  • Changelog not complete. I did not check all commits
  • Not complete milestone https://github.com/quarkusio/quarkus/milestone/229?closed=1
• 3.0.0.Alpha4
  • Changelog not complete. I did not check all commits, but the Gradle PRs mentioned in the changelog seem not to be related to this issue
  • Milestone https://github.com/quarkusio/quarkus/milestone/228?closed=1
• 3.0.0.Alpha5
  • Bump Gradle to version 8.0 #31159
  • Milestone https://github.com/quarkusio/quarkus/milestone/235?closed=1
• 3.0.0.Alpha6
  • I don't think there are related PRs to this issue in this release
  • Milestone https://github.com/quarkusio/quarkus/milestone/239?closed=1
• 3.0.0.Beta1
  • I don't think there are related PRs to this issue in this release
  • Milestone https://github.com/quarkusio/quarkus/milestone/240?closed=1
• 3.0.0.CR1
  • Make QuarkusBuild not pollute Gradle's build cache #31166
  • Milestone https://github.com/quarkusio/quarkus/milestone/241?closed=1
• 3.0.0.CR2
  • I don't think there are related PRs to this issue in this release
  • Milestone https://github.com/quarkusio/quarkus/milestone/243?closed=1
• 3.0.0.Final
  • Gradle-Plugin: Do not let Gradle build fail w/ configuration cache #32508
  • Gradle: ability to disable worker process isolation, pass all envs+sys-props #32524
  • Gradle: do not pass all system properties #32535
  • Milestone https://github.com/quarkusio/quarkus/milestone/244?closed=1
• 3.0.1.Final
  • Gradle integration tests: tackle (Windows) CI issues #32571
  • Milestone https://github.com/quarkusio/quarkus/milestone/246?closed=1
• 3.0.2.Final
  • I don't think there are related PRs to this issue in this release
  • Milestone https://github.com/quarkusio/quarkus/milestone/248?closed=1
• 3.0.3.Final
  • I don't think there are related PRs to this issue in this release
  • Milestone https://github.com/quarkusio/quarkus/milestone/249?closed=1

[ThoSap]

@snazy

[ThoSap]

@geoand @zakkak sorry to tag you, do you have an idea why this happens?

[geoand]

cc @radcortez

[radcortez]

Might be related with #31166. At least there was a big change around how Config is used in Gradle in that PR. Needs further investigation.

[dagrammy]

@radcortez Thanks for your link! I looked I looked through the PR and came across an interesting section that I thought may have something to do with the problem.

https://github.com/quarkusio/quarkus/blob/main/devtools/gradle/gradle-application-plugin/src/main/java/io/quarkus/gradle/tasks/QuarkusTask.java#L96

This "very big hammer" (as the code is described in the comment) seems to be a bit too big. 😅

I commented out the code block...

 configMap.entrySet().stream().filter(e -> e.getKey().startsWith("quarkus."))
   .forEach(e -> forkOptions.systemProperty(e.getKey(), e.getValue()));

...and it worked!

As the comment describes, there is a reason for this code block. But this statement causes that no quarkus.* property can be set or overwritten in the way described above (quarkus.banner.enabled=${BANNER:false}) when building native with gradle.

@snazy Wdyt?

[maxandersen]

Good find. The Big hammer does seems to go against what the pr describes - that config sources should now be correct incl. using env variables.

Does feel Like its too Big a hammer.

[snazy]

Ouch.

I suspect, this "big hammer" needs to be restricted to only "override" the (build time) properties that are explicitly configured via the Gradle build (so System properties set for the Gradle build, Gradle project properties, properties on the Quarkus Gradle extension, and the properties "forced" from some tasks. That would leave other configs (like the "banner enabled" one) untouched.

[maxandersen]

Isn't env vars supposed to override any build file settings ?

[snazy]

Isn't env vars supposed to override any build file settings ?

The issue is that the config settings are (now) evaluated, so the currently effective value's used - not the value "expression".

[dagrammy]

Thank you @snazy for your input and evaluation.

If I understand it correctly, not all but only the following (-> YES/NO) configurations should be set.

see https://github.com/quarkusio/quarkus/blob/main/devtools/gradle/gradle-application-plugin/src/main/java/io/quarkus/gradle/tasks/EffectiveConfig.java#L53

// Effective "ordinals" for the config sources:
// (see also https://quarkus.io/guides/config-reference#configuration-sources)
// 600 -> forcedProperties -> YES
// 500 -> taskProperties -> YES?
// 400 -> System.getProperties() -> NO? (because already set?)
// 300 -> System.getenv() -> NO?  (because already present?)
// 290 -> quarkusBuildProperties -> YES
// 280 -> projectProperties -> YES
// 250 -> application.(properties|yaml|yml) (in classpath/source) -> NO
// 100 -> microprofile.properties (in classpath/source) -> NO

But what I still don't understand respectively haven't found out. Why does it only affect the native build?

[ThoSap]

If you find a fix for this issue, will there be a Quarkus 3.0.5.Final or 3.1.1.Final bugfix version?
Waiting until 3.2.0.Final releases on June 21st is probably not an option for some people.

[dagrammy]

After long hours over the past few days looking at the code and trying various approaches, here is a PR trying to solve the problem while maintaining the expected behaviour of the gradle build.

This PR changes/restricts which properties are passed to the gradle worker process as system properties.

Instead of all properties, only the following properties are passed now:

• quarkus.package.*
• quarkus.application.*
• quarkus.gradle-worker.*

These properties are important so the gradle worker processes uses the correct outpur directory, package type (jar, native image,...).

Keep in mind, that this only affects properties passed to the gradle worker process. To build the actual artifact (e.g. jar, native image) all properties are used.

This PR fixes #33321.
Locally all gradle unit & integration test are executed successfully.

I would appreciate feedback and review of the PR. @snazy @maxandersen @radcortez

[maxandersen]

Is this not different from how maven build are done?

If different. Is it expected ? Should maven be adjusted too to get consistency?

@aloubyansky

[snazy]

@maxandersen Maven and Grade differ here. The reason is that the Gradle build always defined a way to pass properties from "more sources" like the Gradle task configuration or Gradle project properties. Further the Gradle build has to respect that Quarkus build can happen in parallel - so using/modifying System properties and env vars isn't an option. That made it necessary to evaluate and re-assemble the properties for a particular Gradle build "again".
The issue here is that users can set any property, not just quarkus.* properties via e.g. the Gradle Quarkus tasks.

While I think that #33819 is good for now, the long term fix should probably be a bit different. I suspect, the EffectiveConfig approach is good to let the Quarkus Gradle plugin check "its own" settings.

• Ideally, all relevant settings should be passed to the build worker(s) - "relevant" means configurations, that come from a higher-priority config-source than microprofile/application.properties/yaml
• Pass "config expressions" as they are and not evaluate those "prematurely" (e.g. quarkus.enable.foo=${foo.enabled}) - not sure whether this would still be an issue with the above bullet point implemented (aka I don't know whether a system property quarkus.enable.foo=${foo.enabled} would be indirectly evaluate foo.enabled).

(Apologies for the late reply to some issues/PRs, been quite busy with "my main work stuff".)

[hartmut-co-uk]

Dear issue/PR authors, would you be so kind to verify if the changes made related to this issue have caused the workaround for arm64 native build to stop working?
#30545 (comment)

If so, would it be valid to create a PR adding "quarkus.native." to the list of whitelisted properties?

[dagrammy]

Hi @hartmut-co-uk,

I am in the original author of the mentioned PR/commit. It sounds plausible that the workaround no longer works for this reason, because the properties are no longer passed on to the gradle worker process. I see no harm in adding quarkus.native.* to the whitelist.

Maybe you could try to do a local Quarkus build with the following change, see if it works again and open a PR?

private static final List<String> WORKER_BUILD_FORK_OPTIONS = List.of("quarkus.package.",
        "quarkus.application.", "quarkus.gradle-worker.", "quarkus.native.");

[hartmut-co-uk]

Hi @hartmut-co-uk,

I am in the original author of the mentioned PR/commit. It sounds plausible that the workaround no longer works for this reason, because the properties are no longer passed on to the gradle worker process. I see no harm in adding quarkus.native.* to the whitelist.

Maybe you could try to do a local Quarkus build with the following change, see if it works again and open a PR?

private static final List<String> WORKER_BUILD_FORK_OPTIONS = List.of("quarkus.package.",
        "quarkus.application.", "quarkus.gradle-worker.", "quarkus.native.");

Update: #37021 seems to have fixed this. At least with quarkus 3.6.3 it's working as it did before (<3.2.0).

[dagrammy]

Well, it fixed your problem but reintroduced ours. 😉

Tbh, we have now simply replaced gradle with maven. Just kidding....although I had really thought about it for a moment.

But we have completely banned environment variable substitution in application.properties as this feature just seems too unstable in native mode.

Instead we replace complete properties as env var with QUARKUS_REST_SOME_CLIENT_SOME_VALUE_YADA_YADA_YADA.

[ThoSap]

@snazy @maxandersen @radcortez @gsmet @aloubyansky can you please reopen this issue, as #37021 reintroduced the issues from this original issue.
See the comment of @dagrammy above.

Related issues:
#37273
#37649

[radcortez]

Have you had a chance to try #37987? Thanks!

[dagrammy]

Sorry for the late reply.

I have tested with my reproducer (https://github.com/dagrammy/quarkus-env-reproducer-maven) and version 3.6.5 and it works again.

Thank you very much! 🙂 👍