Importing config files across modules

[edudar]

I have two modules in the same project, one by another:

A
|-- application.yaml
|-- application-dev.yaml

B
|-- base-application.yaml
|-- base-application-dev.yaml

Been using custom EnvironmentPostProcessor to load base- config into A but with SB 2.4 I wanted to give config.import a try. But there are a few issues arisen.

(1)
In A/application.yaml I've added the following:

spring:
  config:
    import: classpath:/base-application.yaml

In A/application-dev.yaml it's respectively:

spring:
  config:
    import: classpath:/base-application-dev.yaml

Trying to start A fails with Config data location 'classpath:/base-application-dev.yaml' does not exist. That's rather confusing considering that it does exist indeed and that classpath:/base-application.yaml is loaded just fine from the same location. To verify that, I made a typo in the file name and it failed for 'classpath:/bas-application.yaml' as expected.

Renaming base-application-dev.yaml to something profile-unrelated like some-file.yaml still fails with resource does not exist.

(2)
If I don't change A/application-dev.yaml at all, I do see that SB takes B/base-application-dev.yaml into account when loading classpath:/base-application.yaml and properly loads it. The application starts just fine with all properties. So clearly it does exist in (1). But now the ordering of property sources is not correct. Here's what it is from environment:

...
application-dev.yaml
base-application-dev.yaml
base-application.yaml
application.yaml
...

base-dev overrides dev but application overrides base-application.

[wilkinsona]

@edudar Can you please provide a minimal sample that reproduces the behaviour that you have described?

[edudar]

Published to edudar/sb26210

Right now, classpath:/base-dev.yml is commented out in application-dev.yml. App starts and prints out:

INFO 42529 --- [           main] org.example.sb26210.app.App              : Config resource 'class path resource [application-dev.yml]' via location 'optional:classpath:/'
INFO 42529 --- [           main] org.example.sb26210.app.App              : Config resource 'class path resource [base-dev.yml]' via location 'classpath:/base.yml'
INFO 42529 --- [           main] org.example.sb26210.app.App              : Config resource 'class path resource [base.yml]' via location 'classpath:/base.yml'
INFO 42529 --- [           main] org.example.sb26210.app.App              : Config resource 'class path resource [application.yml]' via location 'optional:classpath:/'

If you uncomment classpath:/base-dev.yml, App would fail with

[main] DEBUG org.springframework.boot.diagnostics.LoggingFailureAnalysisReporter - Application failed to start due to an exception
org.springframework.boot.context.config.ConfigDataLocationNotFoundException: Config data location 'classpath:/base-dev.yml' cannot be found

[scottfrederick]

Thanks very much for the sample.

The multi-module aspect doesn't appear to be important here - I can also reproduce the issue with all four properties files in the same module. The key is to have the imports for base.yml and base-dev.yml while the dev profile is active. The import: classpath:/base-dev.yml isn't necessary, as is shown in your sample output above the profile-specific file is getting included because of the profile activation without the import. But the fact that you're getting an error, and that the error message is misleading, is a problem.

[edudar]

@scottfrederick shouldn't base.yml and application.yml also switch places in property sources ordering? So that the order is app-dev -> base-dev -> app -> base? This way when one imports an external config, it'd be possible to override some values in place.

[scottfrederick]

As stated in the documentation, profile-specific properties files override non-profile-specific properties files and properties from imported files override properties from the file that contains the import.

Without the import: classpath:/base-dev.yml, this appears to be working correctly in your sample, since the properties are resolved to these values when the dev profile is active:

2021-04-16 13:23:03.003  INFO 98563 --- [           main] org.example.sb26210.app.App              : a = app-dev
2021-04-16 13:23:03.005  INFO 98563 --- [           main] org.example.sb26210.app.App              : b = app
2021-04-16 13:23:03.005  INFO 98563 --- [           main] org.example.sb26210.app.App              : c = base-dev
2021-04-16 13:23:03.005  INFO 98563 --- [           main] org.example.sb26210.app.App              : d = base

With the import: classpath:/base-dev.yml in application-dev.yml, the code is a bit confused about whether it should consider base-dev.yml because it is the profile-specific form of base.yml that has already been imported or because it is explicitly imported.

[edudar]

@scottfrederick

properties from imported files override properties from the file that contains the import

I think there are two different views here.

(2.1) I do understand where the doc is coming from. If I have an external config like a configtree or consul or similar deployed in production/staging/uat/younameit, I do want values from there to override my bundled properties from a jar. Essentially, jar provides some predefined properties that can be overridden in production.

(2.2) But the example in this ticket is pretty much the opposite. That's why I named files base and app. If we move away from abstract a-b-c-d, let's imagine that base module contains a configuration for HTTP client and base.yml contains all sorts of timeouts and whatnot, headers, user agents, etc. Now, I'm thinking that as general guidance, I'd set read timeout to 1000ms: base.yml -> timeouts.read: 1000ms. At the same time in dev it can be longer, right, so base-dev.yml -> timeouts.read: 5000ms. Following this, I'm developing a specific application that won't tolerate such extensive timeouts. So my idea would be to import base.yml to get all properties but I override read timeout, so app.yml -> timeouts.read: 100ms. And similar to the base, when in dev, timeouts may be extended, so app-dev.yml -> timeouts.read: 500ms.

When I run my app in dev, I will get 500ms timeouts as it's set in config, because app-dev.yml takes precedence over all other files right now, including base-dev.yml. BUT. If I run my app without a profile, I'll get 1000ms timeout, because it's coming from base.yml and it overrides my 100ms setting for the app.yml. My point is that I'd probably expect the timeout to be 100ms, not 1000ms. However, if we stick to what (2.1) says, then running in dev should probably give me 5000ms timeout coming from base-dev.yml instead of 500ms from app-dev.yml.

[scottfrederick]

As you said initially, there are a few issues here.

One is a bug that's causing a confusing error message unrelated to modules and precedence of property loading. I've now split that out into #26147 so we can address that bug.

The rest of this discussion has to do with a specific problem you are trying to solve and whether your approach will work given the design of Spring Boot's property loading. Let's continue that discussion here.