Adding XD_CONFIG_LOCATION to classpath if defined

[mohdaliiqbal]

Currently the custom config is not on classpath. Please note that the default config folder will not be available on classpath anymore, i thought it was reasonable as it may cause conflicts with default empty settings.

[ericbottard]

This location should not be added to the classpath, as it messes up the ModuleRegistry, because of https://github.com/spring-projects/spring-xd/blob/8f370a1f32922f77cc49850906fbcbddedcaf8a2/spring-xd-dirt/src/main/resources/META-INF/spring-xd/internal/repositories.xml#L24-L26
Indeed, that directory has the same modules/<type>/<name> layout as a module registry, and so files there are wrongly picked.

Instead, could you explain which files you would like to see considered as properties are currently not seen, and we could fix the problem there.

Also, an alternative would be to remove the registry defined in the lines at pointed at above. This has been a longstanding issue, but my understanding is that it is/was needed by the YARN deployment model. Maybe @trisberg / @jvalkeal can chime in.

[jvalkeal]

@ericbottard we've relied on this functionality that we can place hadoop's site xml files into a default config directory where hadoop's Configuration class will pick those up because they are on a classpath. Some users want to keep XD installation and its configs separate because it will help upgrading versions. Naturally you'd want to keep those site xml files in a same place if custom config location is used. Thus this is a reason we thought it was just a bug that custom location is not added to a classpath.

Doesn't this work if you copy everything over from default config dir including modules dir structure?

What about adding both, default config dir and custom config dir(if it's defined) into a classpath?

[ericbottard]

The problem is really that that directory looks like a module registry. If you can confirm that https://github.com/spring-projects/spring-xd/blob/8f370a1f32922f77cc49850906fbcbddedcaf8a2/spring-xd-dirt/src/main/resources/META-INF/spring-xd/internal/repositories.xml#L24-L26 can be removed (it is not needed for non-YARN runtime for sure, it may be still needed for YARN) then I'm fine with adding it to the CP

[mohdaliiqbal]

Dont we have exact same structure in the out of the box xd/config/modules//

If i'm customizing my config dir - i am copying everything that comes OOB and modifying them - updating logback.groovy, hadoop.properties, custom-servers.yml and everything that comes out of the box. Except for the yml file nothing is currently picked from the customized config folder.

I'm not familiar with YARN runtime so yes I wouldn't know what will be messed up there. Can we check the environment in the start up script and have it added only if non-YARN runtime.

[jvalkeal]

@ericbottard good point for checking yarn runtime and how we setup XD in ambari. I'll play with those a little.

[ericbottard]

I should actually point out that $APP_HOME/config (or its custom equivalent) should be fine, it's $APP_HOME that was problematic.
Yet, if we could get rid of that registry line, it would be beneficial

[trisberg]

Closing this since it is so old