Importing config from dependecy

[grundrauschen]

Hello,
I know that if the config.exs imports a config from a submodule, conform imports the settings from the imported config and adds them to the schema when using mix conform.new. Never the less one would have to recreate the schema every time a dependency changes its settings or translations.
As a feature request I would like to ask for a function to import configurations of dependencies to add them into the configuration of the main applications.

As an example of how it should be used:
deps/dependency/config/dependency.schema.exs

[
  mappings: [
    "dependency.setting": [
      doc: "Setting of dependency",
      datatype: :binary,
      default: "foo"
  ],
  translations: []
]

config/myapp.schema.exs

[
  import_settings: [
    :dependency
  ],
  mappings: [
    "myapp.setting": [
      doc: "Setting of myapp",
      datatype: :binary,
      default: "bar"
  ],
  translations: []
]

using mix conform.configure results in

# Setting of dependency
dependency.setting = "foo"

# Setting of myapp
myapp.setting = "bar"

The schemas must also be imported somehow of course to use the translations.

Sincerely,
Tobias

[bitwalker]

@grundrauschen I'm not sure I follow. A couple things:

• You do not need to recreate your schema every time a dependency changes, you just need to add/update the settings you want to expose, or remove them as appropriate. Additionally, settings in dependencies that aren't in the schema will still be present in the running config via Application.get_env.
• conform.new is really intended to just get you started, it's not meant to generate the schema you intend to use in production. It can't be very intelligent about types, and doesn't generate very useful documentation comments. You should run it once, and then manually edit it from that point on.

Importing schemas is something I would be interested in supporting, though I would like to have a real world example to base it on. Could you perhaps provide an example of your specific use case?

[grundrauschen]

@bitwalker: I try to explain it further to make it more specific.

The settings of a dependency, which are defined in the dependency's config.exs are of course available through Application.get_env, but the one I might expose to the user to change are not automatically mergeable into the main applications one as far as I know of.
When I add them manually to my config, I of course have also to take care, that my translations are matching the one of the dependency.

Because the project I am working on is closed-source, I can't send you a link right now, but I try to explain our use case.

We have to set specific settings for the environment for a dependency like IP-addresses to listen on log levels and so forth.
Imagine we have a logging service (just as an example, imagine the other service could run on its own) as an dependency we want to use, and have to hand it a few parameters, like the mentioned IP-addresses.
The logging service now uses conform to provide an easy to use configuration to Ops to set these settings including translations to evaluate these settings and save them properly.

The main application now wants to use these service and has some settings to be set by the operator itself, but the operator should also be able to set the IP-address of the logging service, the main application depends on, in the config file of the application, too.

Because the logging service might change the translations or add new settings to be configured by the
Op in the future, I would love to not add the settings to my schema manually, but just add them as an import in the schema in the main app as described above.

So when a new version of the magic logging service comes out, I can just make a mix deps.update and mix release (or other step to use conform) and have the settings of my main app as well as the ones of the logging service in the main_programm.conf to be used by the op.

I hope my explanation was somehow enlightened.

[bitwalker]

@grundrauschen Yep makes sense, it should certainly be possible to import mappings and translations from multiple schema files. So my thought is that the syntax would be something like this:

[ import: [:some_app],
  extends: [:some_dep], 
  mappings: [...],
  translations: [...] ]

extends would be the new field to define dependencies with conform schemas which this schema will extend. I call this extending rather than importing, since definitions in this file will override those in the imported schema. import already exists as a means of importing modules/functions which can then be consumed by translations in this schema.

[bitwalker]

@grundrauschen These (and many other) changes have been implemented in the v1 branch. Could you give it a look and test with it? I'm trying to get as much feedback as possible before pushing 1.0.

[grundrauschen]

hello, this is great news.
I tried it out with one project I am working on and conform tells me, that the schema extends the other app, but that it does not find it. Although my_dep has a schema in 'deps/my_dep/config/my_dep.schema.exs'. Where does conform look?

[bitwalker]

@grundrauschen Probably a bug, could you describe your project layout a bit more? Also when you are seeing the error specifically?

I'm assuming your schema file looks something like the following, correct?:

[ extends: [:my_dep],
  imports: [],
  mappings: [...],
  transforms: [...] ]

Given the above, it will attempt to locate the my_dep application, and fetch the schema from path/to/my_dep/config/my_dep.schema.exs. If this isn't working, then something is probably broken with the way my_dep is being located, or it's unable to be found.

[bitwalker]

@grundrauschen Found the problem(s), and pushed some fixes to the v1 branch. Could you pull and run your tests again? It's very much appreciated!

[grundrauschen]

Hello @bitwalker, I tried the newest HEAD of the branch and mix.configure as well as mix.effective seem to create the correct output.
I am really glad for you work here.

When I try to use it with exrm (master branch) I get the following error message, but assume it is because exrm is not working on the v1 branch at the moment, right?

Generated my_dep app
==> Building release with MIX_ENV=dev.
==> Generating relx configuration...
==> Generating sys.config...
==> Generating boot script...
==> Performing protocol consolidation...
==> Conform: Loading schema...
==> Failed to execute before_release hook for Elixir.ReleaseManager.Plugin.Conform!
** (UndefinedFunctionError) undefined function: Conform.Schema.read!/1
    Conform.Schema.read!(:my_app)
    lib/exrm/plugins/conform.ex:26: ReleaseManager.Plugin.Conform.before_release/1
    lib/mix/tasks/release.ex:240: anonymous fn/2 in Mix.Tasks.Release.execute_before_hooks/1
    (elixir) lib/enum.ex:1385: Enum."-reduce/3-lists^foldl/2-0-"/3
    lib/mix/tasks/release.ex:69: Mix.Tasks.Release.do_run/1
    (mix) lib/mix/cli.ex:54: Mix.CLI.run_task/2

In the mix.exs of the project I use the following as dependencies:

[    ...,
     {:exrm, github: "bitwalker/exrm", override: true},
     {:conform, github: "bitwalker/conform", branch: "v1", override: true}]

[bitwalker]

@grundrauschen Sorry about that, I was still working on deciding how to deprecate the conform dependency in exrm. If you use the following in your deps it should work as expected:

[ ...,
  {:exrm, github: "bitwalker/exrm", branch: "v1"},
  {:conform, github: "bitwalker/conform", branch: "v1} ]

The change in exrm here is to move the conform plugin to conform itself, and remove the direct dependency within exrm on conform. This way use of conform is completely optional. exrm still contains some conform specific code in the boot script, but it works whether conform is present or not.

[bitwalker]

@grundrauschen ping :)

[grundrauschen]

@bitwalker: Sorry for not responing: I am in another team at the moment but will have a look at that change in the next days

[bitwalker]

@grundrauschen No worries :)

[matehat]

@bitwalker I was trying to use the :extends key in my project but I fail to see how that can be useful because, in a release, conform does not seem to pick up the extended schema at all.

It seems the current behavior is that, when building the release, an initial sys.config gets produced and while doing so it does pick up the extended schemas properly. But when starting a node after that, these schemas are not picked up, and the configuration that would be understood using these schemas are not picked up at all from the .conf file.

[matehat]

Is that the expected behavior?

[bitwalker]

@matehat We probably need to ensure extended schemas are archived like imports are, so that they can be extracted on the target system and consumed. Since the config folder is not present in a release, that probably explains why it's not working as expected. I thought I recall extensions being archived, but it's entirely possible I overlooked that - extends was a feature request which I implemented, but haven't had a need for myself.

[bitwalker]

This appears to no longer be an issue with distillery and conform (latest versions).