Split configuration

[fipwmaqzufheoxq92ebc]

I'm using git to manage configuration files and I don't want to commit passwords to git.

I would like to be able to split up the configuration in multiple files or load some config from another location. That way I could commit the not-secret-configuration only.

Some ideas:

• use values from environment variables as defaults
• replace values which start by e.g. a $-Sign with values from environment
• Allow multiple config files as arguments and merge them
• add for some/all options the possibility to load it from a file (e.g. an option password_from_file)

[adrian-vlad]

An example for some of the ideas is home assistant: https://www.home-assistant.io/docs/configuration/splitting_configuration/

[flyte]

Ah yes, this is a good idea. I think that perhaps having a placeholder type thing would be good:

mqtt:
  host: test.mosquitto.org
  user: {{ env("MQTT_USER", "my_default_user") }}

This is the same syntax I use for https://github.com/flyte/confp (which could be used for this in the meantime), and it means we can just run the config file through Jinja2 first, then load it as normal. I'll have a look at confp to see if it's possible to just utilise that as a dependency, since it might make things more flexible by being able to load values not just from env vars, but from etcd, redis, S3 etc. as well (or anywhere really, by writing a new module for confp).

[flyte]

As far as splitting up config files goes, I'm not really convinced about that yet. I know some people use this software for much larger projects than I have, but we don't have comprehensive automation scripts and such like Home Assistant does, so I can't imagine config files getting too complicated to necessitate splitting.

Happy to be proven otherwise though.

[flyte]

Another benefit of using Jinja2 is that loops and conditionals can be used. That might make config files much less tedious to write, for example:

mqtt:
  host: test.mosquitto.org
  user: myuser
  password: {{ env("MQTT_PASS") }}

gpio_modules:
  - name: pcf
    module: pcf8574

digital_inputs:
{% for i in range(8) %}
  - name: pcf{{ i }}
    pin: {{ i }}
    module: pcf
{% endfor %}

[justinmusgrove]

Hey @flyte wondering if you had a chance to document how you would pass environment variables via docker container? I would gladly submit a PR if I knew what looks like. Thoughts?

docker-compose.yml

mqtt-io:
  container_name: mqtt-io
  image: flyte/mqtt-io:latest
  network_mode: host
  privileged: true
  restart: always
  volumes:
    - ./mqtt-io/config.yml:/config.yml
    - /etc/localtime:/etc/localtime:ro
  depends_on: 
    - mqtt
  environment:
    TZ: ${TIMEZONE}
    MQTT_GPIO_CONFIG_MQTT_HOST: ${HOST_IP}
    MQTT_GPIO_CONFIG_MQTT_USER: ${MQTT_USERNAME}
    MQTT_GPIO_CONFIG_MQTT_PASSWORD: ${MQTT_PASSWORD} 

config.yml

mqtt:
  host: <INSERT VARIABLE>
  topic_prefix: gpio
  user: <INSERT VARIABLE>
  password: <INSERT VARIABLE>
  ha_discovery:
    enabled: yes

[goshansp]

Thanks to @pmgledhill102 I was able to render environmental variables using the following configuration on Fedora 36 IoT / Rpi3.

config.yml

mqtt:
  host: <hostname>
  user: <username>
  password: {{ my_env('MQTT_PASSWORD') }}
...

render.yml

backends:
  my_env:
     type: env

testing it

MQTT_PASSWORD=<password> python3 -m mqtt_io --render render.yaml config.yaml