How to generate configs (with iterate over inventory)

[knutov]

What is the best/supposed way to generate configs for things like prometheus with iteration over hosts from inventory?

Is any builtin templating (go text/template?) planned?

It's possible to use external templating like https://github.com/kblomqvist/yasha (jinja2), but it's unclear how to connect it with data from inventory and exported variables

[umputun]

I don't understand the question. What exactly "prometheus with iteration over hosts from inventory" means? Can you explain the use case in more details, i.e., what is the problem you are trying to resolve?

I guess this is about the playbook, what does something with pormetheus, but it is unclear to me what exactly it supposes to do (deploy promethesus? Use promethesus API to build inventory. React on prometheus's event in some way?)

The "iteration over inventory" part is also confusing. In order to run a playbook for multiple hosts from inventory you can pass/set inventory group or tag, no need to generate anything for such iteration, except the inventory.

[knutov]

Ansible is used a lot to generate configs with it's jinjia templating. Example block form prometheus config:

scrape_configs:
- job_name: prometheus
  honor_timestamps: true
  scrape_interval: 15s
  scrape_timeout: 10s
  metrics_path: /metrics
  scheme: http
  follow_redirects: true
  enable_http2: true
  static_configs:
  - targets: # <------- this part should be generated
    - server1.domain.tld:9090
    - server2.domain.tld:9090
    - ...

Common practice is generate targets block

• via merging yaml - like https://github.com/cloudalchemy/ansible-prometheus/blob/e4393788ac1d8e6608777c9edb6b711b933d9c39/defaults/main.yml#LL72C10-L72C10
• via jinja template (minimal example converted from https://stackoverflow.com/questions/75287186/jinja2-ansible-loop-over-lines-in-several-text-files :

          {% for host in groups['production'] %}
          - {{ url }}:9090
          {% endfor %}

It's ok to use precreated config on the local side, but the list of servers to generate the list should be taken from inventory. How to deal with those scenarios with spot?

[umputun]

I understand what it does but need to understand why and how this would be useful in spot. To me, the list of hosts is the property of an inventory file rather than something that should be generated.

If we need to run a playbook against all the servers in "production" group or on all servers tagged as "production" we can set target: production (or targets: [production, ....] ) directly in the playbook. If we want to run the same playbook against different target groups/tags/names/hosts, we pass those as -t / --target parameter.

In your example, all the template does is add the port to all the hosts in that "production" group. I struggle to see what the reason for doing it; in my view, this is something that inventory should be handling, i.e. if a custom (non-22) ssh port should be set, this has to be in the inventory or passed in as a direct list of targets.

Theoretically, support of the templates can be added, however, this is a pretty complex thing from the user's point of view, and we need very compelling reasons to add it to the spot. The fact that ansible does templates is not a good enough reason by itself, our goal is not to repeat all the ansible's features.

[umputun]

The other interpretation of this request, and I think this is what you meant - you want spot to generate configs for third-party systems based on the list from the inventory, i.e. using spot as some sort of template runner. If this is the case, I don't think this should be a part of the spot core functionality. Well, maybe we can add "generate targets list" command, so one can get back targets in some format, for example, json, and process them in whatever way they like.

[knutov]

yes, "generate targets list" can solve the problem with external templating systems, so the usecase can be something like:

1. call spot to run something and return generted target list (based on inventory, --target and playbook)
2. generate configs for targets with external tool
3. call spot again with the same inventory and --target and deploy generated files

[umputun]

You can do it without running the spot twice. It can be done with a "local" command invoking the spot generator, i.e.

task:
 - name: prepare config for the target host
    command: sport --gen --gen.tempate=blah.tmpl --get.output=file.conf --target=$SPOT_REMOTE_NAME
    options: {local: true}
 - name: deploy generate config
    copy: {src: file.conf, dst: /srv/file.conf}

[umputun]

see #105