[Fleet] Support input-level vars & templates

[jen-huang]

Summary

Resolves #82599. This PR enables input-level agent YAML templates, similar to stream-level templates. Input-level vars will be passed to the specified template in package info. The input can have streams or no streams. Here is an example showing the package info containing template_path field on the input, the contents of that template, and the final agent yaml generated by Kibana with this change:

(for cleaner illustration purposes, this input does not have streams)

Package info

  "policy_templates": [
    {
      "name": "input_level_templates",
      "title": "Input level templates",
      "description": "Input with input-level template to use input-level vars with",
      "inputs": [
        {
          "type": "logs",
          "vars": [
            {
              "name": "paths",
              "type": "text",
              "title": "Paths",
              "multi": true,
              "required": true,
              "show_user": false,
              "default": [
                "/foo/bar",
                "/test/path"
              ]
            },
            {
              "name": "host",
              "type": "text",
              "title": "Host",
              "multi": false,
              "required": true,
              "show_user": false
            }
          ],
          "title": "Collect logs",
          "template_path": "template.yml.hbs"
        }
      ],
      "multiple": true
    }
  ]

/agent/input/template.yml.hbs

fixedField: some value
paths:
{{#each paths}}
  - {{this}}
{{/each}}
host: {{host}}

Agent yaml

inputs:
  - id: b42d8fc8-6fb7-4c5a-96bb-c4fa3120b6b8
    name: input_level_templates-1
    revision: 1
    type: logs
    use_output: default
    meta:
      package:
        name: input_level_templates
        version: 1.0.0
    data_stream:
      namespace: default
    fixedField: some value
    paths:
      - /foo/bar
      - /test/path
    host: localhost

Here's what the integration policy form UI looks like for the same. Note that there is only a top-level Settings section for the input-level vars, and no stream toggles are displayed underneath it:

Testing

1. Run Kibana using a local package registry running latest main or this PR branch
2. Find and add the test integration called "Input level templates"
3. The UI and agent policy yaml output should reflect above screenshots

[elasticmachine]

Pinging @elastic/ingest-management (Team:Ingest Management)

[jen-huang]

@elasticmachine merge upstream

[ruflin]

@jen-huang Can we skip the streams part?

[nchaulet]

Looks good to me 👍 and work as expected

[jen-huang]

@nchaulet Thanks for testing!

@ruflin 493e54a Sure, removed empty streams: [] from agent yaml when there are no streams.

[kibanamachine]

💚 Build Succeeded

• continuous-integration/kibana-ci/pull-request
• Commit: ac46477

Metrics [docs]

Async chunks

Total size of all lazy-loaded chunks that will be downloaded as the user navigates the app

id	before	after	diff
fleet	1.1MB	1.1MB	+558.0B

Page load bundle

Size of the bundles that are downloaded on every page load. Target size is below 100kb

id	before	after	diff
fleet	374.7KB	374.8KB	+93.0B

Saved Objects .kibana field count

Every field in each saved object type adds overhead to Elasticsearch. Kibana needs to keep the total field count below Elasticsearch's default limit of 1000 fields. Only specify field mappings for the fields you wish to search on or query. See https://www.elastic.co/guide/en/kibana/master/development-plugin-saved-objects.html#_mappings

id	before	after	diff
ingest-package-policies	30	31	+1

History

• 💔 Build #90071 failed 493e54a
• 💚 Build #89660 succeeded 30459b9
• 💚 Build #89140 succeeded 62effd3

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

[jalvz]

works great, thanks!