[Fleet] Config in "Configs" tab in integrations app is malformed

[flash1293]

Kibana version: main

When navigating to the "Configs" tab in the UI, the config should represent the default config and work mostly out of the box when copy/pasted into an elastic-agent.yml.

However, the "Configs" tab is showing only a flat list of inputs without nested datastreams, e.g. for nginx:

  - id: logfile-nginx.access
    type: logfile
    data_stream:
      dataset: nginx.access
      type: logs
    ignore_older: 72h
    paths:
      - /var/log/nginx/access.log*
    tags:
      - nginx-access
    exclude_files:
      - .gz$
    processors:
      - add_locale: null

When creating an agent policy for nginx, then selecting the standalone tab in the flyout, it shows the following structure instead:

  - id: logfile-nginx-a6c2c187-8cb7-4cd2-93de-779d23aa827e
    name: nginx-1
    revision: 1
    type: logfile
    use_output: default
    meta:
      package:
        name: nginx
        version: 1.20.0
    data_stream:
      namespace: default
    streams:
      - id: logfile-nginx.access-a6c2c187-8cb7-4cd2-93de-779d23aa827e
        data_stream:
          dataset: nginx.access
          type: logs
        ignore_older: 72h
        paths:
          - /var/log/nginx/access.log*
        tags:
          - nginx-access
        exclude_files:
          - .gz$
        processors:
          - add_locale: null

When using the flat config from the "Configs" tab, the agent doesn't ship any data while the nested config specifying streams works just fine.

[elasticmachine]

Pinging @elastic/fleet (Team:Fleet)

[flash1293]

cc @kpollich @criamico

[kpollich]

Hey @flash1293 - we are starting to take a look at this. Can you clarify whether you were following the notes on the config tab page and pasting these into .yml files within the inputs.d subdirectory of the agent directory, or if you're expecting there to be a fully rendered agent policy on the tab? I think right now we just generate what would go in the inputs section or the inputs.d folder, but I want to clarify the expectations here. @criamico is also taking a look at the existing behavior to confirm.

[flash1293]

@kpollich I copy-pasted the contents of the "Config" tab into the "inputs" section of my elastic-agent.yml (without the first line)

I configured the output separately / on my own.

[criamico]

I traced back the source of the issue to this line. As @flash1293 noticed, we are flattening the streams.

I don't recall anymore why is that, but that's the main difference between the inputs in Configs vs generated agent policy. I'll have a fix for it and I'll make sure to backport as well.

[criamico]

Example of ningx inputs generated in Configs tab without the flattening:

inputs:
  - id: logfile-nginx
    streams:
      - id: logfile-nginx.access
        data_stream:
          dataset: nginx.access
          type: logs
        ignore_older: 72h
        paths:
          - /var/log/nginx/access.log*
        tags:
          - nginx-access
        exclude_files:
          - .gz$
        processors:
          - add_locale: null
      - id: logfile-nginx.error
        data_stream:
          dataset: nginx.error
          type: logs
        ignore_older: 72h
        paths:
          - /var/log/nginx/error.log*
        tags:
          - nginx-error
        exclude_files:
          - .gz$
        multiline:
          pattern: '^\d{4}\/\d{2}\/\d{2} '
          negate: true
          match: after
        processors:
          - add_locale: null
  - id: httpjson-nginx
    streams:
      - id: httpjson-nginx.access
        data_stream:
          dataset: nginx.access
          type: logs
        config_version: 2
        interval: 10s
        cursor:
          index_earliest:
            value: '[[.last_event.result.max_indextime]]'
        request.url: 'https://server.example.com:8089/services/search/jobs/export'
        request.ssl: null
        request.method: POST
        request.transforms:
          - set:
              target: url.params.search
              value: >-
                search sourcetype=nginx:plus:access | streamstats
                max(_indextime) AS max_indextime
          - set:
              target: url.params.output_mode
              value: json
          - set:
              target: url.params.index_earliest
              value: '[[ .cursor.index_earliest ]]'
              default: '[[(now (parseDuration "-10s")).Unix]]'
          - set:
              target: url.params.index_latest
              value: '[[(now).Unix]]'
          - set:
              target: header.Content-Type
              value: application/x-www-form-urlencoded
        response.decode_as: application/x-ndjson
        response.split:
          type: string
          target: body.result._raw
          delimiter: |+
        tags:
          - forwarded
          - nginx-access
        publisher_pipeline.disable_host: true
      - id: httpjson-nginx.error
        data_stream:
          dataset: nginx.error
          type: logs
        config_version: 2
        interval: 10s
        cursor:
          index_earliest:
            value: '[[.last_event.result.max_indextime]]'
        request.url: 'https://server.example.com:8089/services/search/jobs/export'
        request.ssl: null
        request.method: POST
        request.transforms:
          - set:
              target: url.params.search
              value: >-
                search sourcetype=nginx:plus:error | streamstats max(_indextime)
                AS max_indextime
          - set:
              target: url.params.output_mode
              value: json
          - set:
              target: url.params.index_earliest
              value: '[[ .cursor.index_earliest ]]'
              default: '[[(now (parseDuration "-10s")).Unix]]'
          - set:
              target: url.params.index_latest
              value: '[[(now).Unix]]'
          - set:
              target: header.Content-Type
              value: application/x-www-form-urlencoded
        response.decode_as: application/x-ndjson
        response.split:
          type: string
          target: body.result._raw
          delimiter: |+
        tags:
          - forwarded
          - nginx-error
        publisher_pipeline.disable_host: true
  - id: nginx/metrics-nginx
    streams:
      - id: nginx/metrics-nginx.stubstatus
        data_stream:
          dataset: nginx.stubstatus
          type: metrics
        metricsets:
          - stubstatus
        hosts:
          - 'http://127.0.0.1:80'
        tags:
          - nginx-stubstatus
        period: 10s
        server_status_path: /nginx_status

And this is the inputs section in the generated agent policy:

inputs:
  - id: logfile-nginx-2a30bd3c-7a6c-469a-8ad3-f8aa11f2392b
    name: nginx-2
    revision: 1
    type: logfile
    use_output: default
    meta:
      package:
        name: nginx
        version: 1.20.0
    data_stream:
      namespace: default
    package_policy_id: 2a30bd3c-7a6c-469a-8ad3-f8aa11f2392b
    streams:
      - id: logfile-nginx.access-2a30bd3c-7a6c-469a-8ad3-f8aa11f2392b
        data_stream:
          dataset: nginx.access
          type: logs
        ignore_older: 72h
        paths:
          - /var/log/nginx/access.log*
        tags:
          - nginx-access
        exclude_files:
          - .gz$
        processors:
          - add_locale: null
      - id: logfile-nginx.error-2a30bd3c-7a6c-469a-8ad3-f8aa11f2392b
        data_stream:
          dataset: nginx.error
          type: logs
        ignore_older: 72h
        paths:
          - /var/log/nginx/error.log*
        tags:
          - nginx-error
        exclude_files:
          - .gz$
        multiline:
          pattern: '^\d{4}\/\d{2}\/\d{2} '
          negate: true
          match: after
        processors:
          - add_locale: null
  - id: httpjson-nginx-2a30bd3c-7a6c-469a-8ad3-f8aa11f2392b
    name: nginx-2
    revision: 1
    type: httpjson
    use_output: default
    meta:
      package:
        name: nginx
        version: 1.20.0
    data_stream:
      namespace: default
    package_policy_id: 2a30bd3c-7a6c-469a-8ad3-f8aa11f2392b
    streams:
      - id: httpjson-nginx.access-2a30bd3c-7a6c-469a-8ad3-f8aa11f2392b
        data_stream:
          dataset: nginx.access
          type: logs
        config_version: 2
        interval: 10s
        cursor:
          index_earliest:
            value: '[[.last_event.result.max_indextime]]'
        request.url: 'https://server.example.com:8089/services/search/jobs/export'
        request.ssl: null
        request.method: POST
        request.transforms:
          - set:
              target: url.params.search
              value: >-
                search sourcetype=nginx:plus:access | streamstats
                max(_indextime) AS max_indextime
          - set:
              target: url.params.output_mode
              value: json
          - set:
              target: url.params.index_earliest
              value: '[[ .cursor.index_earliest ]]'
              default: '[[(now (parseDuration "-10s")).Unix]]'
          - set:
              target: url.params.index_latest
              value: '[[(now).Unix]]'
          - set:
              target: header.Content-Type
              value: application/x-www-form-urlencoded
        response.decode_as: application/x-ndjson
        response.split:
          type: string
          target: body.result._raw
          delimiter: |+
        tags:
          - forwarded
          - nginx-access
        publisher_pipeline.disable_host: true
      - id: httpjson-nginx.error-2a30bd3c-7a6c-469a-8ad3-f8aa11f2392b
        data_stream:
          dataset: nginx.error
          type: logs
        config_version: 2
        interval: 10s
        cursor:
          index_earliest:
            value: '[[.last_event.result.max_indextime]]'
        request.url: 'https://server.example.com:8089/services/search/jobs/export'
        request.ssl: null
        request.method: POST
        request.transforms:
          - set:
              target: url.params.search
              value: >-
                search sourcetype=nginx:plus:error | streamstats max(_indextime)
                AS max_indextime
          - set:
              target: url.params.output_mode
              value: json
          - set:
              target: url.params.index_earliest
              value: '[[ .cursor.index_earliest ]]'
              default: '[[(now (parseDuration "-10s")).Unix]]'
          - set:
              target: url.params.index_latest
              value: '[[(now).Unix]]'
          - set:
              target: header.Content-Type
              value: application/x-www-form-urlencoded
        response.decode_as: application/x-ndjson
        response.split:
          type: string
          target: body.result._raw
          delimiter: |+
        tags:
          - forwarded
          - nginx-error
        publisher_pipeline.disable_host: true
  - id: nginx/metrics-nginx-2a30bd3c-7a6c-469a-8ad3-f8aa11f2392b
    name: nginx-2
    revision: 1
    type: nginx/metrics
    use_output: default
    meta:
      package:
        name: nginx
        version: 1.20.0
    data_stream:
      namespace: default
    package_policy_id: 2a30bd3c-7a6c-469a-8ad3-f8aa11f2392b
    streams:
      - id: nginx/metrics-nginx.stubstatus-2a30bd3c-7a6c-469a-8ad3-f8aa11f2392b
        data_stream:
          dataset: nginx.stubstatus
          type: metrics
        metricsets:
          - stubstatus
        hosts:
          - 'http://127.0.0.1:80'
        tags:
          - nginx-stubstatus
        period: 10s
        server_status_path: /nginx_status

Diff:

The original data structure is now preserved, the only difference between the two is that the following parameters are omitted, as they are specific to an installed package policy:
name, revision, type, use_output, data_stream, use_output

[kpollich]

Thanks for the detailed explanation, @criamico - the results look great to me. 🎉

[flash1293]

Thanks for taking care of it, @criamico !

[flash1293]

@kpollich @criamico I just realized there is still a problem with the generated config - now it has the streams, but there is no type parameter for the input which breaks the config as far as I understand.

E.g. the example above, it still needs the type: logfile for the first logfile-nginx input definition.

Is there a way we can test this e2e as it has been proven to be a brittle piece?

[criamico]

Hi @flash1293, I opened a PR to fix the missing type fields. @kpollich do we have any e2e tests for standalone agents? Maybe we could ask the QA team to include this one in their test cases?

[kpollich]

@kpollich do we have any e2e tests for standalone agents? Maybe we could ask the QA team to include this one in their test cases?

No, not really. I think the best thing to do here would probably be a regression test that the QA team can run. @amolnater-qasource what do you think of this as a test case?

1. Navigate to Integrations app in Kibana -> System integration -> "Config" tab
2. Copy config provided into standalone agent's inputs.d directory as inputs.yml
3. Start standalone agent
4. Verify that logs and metrics are collected

@criamico @flash1293 - Does that match how we've been testing this feature manually?

It could be interesting to create an E2E test that calls this Fleet API to retrieve the inputs config and tests that a standalone agent ships data as expected with that config, but I think we'd need some help from the agent team to put that together.

Elastic Agent has an E2E test suite that runs a live Kibana server we could use for this purpose. @cmacknz do you think it make sense to test this Kibana API in that test suite? The UI & Integrations team could contribute the test if you think it's sensible.

[cmacknz]

Elastic Agent has an E2E test suite that runs a live Kibana server we could use for this purpose. @cmacknz do you think it make sense to test this Kibana API in that test suite? The UI & Integrations team could contribute the test if you think it's sensible.

That test framework could do that, but it'd be running on a schedule or triggered based on changes to agent itself which would make correlating back to problems in Kibana difficult. The agent test framework was developed to test agent, that is it installs the agent directly in isolated VMs across many different OS flavours. That is a bit heavyweight for what should be needed here.

Is it feasible to have an agent+fleet server Docker containers in Kibana CI (basically just the agent parts of the elastic-package tooling stack)? That seems like a much nicer way to deal with this, unless there are specific platform dependencies you need to test (Windows for example).

[kpollich]

Is it feasible to have an agent+fleet server Docker containers in Kibana CI (basically just the agent parts of the elastic-package tooling stack)? That seems like a much nicer way to deal with this, unless there are specific platform dependencies you need to test (Windows for example).

This might be a possibility - on its face it doesn't seem overly concerning to me, at least. We'd probably want to talk with Kibana CI about it to confirm. We definitely don't need cross-platform testing here, as this is mainly just to test the Fleet config API to confirm it returns usable configuration for standalone agent.

[kpollich]

Closing. I'll create a follow up issue to discuss E2E testing for this separately.

[kpollich]

I created an issue to explore E2E testing in the future here. #182507

FYI I designated it as a spacetime project as I think there will be a lot of investigation required to get an Elastic Agent container spun up inside of Kibana CI.