[Self-Managed]: No data for Standalone agent available when configured with logstash output

[harshitgupta-qasource]

Kibana Build details:

VERSION: 8.12.0 BC3
BUILD: 69985
COMMIT: 2a8afed8572a4c709aa1c64216748197eeb9b18f

Host OS and Browser version: All, All
Preconditions:

1. 8.12.0 Kibana self-managed environment should be available.
2. Logstash should be configured with certs.

Steps to reproduce:

1. Navigate to Agent policies.
2. Set output for integrations as Logstash.
3. Install standalone agent using this agent policy.
4. Observe no data for Standalone agent available under Data Streams tab.

Expected:
Data for Standalone agent should be available when configured with logstash output

Screen Shot:

Logs
elastic-agent-diagnostics-2023-12-26T06-20-50Z-00.zip

[elasticmachine]

Pinging @elastic/elastic-agent-control-plane (Team:Elastic-Agent-Control-Plane)

[harshitgupta-qasource]

@amolnater-qasource Kindly review

[amolnater-qasource]

Secondary Review for this ticket is Done.

[cmacknz]

There is an error in the output configuration:

        units:
            input-log-1d4167d9-c043-40c4-97dd-6ac75c2c6a13-logfile-nginx-a7b51d40-b086-41f0-8d65-f43f24abc846:
                message: Starting
                state: 0
            input-log-1d4167d9-c043-40c4-97dd-6ac75c2c6a13-logfile-system-9997e7d1-bb8e-4799-9361-1678ea5e53c1:
                message: Starting
                state: 0
            output-log-1d4167d9-c043-40c4-97dd-6ac75c2c6a13:
                message: 'could not start output: failed to reload output: could not setup output certificates reloader: unpacking ''ssl'' config: key file not configured accessing ''logstash.ssl'''
                state: 4

Looks like https://www.elastic.co/guide/en/beats/filebeat/current/configuration-ssl.html#client-key is missing.

How to create it is documented in https://www.elastic.co/guide/en/fleet/current/secure-logstash-connections.html

[cmacknz]

@harshitgupta-qasource was the agent reported as healthy while this was happening?

[harshitgupta-qasource]

Hi @cmacknz

Thank you for looking into this issue.

As this agent is standalone agent so we were not able to see the agent status on UI.

However, we had set agent monitoring output to elasticsearch, and we were getting data for that.

Further we were also getting data for Fleet based agents through logstash.

Kindly let us know if we missed anything.

Thanks

[harshitgupta-qasource]

Bug Conversion

• Test-Case not required as this particular checkpoint is already covered in the following testcase:
  • https://elastic.testrail.io/index.php?/cases/view/167099

Thanks!

[amolnater-qasource]

Hi Team,

We have revalidated this issue on 8.13.0 BC1 self-managed environment and found it still reproducible.

Observations:

• No data for Standalone agent available when configured with logstash output.

Build details:
VERSION: 8.13.0
BUILD: 71738
COMMIT: b036a9705a55f6c81d065011ad8c991cbc3101d9

Agent Logs:
elastic-agent-diagnostics-2024-02-23T11-02-04Z-00.zip

Please let us know if anything else is required from our end.
Thanks!

[cmacknz]

The output SSL configuration is wrong:

        units:
            input-log-838567eb-fed1-4076-854f-3fe2dd7b6dce-logfile-system-b9a61cda-4342-43d7-84cb-52b0793ff187:
                message: Starting
                state: 0
            output-log-838567eb-fed1-4076-854f-3fe2dd7b6dce:
                message: 'could not start output: failed to reload output: could not setup output certificates reloader: unpacking ''ssl'' config: key file not configured accessing ''logstash.ssl'''
                state: 4

Looking at the output configuration I see secrets.ssl.key.id which is a Fleet secrets reference. The agent Logstash output doesn't understand this, the value is replaced by Fleet server.

outputs:
    838567eb-fed1-4076-854f-3fe2dd7b6dce:
        hosts:
            - 54.211.7.83:5044
        secrets:
            ssl:
                key:
                    id: JQ551Y0BX8drvHWrVKoy
        ssl:
            certificate: <REDACTED>
            certificate_authorities:
                - <REDACTED>
        type: logstash

It looks like you may have copied the Logstash SSL configuration from a Fleet managed agent policy in the Fleet UI, where the SSL private key is correctly hidden. If you want to get the configuration an agent would run, you have to get the policy from an agent running that policy using the elastic-agent inspect command or by looking at the diagnostics.

[amolnater-qasource]

Hi @cmacknz

Thank you for the update.

We have revalidated this without creating the logstash output with secrets and we are successfully able to get the data under data streams tab for standalone agent configured with Logstash output.

Screenshot:

Hence, we are closing this issue and marking as QA:Validated.

Please let us know if anything else is required from our end.
Thanks!!