New data not generating for logstash output with proxy server.

[harshitgupta-qasource]

Kibana Build details:

VERSION: 8.9.0 BC2
BUILD: 64459
COMMIT: 6950a2b8207d8388ee8c842d6c0e2b1e1031fd36

Host OS and Browser version: All, All

Preconditions:

1. 8.9.0 BC2 Kibana cloud environment should be available.
2. Few agents should be installed.
3. Logstash output should be setup.

Steps to reproduce:

1. Navigate to Fleet>Agents tab.
2. Click on setting tab.
3. Add the Elasticsearch and logstash output with proxy server.
4. Observe that New data not generating for logstash output with proxy server.

Expected:
New data should generate for logstash output with proxy server.

What's working fine

• Elasticsearch output working fine with proxy server.
• Logstash output is working fine without proxy server.

Screen Shot:

Debug Logs:
elastic-agent-diagnostics-2023-07-05T06-19-09Z-00.zip

[elasticmachine]

Pinging @elastic/elastic-agent-control-plane (Team:Elastic-Agent-Control-Plane)

[harshitgupta-qasource]

@amolnater-qasource Kindly review

[amolnater-qasource]

Secondary review for this ticket is Done

[cmacknz]

Looking at the configuration I see an ES and an LS output defined:

outputs:
  c83b92b0-1a4f-11ee-9969-4bbdb32e834c:
    hosts:
    - 54.163.27.249:5044
    proxy_url: http://54.167.55.200:3128
    ssl:
      certificate: <REDACTED>
      certificate_authorities: <REDACTED>
      key: <REDACTED>
    type: logstash
  default:
    api_key: <REDACTED>
    hosts:
    - https://0dd2580f6ca74c8784ca47d9767439a9.europe-west1.gcp.cloud.es.io:443
    proxy_url: http://54.167.55.200:3128
    type: elasticsearch

The default ES output is only used for monitoring, the LS output is used for everything else.

In state.yaml I see all of the LS output units are failed:

- id: log-c83b92b0-1a4f-11ee-9969-4bbdb32e834c
  state:
    state: 2
    message: 'Healthy: communicating with pid ''726'''
    units:
      ? unittype: 0
        unitid: log-c83b92b0-1a4f-11ee-9969-4bbdb32e834c-logfile-system-fa3b1444-a61e-457d-8bbd-95a7e3b73b72
      : state: 1
        message: Configuring
      ? unittype: 1
        unitid: log-c83b92b0-1a4f-11ee-9969-4bbdb32e834c
      : state: 4
        message: 'could not start output: failed to reload output: more than one namespace
          configured'

This is the same problem as #2554, but how we got into this state appears to be different.

Looking at the logs I see:

{"log.level":"error","@timestamp":"2023-07-05T06:11:11.642Z","log.origin":{"file.name":"coordinator/coordinator.go","file.line":991},"message":"Unit state changed log-c83b92b0-1a4f-11ee-9969-4bbdb32e834c (STARTING->FAILED): could not start output: failed to reload output: proxy: unknown scheme: http accessing 'logstash'","log":{"source":"elastic-agent"},"component":{"id":"log-c83b92b0-1a4f-11ee-9969-4bbdb32e834c","state":"HEALTHY"},"unit":{"id":"log-c83b92b0-1a4f-11ee-9969-4bbdb32e834c","type":"output","state":"FAILED","old_state":"STARTING"},"ecs.version":"1.6.0"}

The problem is that an HTTP proxy was configured for Logstash, Logstash only supports SOCKS5 proxies:

https://github.com/elastic/beats/blob/62979b558ccbd6b221dcb2268c5e73665ba653d9/libbeat/outputs/logstash/docs/logstash.asciidoc#L318-L322

[cmacknz]

@jlind23 @juliaElastic it might be worth putting a note in the Fleet UI that Logstash only supports SOCKS5 proxies, or to prevent configuring proxies that use the http or https schemes with Logstash.

The error here is not very obvious as is, although I think it might improve once #2554 is fixed. I am going to close this because the behaviour is expected, we can reopen and transfer to Fleet if we want to adjust the wording here.

[jlind23]

@cmacknz it is already stated in the docs that only SOCK5 is usable - https://www.elastic.co/guide/en/fleet/current/logstash-output.html
I really don't want to overload the ui though.

[harshitgupta-qasource]

Hi @jlind23

Thank you for looking into this and confirming for the expected behavior.

We will be skipping the testcase related to this feature in our future regression.

Thanks