[Monitoring] Logstash pipeline main is throwing unknown vertex type separator error

[liza-mae]

Kibana version: 7.7.0 BC5

Elasticsearch version: 7.7.0 BC5

Server OS version: Linux

Browser version: Chrome Latest

Original install method (e.g. download page, yum, from source, etc.):
Staging

Description of the problem including expected versus actual behavior:
Navigating to Monitoring Logstash Pipelines is throwing an error Unknown vertex type separator!

Steps to reproduce:

1. Stack Monitoring self monitoring enabled
2. Go to -> Logstash -> Pipeline -> Main

Errors in browser console (if relevant):
kbn-ui-shared-deps.js:334 Error: Unknown vertex type separator! This shouldn't happen!
at vertexFactory (monitoring.bundle.js:2)
at monitoring.bundle.js:2
at Array.forEach ()
at Graph.update (monitoring.bundle.js:2)
at Config.update (monitoring.bundle.js:2)
at PipelineState.update (monitoring.bundle.js:2)
at new PipelineState (monitoring.bundle.js:2)
at monitoring.bundle.js:2
at c.$digest (kbn-ui-shared-deps.js:334)
at c.$apply (kbn-ui-shared-deps.js:334)

[elasticmachine]

Pinging @elastic/stack-monitoring (Team:Monitoring)

[liza-mae]

This test fails on 7.6.2 also, but not on 7.6.1.

[chrisronline]

This look related: #58205

[chrisronline]

@liza-mae Can you share your logstash setup? configs and pipelines?

[liza-mae]

@chrisronline just using default pipeline -> main, no additional configuration.

[chrisronline]

Thanks, I can confirm the behavior now. Investigating

[chrisronline]

@liza-mae I confirmed this behavior with Metricbeat monitoring. Did you use that, or internal collection?

The issue with Metricbeat monitoring is a beats issues, recently fixed: elastic/beats#17497

[ycombinator]

To add to what @chrisronline said, elastic/beats#17497 did not make it to BC5, so when you use the next BC you shouldn't see this issue.

[liza-mae]

Thanks I will confirm on the next BC, this one was Logstash not Metricbeat will it fix both? Also some errors in 7.6.2 but that might be a different one I can recheck.

[robbavey]

@chrisronline, @ycombinator The unknown vertex type "separator" is likely due to this change in Logstash, which introduced a new vertex type separator into logstash to fix an issue with ingestion performance with event ordered pipelines.

cc @colinsurprenant

[chrisronline]

@robbavey Ah interesting. We maintain a whitelist which will need to be updated for this to work properly in monitoring

[chrisronline]

It turns out there are actually three issues related to this.

1. The actual issue in the ticket, which is a result of [Monitoring] Logstash pipeline main is throwing unknown vertex type separator error  #63082 (comment)

2. An issue with Metricbeat monitoring Logstash: Fix initialization of Logstash service URL beats#17497

3. An issue with internal monitoring with Logstash: Issue sending metrics when pipeline name is not 'main'. logstash#11775

---

2. and 3. have fixes, but nothing for 1 yet. I'll work on that

[chrisronline]

I have a WIP PR to address the blocker in Kibana: #63362

The main issue is there is complexity here I don't understand. I wrote enough code to load the page, but I don't think it will fully work. I don't know how to model/view the changes introduced in elastic/logstash#11710 and would most likely need help from the Logstash team to understand how to properly show this in the UI.

[chrisronline]

Update here. A fix is coming from the logstash side: elastic/logstash#11779

[chrisronline]

Closing this as a fix will be available in the next BC