Unable to configure index patterns for SIEM UI

[canterberry]

Describe the feature:

The Logs UI settings in Kibana provides for the ability to set the index patterns for querying log and metrics data, but the SIEM UI does not provide for the ability to configure this. As a result, the SIEM UI cannot find my data.

I would like to be able to configure (via settings or via the SIEM UI) index patterns to help the SIEM UI find my data.

For reference, here's what the Infrastructure UI looks like for configuring the index patterns:

Describe a specific use case for the feature:

I use versioned indexes like logstash-7.2.0-auditbeat-2019.07.01, and the SIEM UI just can't seem to find the Beats data I know is there (from FileBeat, MetricBeat, AuditBeat, and PacketBeat). As a result, I am unable to use what seems like a really useful part of Kibana.

[elasticmachine]

Pinging @elastic/secops

[cwurm]

Hi @canterberry - the index patterns used by the SIEM app are configurable through the setting siem:defaultIndex in Kibana → Management → Advanced Settings. By default, they are the Beats index patterns auditbeat-*, winlogbeat-*, filebeat-*, packetbeat-*.

So in your case, you could change them by prepending logstash (e.g. logstash-*-auditbeat-*).

We put this in the documentation, but I agree it's not easy to find.

One more thing: When using Logstash to send data intended for the SIEM app, it's important to first import the respective Beats index template (e.g. by running ./auditbeat setup) so that the fields will have the correct types (most importantly, host.name has to be a keyword field to allow aggregations on it).

[canterberry]

Thanks, @cwurm. I didn't know there even was a dedicated SIEM guide -- I was looking at Kibana's SIEM UI reference. Upon reinspection, this does indeed appear to be covered there as well.

I suppose my feature request, then, pivots to surfacing this in the UI like the Infrastructure and Logs apps do (not necessarily in the same way, just indicating some prior art in this area to help clarify the intent of the request).

[canterberry]

☝️ Also, thanks for the heads-up on running auditbeat setup. We haven't run the setup step for any of our Beats, so this would probably have bitten us next.

[canterberry]

"One more thing: When using Logstash to send data intended for the SIEM app, it's important to first import the respective Beats index template (e.g. by running ./auditbeat setup) so that the fields will have the correct types (most importantly, host.name has to be a keyword field to allow aggregations on it)."

This is rather unfortunate, and we're back to square one in a slightly different way: I have the host.name data the SIEM UI wants, in aggregatable keyword form as host.name.keyword. This is how Elasticsearch treats strings by default (i.e: index as both string and keyword fields), so I would propose that the SIEM UI be capable of reading from this field (and/or maybe use this field by default, since it's how the data is structured from beats right out of the box).

Instead, I get a big ol' nasty column of errors when I view the SIEM UI:

Happy to create a separate issue for this. Again, the SIEM UI looks like it could be very useful. I'm excited to see it mature into a solid part of Kibana and useful with minimal to no friction.

[cwurm]

Ah, I think I know what's happening: Because you're using a different index template, ./auditbeat setup didn't do the right thing for you. By default, it creates an index template for auditbeat-*, but you want logstash-*-auditbeat-*.

To make it do the right thing, the settings inside auditbeat.yml when running setup should be:

setup.ilm.enabled: false # Disable Index Lifecycle Management (ILM) - we only want the template
setup.template.name: "logstash-auditbeat-7.2.0"
setup.template.pattern: "logstash-*-auditbeat-*"

You can verify template loading succeeded by checking GET _template/logstash-auditbeat-7.2.0 in the Kibana Dev Tools Console.

[canterberry]

Thanks, @cwurm. Getting auditbeat setup to work in our environment was pretty painful, but we did end up running it. We checked and double-checked every configuration option, among which was the setup.template.pattern set to the above pattern. The issue is that this cannot retroactively change the field mappings on the indexes that already exist. I'm sure there's a way to manually reindex with a different schema, though, so that's what I think we'll need to do in order to use the SIEM UI with the existing data.

☝️ Aside: Our Beats agents have no awareness of Elasticsearch, including which index the beats ultimately get shipped to. This is Logstash's responsibility, so for one of these agents to suddenly be aware of Elasticsearch blurs the line of responsibility. I actually think I saw a section in the documentation for this scenario, where AuditBeat can be configured to export the pattern to a file that can be later imported from a console with ES access. This has the added benefit of being able to review the changes before applying them, and overall seems like a much cleaner, portable, and straightforward method of configuring the index template than auditbeat setup, which only works without friction in the simplest of deployments.
Reference: Load the template manually (alternate method)

[cwurm]

Yeah, if you only want the index template exporting it to a file with ./auditbeat export template and importing it manually is a good idea. setup does a few more things like setting up Kibana index patterns and pre-configured dashboards.