[POC] Run log rate analysis on metrics data

[benakansara]

Create a POC to review if Log rate analysis can be used for metrics data to have meaningful analysis results.

Acceptance criteria

• Test Log rate analysis with metrics data in Custom threshold rule
• Capture analysis results

[elasticmachine]

Pinging @elastic/obs-ux-management-team (Team:obs-ux-management)

[benakansara]

Using CCS

Scanario 1: Group by host.hostname, Document count alert on metrics data

Scanario 2: Group by kubernetes.pod.name, avg (kubernetes.pod.cpu.usage.limit.pct) alert

Scenario 3: sum (kubernetes.pod.network.rx.bytes)/1000000000000) + avg (kubernetes.container.cpu.usage.limit.pct) + max (kubernetes.node.cpu.capacity.cores) alert, Without group by

Scenario 4: (avg (kubernetes.container.cpu.usage.limit.pct) + avg (kubernetes.container.memory.usage.bytes)) / 10000000 alert, Group by kubernetes.pod.name

Using kbn-data-forge

Metrics data

Scanario 1: Increase in number of hosts, Before spike: 2 hosts (host-0, host-1), At spike: 10 hosts (host-0 ... host-9), Without group by, Average(system.cpu.user.pct) alert, Higher cpu usage in new hosts

The analysis identifies new hosts, containers, labels, network.

Scanario 2: Increase in number of hosts, Before spike: 2 hosts (host-0, host-1), At spike: 10 hosts (host-0 ... host-9), Without group by, Document count alert on metrics data

The analysis identifies new hosts, containers, labels, network.

Scanario 3: Change in metric values for system.cpu.user.pct metric, Without group by, Average(system.cpu.user.pct) alert

The analysis did not find any results.

Scanario 4: Change in metric values for system.cpu.user.pct metric, new metric added (system.cpu.system.pct) in spiked documents, Without group by, Average(system.cpu.user.pct) alert

The analysis did not find any results.

Scanario 5: Change in metric values for system.cpu.user.pct metric, Group by host.name, Average(system.cpu.user.pct) alert

The analysis did not find any results.

Scanario 6: Change in document count, Group by host.name, Average(system.cpu.user.pct) alert

The analysis did not find any results.

Scanario 7: Change in document count, Group by host.name, Document count alert on metrics data

The analysis did not find any results.

[benakansara]

Based on these observations, the "Log rate analysis" works best when there are differences in documents in terms of particular keyword/text fields that appear more frequently or less frequently. It does not find differences in numeric values.

In a more controlled setup like using kbn-data-forge where we decide the shape of data to test, the results are not promising. It's because we generate documents with same set of fields changing only particular metric value.

In conclusion, the analysis itself could work with any kind of documents, not just logs. In real metrics data, generally there are other "keyword" fields in a document beside metric fields. The analysis would find variations in those fields when ran on metrics data.