Validate command for benchmark templates

[ruflin]

When building a template for a benchmark, currently the only way to validate the template and test it, is shipping it to Elasticsearch with the stream command, for example:

elastic-package benchmark stream --benchmark error-benchmark -vv

Instead, it would be nice to be able to do multiple things with and without Elasticsearch:

• Validate if the template can be generated
  • Validate that it is JSON
  • Validate that it works for X number of events
• Preview output
  • Show on the console how an event looks like that is sent to Elasticsearch
• Test sending event
  • X number of events are sent to Elasticsearch, on success, read out of ES again and printed to console with the result. In case of error, print out the full error from Elasticsearch

The above should make developing templates much quicker.

Eventually, at least the validation step should also be run as part of the check command to ensure all templates in a package are valid.

[ruflin]

One additional thing that would be help is being able to get a sample event on how data would be shipped by elastic-agent before. I wonder if the combination of what @marc-gr did where schema A is collect, with the sample log messages we have, if there is an easy way to generate schema B? Taking schema B, the for the template only variables have to be filled in.

[ruflin]

@aspacca @aliabbas-elastic @jsoriano I would hope such a command would help with errors we found in elastic/integrations#8800 For example, if templates exists, we can do a quick test run for the package and if it fails, we also see it directly on the PR itself.

[paolafrancesca]

I wonder if the combination of what @marc-gr did where schema A is collect, with the sample log messages we have, if there is an easy way to generate schema B? Taking schema B, the for the template only variables have to be filled in.

a way to achieve the above is to apply a pipeline like the one from @marc-gr for system tests, and then read back the document from ES to use as base for the schema-B template.
at least for logs the solution should be generic enough.

[paolafrancesca]

There're valid reasons to split "validation" and "testing" in two different commands.
Mainly due to the fact that one requires a cluster and the other not.

I'm not sure about the "preview": it either could be enabled with a flag on both commands or another separated command, I have no strong opinion on that.

I suggest to start with the "validation" command.

@ruflin are we on the same page?

[ruflin]

@ruflin are we on the same page?

Yep. In general I see this more as a meta issue and we will get to the end goal in small steps.