Add logstash as a component in the stack boot up by elastic-package

[mrodm]

It could be added logstash as a new component/service in the Elastic stack boot up by elastic-package to be able to test different scenarios.

This logstash should not be started by default. It should be started and configured optionally (using a new parameter in stack up?).

This logstash could be configured optionally in the elastic-agent (as an output) and it should be configured Elasticsearch as an output of this logstash. Doing so, the documents sent by the elastic-agent are ingested using logstash. This could leverage the testing to test new scenarios where there are more fields present in the document, for instance event.original (based on elastic/integrations#3451 (comment)).

[bhapas]

@mrodm @jsoriano
cc: @andrewkroh
Just trying to summarise the solution thought for a discussion

• A new profile Config option to be added - stack.logstash_enabled which takes a bool in the form of string
• Add logstash to the docker-compose template file
  Not sure if we can enable/disable a service (logstash in this case) based on a bool config - Probably using profiles with compose
• Alternatively, Add another docker-compose-stack-logstash.yml.tmpl file with logstash in it and conditionally load the template file here
• To start with, a hardcoded logstash.conf.tmpl file with default output being elasticsearch and a basic filter plugin for elastic_integration , but would be ideal if we can inject user config for logstash based testing [ Did not arrive at a concrete solution for this. Happy to hear ].
• Configure the output of elastic agent to be logstash.

Please feel free to comment on one or all the points above and I would also like to get an opinion from logstash team for the docker compose logstash setup , once the PR is ready.

[bhapas]

Fixed in #1458