Logstash Improvements.

[mashhurs]

Description

1. Logstash team is aiming to create an E2E test using integration packages. During the tests, we would need a verification task which Logstash stats API will be helpful. We are introducing the port add to the docker network so that we can access stats API.
2. We are packaging elastic_integration plugin with Logstash and from LS 8.13 (planned) Logstash carries specific versions of the plugin. If we always install the plugin when LS container starts, we may lose the compatibility issue since it always installs the recent one. Introducing a logic to check if plugin is not bundled yet and install if so.

How to test

• Run make build
• ./elastic-package stack up -d
• Go to the LS docker and check for following logs

Missing plugin logstash-filter-elastic_integration, installing now
Using bundled JDK: /usr/share/logstash/jdk
Validating logstash-filter-elastic_integration
Resolving mixin dependencies
Installing logstash-filter-elastic_integration

• to check the exposed port, run curl localhost:9600/_node/stats?pretty. It should show current LS node stats.

[mashhurs]

@bhapas, I need your (or any of team member's) help for followings:

• check if BK failures are due to this change, at the glance fails don't seems so;
• I have no idea about version bump; It seems Makefile dynamically resolving the version considering commit hash, please advise a suggestion if I need to change commit/PR info.

Thanks.

[bhapas]

/test

[jsoriano]

1. Logstash team is aiming to create an E2E test using integration packages.

Do you have more details about this? Is this going to be to test logstash or to test packages?

check if BK failures are due to this change, at the glance fails don't seems so;

There seems to be a regression in the docker pre-installation in the image used in CI. We are trying to downgrade it in #1639, once merged try to update the branch.

[mashhurs]

1. Logstash team is aiming to create an E2E test using integration packages.

Do you have more details about this? Is this going to be to test logstash or to test packages?

elastic-package has a ready infrastructure (ES, LS, KB, Fleet, deploy packages to create Ingest Pipelines) and the goal is to test Logstash to run Ingest Pipelines. For this reason, I opened this PR to get the statistics about Logstash.

check if BK failures are due to this change, at the glance fails don't seems so;

There seems to be a regression in the docker pre-installation in the image used in CI. We are trying to downgrade it in #1639, once merged try to update the branch.

Thanks for heads up. I highly appreciate If you can please review this change and let us know the release plan. We will use the release distro.

[jsoriano]

Looks good, could you please try to update the branch to have a green CI?

[jsoriano]

We will need to move this to a script if we continue adding things here :)

[mashhurs]

One we package the plugin (soon with this PR), we can actually remove this script, no need.

[mrodm]

buildkite test this

[elasticmachine]

💚 Build Succeeded

• Buildkite Build
• Commit: 4f1422b

History

• 💔 Build #2181 failed 5027cc3
• 💔 Build #2170 failed 5027cc3

cc @mashhurs