Pre-build logstash image with plugin and configuration #1687

jsoriano · 2024-02-15T17:26:20Z

Polish some things in the logstash scenario:

Pre-build logstash image:
- It includes the plugin now so it doesn't need to be installed on every run, speeding up startup after first run.
- It includes the configuration, so it can be modified with auto-reload, as introduced in Logstash improvements: auto pipeline reload. #1668.
Use private key without converting it to PKCS8.
Use actual CA instead of elasticsearch certificate as CA in logstash output.
Startup script removed as operations moved to other places or not needed.
Use the same configuration template for serverless and compose providers.

Part of #1682.

internal/stack/_static/logstash.conf.tmpl

jsoriano · 2024-02-15T17:29:01Z

internal/stack/_static/docker-compose-stack.yml.tmpl

-    command: bash /usr/share/logstash/startup.sh
    volumes:
      - "../certs/logstash:/usr/share/logstash/config/certs"
-      - "../certs/elasticsearch/cert.pem:/usr/share/logstash/config/certs/elasticsearch.pem:ro"


This was creating a file owned by root in the user profile, we should avoid writing there.

jsoriano · 2024-02-15T17:43:37Z

It breaks serverless now, I will fix it.

mrodm · 2024-02-15T17:44:17Z

internal/stack/resources.go

-			Mode:         resource.FileMode(0755),
+			Path:    "Dockerfile.logstash",
+			Content: staticSource.File("_static/Dockerfile.logstash"),
 		},


There is also another reference to this file in serverlessresources.go

Trying to creaste a serverless Elastic stack raised this error:

2024/02/15 18:41:48 INFO Starting local services Error: booting up the stack failed: failed to start local services: could not initialize compose files for local services: there where errors: template: pattern matches no files: `_static/logstash_startup.sh`

elastic-package/internal/stack/serverlessresources.go

Line 38 in b2ee264

Content: staticSource.Template("_static/logstash_startup.sh"),

internal/stack/_static/logstash.conf.tmpl

internal/stack/_static/docker-compose-stack.yml.tmpl

internal/stack/_static/logstash.conf.tmpl

internal/stack/_static/docker-compose-stack.yml.tmpl

mashhurs · 2024-02-15T18:34:33Z

internal/stack/_static/serverless-logstash.conf.tmpl

-    user => '{{ fact "username" }}'
-    password => '{{ fact "password" }}'
-    ssl_enabled => true
-    data_stream => "true"


we should confirm this removal with @bhapas there is document_id => "%{[@metadata][_ingest_document][id]}" in on-prems... That was a reason I leaved it as duplicated.

The document_id was to avoid duplicate events to get ingested. It can be applied to serverless too.

mashhurs · 2024-02-15T19:28:30Z

internal/stack/_static/Dockerfile.logstash

+ARG IMAGE
+FROM ${IMAGE}
+
+RUN bin/logstash-plugin install logstash-filter-elastic_integration


This will pull new version of the plugin, the expectation to use as it is if plugin is bundled with current Logstash version. Wouldn't be problem with other plugins but this one is tightly coupled with ES.

Suggested change

RUN bin/logstash-plugin install logstash-filter-elastic_integration

RUN if [ ! "$(bin/logstash-plugin list)" = *logstash-filter-elastic_integration* ]; then \

echo "Missing plugin logstash-filter-elastic_integration, installing now."; \

bin/logstash-plugin install logstash-filter-elastic_integration; \

fi

~~FYI: couldn't test if this works...~~
Updated the suggestion, will work I think...

Change applied, thanks.

mashhurs

Thank you @jsoriano for a quality of work here. This is the best design.
I have pulled and tested both on-prems and serverless with Logstash. Put one comment to install the elastic_integration plugin IFF only not bundled/installed.

mrodm · 2024-02-19T10:16:18Z

I've tried to create a ~~cluster~~ Serverless project using this branch and adding the system package into the Agent policy. This configuration is not able to ingest data in Elasticsearch.

This is an example of the error that I get from logstash container:

Could not index event to Elasticsearch. .... :response=>{"create"=>{"status"=>400, "error"=>{"type"=>"document_parsing_exception", "reason"=>"[1:1011] failed to parse: _id must be unset or set to [7QWrKrJpmTXFC8zWAAABjcDXW-A] but was [%{[@metadata][_ingest_document][id]}] because [.ds-metrics-system.diskio-default-2024.02.19-000001] is in time_series mode", "caused_by"=>{"type"=>"illegal_argument_exception", "reason"=>"_id must be unset or set to [7QWrKrJpmTXFC8zWAAABjcDXW-A] but was [%{[@metadata][_ingest_document][id]}] because [.ds-metrics-system.diskio-default-2024.02.19-000001] is in time_series mode"}}}}}

cc @jsoriano @bhapas

EDIT: the cluster created was a Serverless project.

bhapas · 2024-02-19T10:22:52Z

I've tried to create a cluster using this branch and adding the system package into the Agent policy. This configuration is not able to ingest data in Elasticsearch.

This is an example of the error that I get from logstash container:

Could not index event to Elasticsearch. .... :response=>{"create"=>{"status"=>400, "error"=>{"type"=>"document_parsing_exception", "reason"=>"[1:1011] failed to parse: _id must be unset or set to [7QWrKrJpmTXFC8zWAAABjcDXW-A] but was [%{[@metadata][_ingest_document][id]}] because [.ds-metrics-system.diskio-default-2024.02.19-000001] is in time_series mode", "caused_by"=>{"type"=>"illegal_argument_exception", "reason"=>"_id must be unset or set to [7QWrKrJpmTXFC8zWAAABjcDXW-A] but was [%{[@metadata][_ingest_document][id]}] because [.ds-metrics-system.diskio-default-2024.02.19-000001] is in time_series mode"}}}}}

cc @jsoriano @bhapas

@mrodm I think for time series index mode cannot use the generated _id if I remember from @mashhurs .

One way is to remove the [%{[@metadata][_ingest_document][id]}] from logstash config and let it ingest duplicate entries [ As this is just for testing ] . Or have a different config for time_series mode.

jsoriano · 2024-02-19T12:45:04Z

One way is to remove the [%{[@metadata][_ingest_document][id]}] from logstash config and let it ingest duplicate entries [ As this is just for testing ] . Or have a different config for time_series mode.

I am removing it from serverless as it was before this change. In local it is not failing for me or for CI.

elasticmachine · 2024-02-19T13:21:35Z

💚 Build Succeeded

Buildkite Build
Commit: 29c41c4

History

💚 Build #2422 succeeded 98b3633
💚 Build #2421 succeeded e0ea34e
💚 Build #2416 succeeded b1f174b

cc @jsoriano

jsoriano added 2 commits February 15, 2024 18:15

Pre-build logstash image with plugin and config

3a2bef2

Remove PKCS8 generation

b1f174b

jsoriano requested review from bhapas and mashhurs February 15, 2024 17:26

jsoriano self-assigned this Feb 15, 2024

jsoriano commented Feb 15, 2024

View reviewed changes

internal/stack/_static/logstash.conf.tmpl Show resolved Hide resolved

jsoriano commented Feb 15, 2024

View reviewed changes

jsoriano requested a review from a team February 15, 2024 17:29

mrodm reviewed Feb 15, 2024

View reviewed changes

jsoriano marked this pull request as draft February 15, 2024 17:53

mashhurs reviewed Feb 15, 2024

View reviewed changes

internal/stack/_static/logstash.conf.tmpl Show resolved Hide resolved

internal/stack/_static/docker-compose-stack.yml.tmpl Show resolved Hide resolved

Share logstash resources with serverless

e0ea34e

jsoriano marked this pull request as ready for review February 15, 2024 18:42

jsoriano requested review from mashhurs and mrodm February 15, 2024 18:43

bhapas reviewed Feb 15, 2024

View reviewed changes

internal/stack/_static/logstash.conf.tmpl Show resolved Hide resolved

internal/stack/_static/logstash.conf.tmpl Outdated Show resolved Hide resolved

internal/stack/_static/docker-compose-stack.yml.tmpl Show resolved Hide resolved

bhapas approved these changes Feb 15, 2024

View reviewed changes

Remove unneeded setting

98b3633

mashhurs reviewed Feb 15, 2024

View reviewed changes

mashhurs approved these changes Feb 15, 2024

View reviewed changes

Apply feedback from review

29c41c4

mrodm approved these changes Feb 19, 2024

View reviewed changes

jsoriano merged commit 26e81fe into elastic:main Feb 19, 2024

jsoriano deleted the generate-pkcs8-keys branch February 19, 2024 14:15

jsoriano mentioned this pull request Feb 19, 2024

[serverless][local stack] Refactor ssl changes #1682

Open

2 tasks

-RUN bin/logstash-plugin install logstash-filter-elastic_integration
+RUN if [ ! "$(bin/logstash-plugin list)" = *logstash-filter-elastic_integration* ]; then \
+        echo "Missing plugin logstash-filter-elastic_integration, installing now."; \
+        bin/logstash-plugin install logstash-filter-elastic_integration; \
+    fi

Pre-build logstash image with plugin and configuration #1687

Pre-build logstash image with plugin and configuration #1687

Uh oh!

Conversation

jsoriano commented Feb 15, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

jsoriano Feb 15, 2024

Choose a reason for hiding this comment

Uh oh!

jsoriano commented Feb 15, 2024

Uh oh!

mrodm Feb 15, 2024

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

mashhurs Feb 15, 2024

Choose a reason for hiding this comment

Uh oh!

bhapas Feb 15, 2024

Choose a reason for hiding this comment

Uh oh!

mashhurs Feb 15, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

jsoriano Feb 19, 2024

Choose a reason for hiding this comment

Uh oh!

mashhurs left a comment

Choose a reason for hiding this comment

Uh oh!

mrodm commented Feb 19, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

bhapas commented Feb 19, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

jsoriano commented Feb 19, 2024

Uh oh!

elasticmachine commented Feb 19, 2024

💚 Build Succeeded

History

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

jsoriano commented Feb 15, 2024 •

edited

Loading

mashhurs Feb 15, 2024 •

edited

Loading

mrodm commented Feb 19, 2024 •

edited

Loading

bhapas commented Feb 19, 2024 •

edited

Loading