Add further documentation for Logstash volumes #7022

robbavey · 2023-07-14T14:16:13Z

This commit adds some further details, examples and a recipe to document the use of Logstash volumes.

This commit also updates examples using the logstash elasticsearch output to use updated SSL settings after SSL standardization effort

`cacert` and `ssl` were recently deprecated in favor of `ssl_certificate_authorities` and `ssl_enabled` respectively. This commit updates the recipes and docs to reflect that.

pebrc

Looks really good!

docs/orchestrating-elastic-stack-applications/logstash.asciidoc

Co-authored-by: Peter Brachwitz <peter.brachwitz@gmail.com>

thbkrkr

Nice reading! Left some nits.

docs/orchestrating-elastic-stack-applications/logstash.asciidoc

Co-authored-by: Thibault Richard <thbkrkr@users.noreply.github.com>

* Add documentation for Logstash persistent storage * Update recipes and examples to use new settings for Elasticsearch output --------- Co-authored-by: Peter Brachwitz <peter.brachwitz@gmail.com> Co-authored-by: Thibault Richard <thbkrkr@users.noreply.github.com>

* Add documentation for Logstash persistent storage * Update recipes and examples to use new settings for Elasticsearch output --------- Co-authored-by: Rob Bavey <rob.bavey@elastic.co> Co-authored-by: Peter Brachwitz <peter.brachwitz@gmail.com>

karenzone

Nice work! I left some comments inline for consideration, mostly around using attribute (such as {ls} and {k8s} for consistency) and being through and consistent in the way we reference "persistent queue" and "PQ, and "dead letter queue" and "DLQ for users who search on one term or the other.

karenzone · 2023-07-20T18:52:48Z

docs/orchestrating-elastic-stack-applications/logstash.asciidoc

@@ -290,8 +291,16 @@ added:[2.9.0]

 WARNING: Volume support for Logstash is a breaking change to earlier versions of ECK and requires you to recreate your Logstash objects.

+
+[float]


Suggested change

[float]

[id="{p}-volume-claim-settings"]

[discrete]

Or, an assigned ID of your choosing. We have more control when we set IDs rather than allowing the system to generate them.

docs/orchestrating-elastic-stack-applications/logstash.asciidoc

karenzone · 2023-07-20T19:45:06Z

docs/orchestrating-elastic-stack-applications/logstash.asciidoc

@@ -312,7 +321,10 @@ spec:
 ----


-Separate storage, for example for Logstash configurations using persistent queues (PQ) and/or dead letter queues (DLQ), can be added by including an additional `spec.volumeClaimTemplate` along with a corresponding `spec.podTemplate.spec.containers.volumeMount` for each requested volume, as shown in the following example:
+Separate storage, for example for Logstash configurations using persistent queues (PQ) and/or dead letter queues (DLQ), can be added by including an additional `spec.volumeClaimTemplate` along with a corresponding `spec.podTemplate.spec.containers.volumeMount` for each requested volume.


Suggested change

Separate storage, for example for Logstash configurations using persistent queues (PQ) and/or dead letter queues (DLQ), can be added by including an additional `spec.volumeClaimTemplate` along with a corresponding `spec.podTemplate.spec.containers.volumeMount` for each requested volume.

You can add separate storage by including an additional `spec.volumeClaimTemplate` along with a corresponding `spec.podTemplate.spec.containers.volumeMount` for each requested volume.

{ls} configurations using persistent queues (PQs) and/or dead letter queues (DLQs) are examples where you might want to add separate storage.

^^ That's more words, but it seems easier to parse.

For clarification: Is adding separate storage a decision or a "must" in the examples given?

karenzone · 2023-07-20T19:46:43Z

docs/orchestrating-elastic-stack-applications/logstash.asciidoc

-Separate storage, for example for Logstash configurations using persistent queues (PQ) and/or dead letter queues (DLQ), can be added by including an additional `spec.volumeClaimTemplate` along with a corresponding `spec.podTemplate.spec.containers.volumeMount` for each requested volume, as shown in the following example:
+Separate storage, for example for Logstash configurations using persistent queues (PQ) and/or dead letter queues (DLQ), can be added by including an additional `spec.volumeClaimTemplate` along with a corresponding `spec.podTemplate.spec.containers.volumeMount` for each requested volume.
+
+The following example shows how to setup separate storage for a PQ:


Suggested change

The following example shows how to setup separate storage for a PQ:

This example shows how to setup separate storage for a PQ:

karenzone · 2023-07-20T19:48:23Z

docs/orchestrating-elastic-stack-applications/logstash.asciidoc

+<2> Set the `path.queue` setting in the configuration to match the `mountPath` in the `volumeMount`.
+
+
+The following example shows how to configure Logstash with a Dead Letter Queue setup on the main pipeline, and a separate pipeline to read items from the DLQ.


Suggested change

The following example shows how to configure Logstash with a Dead Letter Queue setup on the main pipeline, and a separate pipeline to read items from the DLQ.

This example shows how to configure {ls} with a Dead Letter Queue setup on the main pipeline, and a separate pipeline to read items from the DLQ.

karenzone · 2023-07-20T20:33:12Z

docs/orchestrating-elastic-stack-applications/logstash.asciidoc

@@ -597,6 +735,16 @@ kubectl apply -f {logstash_recipes}/logstash-es-role.yaml

 Deploys Logstash and Elasticsearch, and creates an updated version of the `eck_logstash_user_role` to write to a user specified index.

+[id="{p}-logstash-configuration-pq-dlq"]
+=== Creating persitent volumes for PQ and DLQ


Suggested change

=== Creating persitent volumes for PQ and DLQ

=== Creating persistent volumes for PQ and DLQ

karenzone · 2023-07-20T20:38:51Z

docs/orchestrating-elastic-stack-applications/logstash.asciidoc

+kubectl apply -f {logstash_recipes}/logstash-volumes.yaml
+----
+
+Deploys Logstash, Beats and Elasticsearch. Logstash is configured with two pipelines - a main pipeline for reading from the Beats instance, which will send to the DLQ if it is unable to write to Elasticsearch, and a second pipeline, that will read from the DLQ. In addition, persistent queues are set up. This example shows how to configure persistent volumes outside of the default `logstash-data` persistent volume.


Suggested change

Deploys Logstash, Beats and Elasticsearch. Logstash is configured with two pipelines - a main pipeline for reading from the Beats instance, which will send to the DLQ if it is unable to write to Elasticsearch, and a second pipeline, that will read from the DLQ. In addition, persistent queues are set up. This example shows how to configure persistent volumes outside of the default `logstash-data` persistent volume.

Deploys {ls], [beats] and {es]. {ls] is configured with two pipelines:

* a main pipeline for reading from the {beats} instance, which will send to the DLQ if it is unable to write to Elasticsearch

* a second pipeline, that will read from the DLQ.

In addition, persistent queues are set up.

This example shows how to configure persistent volumes outside of the default `logstash-data` persistent volume.

karenzone · 2023-07-20T20:39:23Z

docs/orchestrating-elastic-stack-applications/logstash.asciidoc

-=== No integrated support for persistence
-The operator provides no integrated support for persistence, including PQ, DLQ support and plugins that may require persistent storage to keep track of state - any persistence should be added manually.
+=== Experimental support for persistence
+NOTE: This is a breaking change from version 2.8.0 of the ECK operator and requires re-creation of existing Logstash objects.


Suggested change

NOTE: This is a breaking change from version 2.8.0 of the ECK operator and requires re-creation of existing Logstash objects.

NOTE: Persistence (experimental) is a breaking change from version 2.8.0 of the ECK operator and requires re-creation of existing {ls} objects.

karenzone · 2023-07-20T20:51:14Z

docs/orchestrating-elastic-stack-applications/logstash.asciidoc

+=== Experimental support for persistence
+NOTE: This is a breaking change from version 2.8.0 of the ECK operator and requires re-creation of existing Logstash objects.
+
+The operator now includes support for persistence, with a small (`1Gi`) default `PersistentVolume` called `logstash-data` created, that maps to `/usr/share/logstash/data`, which should be typically used for storage from plugins. The default volume can be overridden by adding a `spec.volumeClaimTemplate` section named `logstash-data` to add more storage, or to use a different `storageClass` from the default, for example. Additional `persistentVolumeClaims` can also be defined in `spec.volumeClaimTemplate` for use with PQ, or DLQ, for example.


Suggested change

The operator now includes support for persistence, with a small (`1Gi`) default `PersistentVolume` called `logstash-data` created, that maps to `/usr/share/logstash/data`, which should be typically used for storage from plugins. The default volume can be overridden by adding a `spec.volumeClaimTemplate` section named `logstash-data` to add more storage, or to use a different `storageClass` from the default, for example. Additional `persistentVolumeClaims` can also be defined in `spec.volumeClaimTemplate` for use with PQ, or DLQ, for example.

The operator now includes support for persistence.

It creates a small (`1Gi`) default `PersistentVolume` called `logstash-data` that maps to `/usr/share/logstash/data`, typically used for storage from plugins.

The default volume can be overridden by adding a `spec.volumeClaimTemplate` section named `logstash-data` to add more storage, or to use a different `storageClass` from the default, for example.

You can define additional `persistentVolumeClaims` in `spec.volumeClaimTemplate` for use with PQ, or DLQ, for example.

karenzone · 2023-07-20T20:54:43Z

docs/orchestrating-elastic-stack-applications/logstash.asciidoc

+
+The operator now includes support for persistence, with a small (`1Gi`) default `PersistentVolume` called `logstash-data` created, that maps to `/usr/share/logstash/data`, which should be typically used for storage from plugins. The default volume can be overridden by adding a `spec.volumeClaimTemplate` section named `logstash-data` to add more storage, or to use a different `storageClass` from the default, for example. Additional `persistentVolumeClaims` can also be defined in `spec.volumeClaimTemplate` for use with PQ, or DLQ, for example.
+
+The current implementation does not allow resizing of volumes even if your chosen storage class would support it. To resize a volume you have to fully delete the Logstash object, delete and recreate or resize the volume and create a new Logstash object. Also note that volume claims will not be deleted when you delete the Logstash resource and have to be deleted manually. This behaviour might change in future versions of the ECK operator.


Suggested change

The current implementation does not allow resizing of volumes even if your chosen storage class would support it. To resize a volume you have to fully delete the Logstash object, delete and recreate or resize the volume and create a new Logstash object. Also note that volume claims will not be deleted when you delete the Logstash resource and have to be deleted manually. This behaviour might change in future versions of the ECK operator.

The current implementation does not allow resizing of volumes, even if your chosen storage class would support it.

To resize a volume, delete the {ls] object, delete and recreate (or resize) the volume, and create a new {ls} object.

Note that volume claims will not be deleted when you delete the {ls} resource, and must be deleted manually.

This behavior might change in future versions of the ECK operator.

thbkrkr · 2023-07-24T07:17:42Z

I will open a new PR with Karen's review feebacks.

Edit:

Additional improvement for Logstash volumes documentation #7053

robbavey added 3 commits July 14, 2023 09:49

Add documentation for Logstash persistent storage

a5467af

Update recipes and examples to use new settings for Elasticsearch output

ff2e231

`cacert` and `ssl` were recently deprecated in favor of `ssl_certificate_authorities` and `ssl_enabled` respectively. This commit updates the recipes and docs to reflect that.

Merge content with basic docs

d27400d

robbavey requested a review from karenzone July 14, 2023 14:16

botelastic bot added the triage label Jul 14, 2023

pebrc added the >docs Documentation label Jul 17, 2023

botelastic bot removed the triage label Jul 17, 2023

pebrc added triage v2.9.0 labels Jul 17, 2023

thbkrkr removed the triage label Jul 17, 2023

pebrc reviewed Jul 19, 2023

View reviewed changes

pebrc and others added 3 commits July 19, 2023 18:47

Merge remote-tracking branch 'upstream/main' into pr/7022

45abb0e

Apply suggestions from code review

2766ed1

Co-authored-by: Peter Brachwitz <peter.brachwitz@gmail.com>

Amend text on terminationGracePeriodSeconds not being honored

1a2d66b

pebrc approved these changes Jul 20, 2023

View reviewed changes

thbkrkr approved these changes Jul 20, 2023

View reviewed changes

Apply suggestions from code review

c997f93

Co-authored-by: Thibault Richard <thbkrkr@users.noreply.github.com>

thbkrkr merged commit d2e915f into elastic:main Jul 20, 2023
5 checks passed

thbkrkr mentioned this pull request Jul 20, 2023

Add further documentation for Logstash volumes (#7022) #7046

Merged

karenzone reviewed Jul 20, 2023

View reviewed changes

karenzone self-assigned this Jul 20, 2023

thbkrkr mentioned this pull request Jul 24, 2023

Additional improvement for Logstash volumes documentation #7053

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add further documentation for Logstash volumes #7022

Add further documentation for Logstash volumes #7022

robbavey commented Jul 14, 2023

pebrc left a comment

thbkrkr left a comment

karenzone left a comment

karenzone Jul 20, 2023

karenzone Jul 20, 2023

karenzone Jul 20, 2023

karenzone Jul 20, 2023

karenzone Jul 20, 2023

karenzone Jul 20, 2023

karenzone Jul 20, 2023

karenzone Jul 20, 2023

karenzone Jul 20, 2023

karenzone Jul 20, 2023

karenzone Jul 20, 2023

thbkrkr commented Jul 24, 2023 •

edited

		@@ -290,8 +291,16 @@ added:[2.9.0]

		WARNING: Volume support for Logstash is a breaking change to earlier versions of ECK and requires you to recreate your Logstash objects.


		[float]

	Separate storage, for example for Logstash configurations using persistent queues (PQ) and/or dead letter queues (DLQ), can be added by including an additional `spec.volumeClaimTemplate` along with a corresponding `spec.podTemplate.spec.containers.volumeMount` for each requested volume.
	You can add separate storage by including an additional `spec.volumeClaimTemplate` along with a corresponding `spec.podTemplate.spec.containers.volumeMount` for each requested volume.
	{ls} configurations using persistent queues (PQs) and/or dead letter queues (DLQs) are examples where you might want to add separate storage.

	The following example shows how to setup separate storage for a PQ:
	This example shows how to setup separate storage for a PQ:

		<2> Set the `path.queue` setting in the configuration to match the `mountPath` in the `volumeMount`.


		The following example shows how to configure Logstash with a Dead Letter Queue setup on the main pipeline, and a separate pipeline to read items from the DLQ.

	The following example shows how to configure Logstash with a Dead Letter Queue setup on the main pipeline, and a separate pipeline to read items from the DLQ.
	This example shows how to configure {ls} with a Dead Letter Queue setup on the main pipeline, and a separate pipeline to read items from the DLQ.

	=== Creating persitent volumes for PQ and DLQ
	=== Creating persistent volumes for PQ and DLQ

	NOTE: This is a breaking change from version 2.8.0 of the ECK operator and requires re-creation of existing Logstash objects.

	NOTE: Persistence (experimental) is a breaking change from version 2.8.0 of the ECK operator and requires re-creation of existing {ls} objects.

-The operator now includes support for persistence, with a small (`1Gi`) default `PersistentVolume` called `logstash-data` created, that maps to `/usr/share/logstash/data`, which should be typically used for storage from plugins. The default volume can be overridden by adding a `spec.volumeClaimTemplate` section named `logstash-data` to add more storage, or to use a different `storageClass` from the default, for example. Additional `persistentVolumeClaims` can also be defined in `spec.volumeClaimTemplate` for use with PQ, or DLQ, for example.
+The operator now includes support for persistence.
+It creates a small (`1Gi`) default `PersistentVolume` called `logstash-data` that maps to `/usr/share/logstash/data`, typically used for storage from plugins.
+The default volume can be overridden by adding a `spec.volumeClaimTemplate` section named `logstash-data` to add more storage, or to use a different `storageClass` from the default, for example.
+You can define additional `persistentVolumeClaims`  in `spec.volumeClaimTemplate` for use with PQ, or DLQ, for example.


		The operator now includes support for persistence, with a small (`1Gi`) default `PersistentVolume` called `logstash-data` created, that maps to `/usr/share/logstash/data`, which should be typically used for storage from plugins. The default volume can be overridden by adding a `spec.volumeClaimTemplate` section named `logstash-data` to add more storage, or to use a different `storageClass` from the default, for example. Additional `persistentVolumeClaims` can also be defined in `spec.volumeClaimTemplate` for use with PQ, or DLQ, for example.

		The current implementation does not allow resizing of volumes even if your chosen storage class would support it. To resize a volume you have to fully delete the Logstash object, delete and recreate or resize the volume and create a new Logstash object. Also note that volume claims will not be deleted when you delete the Logstash resource and have to be deleted manually. This behaviour might change in future versions of the ECK operator.

-The current implementation does not allow resizing of volumes even if your chosen storage class would support it. To resize a volume you have to fully delete the Logstash object, delete and recreate or resize the volume and create a new Logstash object. Also note that volume claims will not be deleted when you delete the Logstash resource and have to be deleted manually. This behaviour might change in future versions of the ECK operator.
+The current implementation does not allow resizing of volumes, even if your chosen storage class would support it.
+To resize a volume, delete the {ls] object, delete and recreate (or resize) the volume, and create a new {ls} object.
+Note that volume claims will not be deleted when you delete the {ls} resource, and must be deleted manually.
+This behavior might change in future versions of the ECK operator.

Add further documentation for Logstash volumes #7022

Add further documentation for Logstash volumes #7022

Conversation

robbavey commented Jul 14, 2023

pebrc left a comment

Choose a reason for hiding this comment

thbkrkr left a comment

Choose a reason for hiding this comment

karenzone left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

thbkrkr commented Jul 24, 2023 • edited

thbkrkr commented Jul 24, 2023 •

edited