Collect queue growth events and bytes metrics when PQ is enabled. #14554

mashhurs · 2022-09-26T18:19:36Z

Release notes

Logstash introduced instance and pipeline level flow metrics, growth_bytes and growth_events for persisted queue to provide a better visibility about how fast pipeline queue is growing each single seconds.

What does this PR do?

Introducing instance and pipeline level flow metrics, growth_bytes and growth_events for persisted queue.

Why is it important/What is the impact to the user?

Refer to #14463 to get to know about motivation and importance.

Checklist

My code follows the style guidelines of this project
I have commented my code, particularly in hard-to-understand areas
I have made corresponding changes to the documentation
~~[ ] I have made corresponding change to the default configuration files (and/or docker env variables)~~
I have added tests that prove my fix is effective or that my feature works

Author's Checklist

Check Lazy gauge initialization logics with Ry
Check integration tests CI before merging.

How to test this PR locally

Checkout the change locally and simply run the Logstash after assembling.
Set the pipeline with PQ enabled.
Call the node stats (or pipeline) API to check the growth flow rates.

Unit tests

$bin/rspec logstash-core/spec/logstash/api/modules/node_stats_spec.rb
$bin/rspec logstash-core/spec/logstash/api/commands/stats_spec.rb:184

Related issues

Closes #14556

Use cases

Screenshots

Pipeline level PQ flow metrics (IFF only for persisted)

Logs

// not sure how to replicate but tested the log readability with mockups.
[2022-09-26T10:14:01,982][WARN ][org.logstash.execution.AbstractPipelineExt] Cannot initialize `growth_events` metric. Retrieving `events` number gauge from `[queue]` namespace(s) failed.
[2022-09-26T10:14:01,982][WARN ][org.logstash.execution.AbstractPipelineExt] Cannot initialize `growth_bytes` metric. Retrieving `queue_size_in_bytes` number gauge from `[queue, capacity]` namespace(s) failed.

logstash-core/src/main/java/org/logstash/execution/AbstractPipelineExt.java

github-actions · 2022-09-28T20:51:26Z

📃 DOCS PREVIEW ✨ https://logstash_14554.docs-preview.app.elstc.co/diff

yaauie

In general, this looks to be on the right track. I'll take an attempt at making it possible for a FlowMetric to take Supplier<Metric<? extends Number>, or some other way to create a flow metric that can tolerate the metrics not yet being populated.

docs/static/monitoring/monitoring-apis.asciidoc

logstash-core/src/main/java/org/logstash/execution/AbstractPipelineExt.java

github-actions · 2022-10-04T19:18:00Z

📃 DOCS PREVIEW ✨ https://logstash_14554.docs-preview.app.elstc.co/diff

Change uptime precision for PQ growth metrics to uptime seconds since PQ events are based on seconds. Co-authored-by: Ry Biesemeyer <yaauie@users.noreply.github.com>

cla-checker-service · 2022-10-10T15:43:01Z

💚 CLA has been signed

github-actions · 2022-10-10T15:53:51Z

📃 DOCS PREVIEW ✨ https://logstash_14554.docs-preview.app.elstc.co/diff

docs/static/monitoring/monitoring-apis.asciidoc

github-actions · 2022-10-10T17:20:26Z

📃 DOCS PREVIEW ✨ https://logstash_14554.docs-preview.app.elstc.co/diff

Enables interface `FlowMetrics::create` to take suppliers that _implement_ a `Metric<? extends Number>` instead of requiring them to be pre-cast, and avoid unnecessary exposure of the metrics value-type into our lazy init.

Avoids routing two enum values through `MetricType#toString()` and `String#equals()` when they can be compared directly.

…space

Flow metrics pq in pipeline namespace

github-actions · 2022-10-12T18:45:28Z

📃 DOCS PREVIEW ✨ https://logstash_14554.docs-preview.app.elstc.co/diff

- Unit and integration tests are added or fixed. - Documentation added along with sample response data

github-actions · 2022-10-12T21:26:39Z

📃 DOCS PREVIEW ✨ https://logstash_14554.docs-preview.app.elstc.co/diff

mashhurs · 2022-10-12T21:27:44Z

docs/static/monitoring/monitoring-apis.asciidoc

@@ -534,6 +534,18 @@ A pipeline with only one single-threaded input may contribute up to 1.00, a pipe

 Additionally, some amount of back-pressure is both _normal_ and _expected_ for pipelines that are _pulling_ data, as this back-pressure allows them to slow down and pull data at a rate its downstream pipeline can tolerate.

+| `queue_persisted_growth_bytes` |


@yaauie, I need your review these important lines. Want to be sure every single point we are going to deliver to the end-user shouldn't be missed. Should be concise!

Sweet. I had been working on the docs and have just rebased and made a PR against your branch -> mashhurs#3

logstash-core/spec/logstash/api/commands/stats_spec.rb

qa/integration/specs/reload_config_spec.rb

… section. Update logstash-core/spec/logstash/api/commands/stats_spec.rb Co-authored-by: Ry Biesemeyer <yaauie@users.noreply.github.com>

Mistake: `flow_status` should be `pipeline_flow_stats` Co-authored-by: Ry Biesemeyer <yaauie@users.noreply.github.com>

github-actions · 2022-10-12T22:56:55Z

📃 DOCS PREVIEW ✨ https://logstash_14554.docs-preview.app.elstc.co/diff

yaauie

🤦🏼 s/Number/Numeric/g

qa/integration/specs/reload_config_spec.rb

qa/integration/specs/monitoring_api_spec.rb

Number should be Numeric in the ruby specs. Co-authored-by: Ry Biesemeyer <yaauie@users.noreply.github.com>

github-actions · 2022-10-12T23:37:35Z

📃 DOCS PREVIEW ✨ https://logstash_14554.docs-preview.app.elstc.co/diff

PR feedback for elastic#14554

github-actions · 2022-10-13T00:17:52Z

📃 DOCS PREVIEW ✨ https://logstash_14554.docs-preview.app.elstc.co/diff

mashhurs · 2022-10-13T00:24:21Z

Generated doc preview:

yaauie · 2022-10-13T04:38:13Z

logstash-core/spec/support/shared_contexts.rb

@@ -56,7 +56,7 @@ def block_until(limit_seconds, &condition)

  before :all do
    clear_data_dir
-    settings = mock_settings("config.reload.automatic" => true)
+    settings = mock_settings("config.reload.automatic" => true, "queue.type" => "persisted")


TODO: don't enable the PQ for all of the shared tests in order to validate behavior of the PQ in just one of those tests. I recently worked with this shared context so I can see if I can find a way to limit the scope in which we have to incur the cost of spooling up PQ's.

PR here -> mashhurs#4

Great approach to apply specific setting through the context:

include_context "api setup", {"queue.type" => "persisted"}

api specs: use PQ only where needed

github-actions · 2022-10-13T21:00:47Z

📃 DOCS PREVIEW ✨ https://logstash_14554.docs-preview.app.elstc.co/diff

yaauie

🎉 LGTM

* main: (131 commits) Collect queue growth events and bytes metrics when PQ is enabled. (elastic#14554) DRA: Handle env variables better (elastic#14644) Follow up PR of elastic#14645, adds version qualifier to the plain version variable (elastic#14646) Avoid to pass SNAPSHOT particle to the version passed to release-manager (elastic#14645) Disable -x in dra build scripts (elastic#14643) Enable debug for DRA shells scripts (elastic#14642) dra_upload.sh: Leave artifacts under build/ (elastic#14639) fix PipelineIR.getPostQueue by accounting for vertex copies (elastic#13621) Fix/dra use another technique to extract branch name (elastic#14636) Re-added execution rights to dra_upload.sh (elastic#14626) timestamp: respect locale's decimal-style when parsing (elastic#14628) Switch branch selector from major.minor to read the current branch name (elastic#14619) api: source pipelines that are fully-loaded (elastic#14595) Updates DRA scripts to build snapshot artifacts (elastic#14600) DRA - Update scripts to use the version qualifier in stack_version (elastic#14589) Adds new close method to Java's Filter API to be used to clean shutdown resources allocated by the filter during registration phase. (elastic#14485) Fix DLQ fails to start due to read 1 byte file (elastic#14605) Extract the branch name passed to release-manager from version file (elastic#14592) Extended Flow Metrics (elastic#14571) update ci release version (elastic#14598) ...

roaksoax added the status:work-in-progress label Sep 26, 2022

mashhurs commented Sep 26, 2022

View reviewed changes

logstash-core/src/main/java/org/logstash/execution/AbstractPipelineExt.java Outdated Show resolved Hide resolved

mashhurs self-assigned this Sep 27, 2022

mashhurs added enhancement v8.6.0 labels Sep 27, 2022

mashhurs changed the title ~~Collect growth events and bytes metrics if PQ is enabled.~~ Collect queue growth events and bytes metrics when PQ is enabled. Sep 27, 2022

mashhurs marked this pull request as ready for review September 28, 2022 20:46

mashhurs requested a review from yaauie September 28, 2022 20:46

roaksoax added status:needs-review and removed status:work-in-progress labels Sep 28, 2022

yaauie reviewed Sep 29, 2022

View reviewed changes

mashhurs mentioned this pull request Oct 5, 2022

Extended Flow Metrics #14571

Merged

4 tasks

mashhurs and others added 5 commits October 10, 2022 08:38

Collect growth events and bytes metrics if PQ is enabled: Java changes.

00bf51c

Move queue flow under queue namespace.

79f6f25

Pipeline level PQ flow metrics: add unit & integration tests.

efa6bbb

Include queue info in node stats sample.

0aee22b

Apply suggestions from code review

ab2cd72

Change uptime precision for PQ growth metrics to uptime seconds since PQ events are based on seconds. Co-authored-by: Ry Biesemeyer <yaauie@users.noreply.github.com>

mashhurs force-pushed the feature/flow-metrics-integration-pq branch from 0728109 to ab2cd72 Compare October 10, 2022 15:42

mashhurs linked an issue Oct 10, 2022 that may be closed by this pull request

Pipeline Health API - PQ Metrics #14556

Closed

mashhurs changed the base branch from feature/flow-metrics-integration-pq to main October 10, 2022 16:55

Add safeguard when using lazy delegating gauge type.

ebbd611

mashhurs commented Oct 10, 2022

View reviewed changes

docs/static/monitoring/monitoring-apis.asciidoc Outdated Show resolved Hide resolved

yaauie added 3 commits October 10, 2022 19:15

flow metrics: simplify generics of lazy implementation

ac12193

Enables interface `FlowMetrics::create` to take suppliers that _implement_ a `Metric<? extends Number>` instead of requiring them to be pre-cast, and avoid unnecessary exposure of the metrics value-type into our lazy init.

flow metrics: use lazy init for PQ gauge-based metrics

57e4f41

noop: use enum equality

1fcb75b

Avoids routing two enum values through `MetricType#toString()` and `String#equals()` when they can be compared directly.

yaauie and others added 2 commits October 12, 2022 17:58

flow metrics: move pipeline queue-based flows into pipeline flow name…

34a22a3

…space

Merge pull request #2 from yaauie/flow-metrics-pq-in-pipeline-namespace

358864a

Flow metrics pq in pipeline namespace

Follow up for moving PQ growth metrics under pipeline.*.flow.

57848bc

- Unit and integration tests are added or fixed. - Documentation added along with sample response data

mashhurs commented Oct 12, 2022

View reviewed changes

flow: pipeline pq flow rates docs

5332935

yaauie mentioned this pull request Oct 12, 2022

PR feedback for elastic/logstash#14554 mashhurs/logstash#3

Merged

yaauie reviewed Oct 12, 2022

View reviewed changes

logstash-core/spec/logstash/api/commands/stats_spec.rb Outdated Show resolved Hide resolved

yaauie reviewed Oct 12, 2022

View reviewed changes

qa/integration/specs/reload_config_spec.rb Outdated Show resolved Hide resolved

mashhurs and others added 2 commits October 12, 2022 15:40

Do not expect flow in the queue section of API. Metrics moved to flow…

2b074a8

… section. Update logstash-core/spec/logstash/api/commands/stats_spec.rb Co-authored-by: Ry Biesemeyer <yaauie@users.noreply.github.com>

Integration test failure fix.

3c72ff3

Mistake: `flow_status` should be `pipeline_flow_stats` Co-authored-by: Ry Biesemeyer <yaauie@users.noreply.github.com>

yaauie reviewed Oct 12, 2022

View reviewed changes

qa/integration/specs/reload_config_spec.rb Outdated Show resolved Hide resolved

qa/integration/specs/monitoring_api_spec.rb Outdated Show resolved Hide resolved

Integration test failures fix.

a188ab6

Number should be Numeric in the ruby specs. Co-authored-by: Ry Biesemeyer <yaauie@users.noreply.github.com>

mashhurs and others added 2 commits October 12, 2022 17:05

Make CI happy.

724852d

Merge pull request #3 from yaauie/flow-metrics-pq-docs

c6a772c

PR feedback for elastic#14554

mashhurs requested a review from yaauie October 13, 2022 00:39

yaauie reviewed Oct 13, 2022

View reviewed changes

api specs: use PQ only where needed

c89e369

yaauie mentioned this pull request Oct 13, 2022

api specs: use PQ only where needed mashhurs/logstash#4

Merged

Merge pull request #4 from yaauie/flow-metrics-fewer-shared-pq-tests

816ea2c

api specs: use PQ only where needed

yaauie approved these changes Oct 13, 2022

View reviewed changes

mashhurs merged commit f19e9cb into elastic:main Oct 13, 2022

roaksoax removed the status:changes-requested label Oct 13, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Collect queue growth events and bytes metrics when PQ is enabled. #14554

Collect queue growth events and bytes metrics when PQ is enabled. #14554

mashhurs commented Sep 26, 2022 •

edited

github-actions bot commented Sep 28, 2022

yaauie left a comment

github-actions bot commented Oct 4, 2022

cla-checker-service bot commented Oct 10, 2022 •

edited

github-actions bot commented Oct 10, 2022

github-actions bot commented Oct 10, 2022

github-actions bot commented Oct 12, 2022

github-actions bot commented Oct 12, 2022

mashhurs Oct 12, 2022

yaauie Oct 12, 2022

github-actions bot commented Oct 12, 2022

yaauie left a comment

github-actions bot commented Oct 12, 2022

github-actions bot commented Oct 13, 2022

mashhurs commented Oct 13, 2022

yaauie Oct 13, 2022

yaauie Oct 13, 2022

mashhurs Oct 13, 2022

github-actions bot commented Oct 13, 2022

yaauie left a comment

		@@ -534,6 +534,18 @@ A pipeline with only one single-threaded input may contribute up to 1.00, a pipe

		Additionally, some amount of back-pressure is both _normal_ and _expected_ for pipelines that are _pulling_ data, as this back-pressure allows them to slow down and pull data at a rate its downstream pipeline can tolerate.

		\| `queue_persisted_growth_bytes` \|

Collect queue growth events and bytes metrics when PQ is enabled. #14554

Collect queue growth events and bytes metrics when PQ is enabled. #14554

Conversation

mashhurs commented Sep 26, 2022 • edited

Release notes

What does this PR do?

Why is it important/What is the impact to the user?

Checklist

Author's Checklist

How to test this PR locally

Unit tests

Related issues

Use cases

Screenshots

Logs

github-actions bot commented Sep 28, 2022

yaauie left a comment

Choose a reason for hiding this comment

github-actions bot commented Oct 4, 2022

cla-checker-service bot commented Oct 10, 2022 • edited

github-actions bot commented Oct 10, 2022

github-actions bot commented Oct 10, 2022

github-actions bot commented Oct 12, 2022

github-actions bot commented Oct 12, 2022

mashhurs Oct 12, 2022

Choose a reason for hiding this comment

yaauie Oct 12, 2022

Choose a reason for hiding this comment

github-actions bot commented Oct 12, 2022

yaauie left a comment

Choose a reason for hiding this comment

github-actions bot commented Oct 12, 2022

github-actions bot commented Oct 13, 2022

mashhurs commented Oct 13, 2022

yaauie Oct 13, 2022

Choose a reason for hiding this comment

yaauie Oct 13, 2022

Choose a reason for hiding this comment

mashhurs Oct 13, 2022

Choose a reason for hiding this comment

github-actions bot commented Oct 13, 2022

yaauie left a comment

Choose a reason for hiding this comment

mashhurs commented Sep 26, 2022 •

edited

cla-checker-service bot commented Oct 10, 2022 •

edited