ingest: processor stats #34724

jakelandis · 2018-10-22T21:17:42Z

Note - this change was originally merge with PR #34202, but was reverted due to a failing test

27c7077 has been introduced to fix the failing test. All other code has been reviewed (cherry-picked from before reverted)

This change introduces stats per processors. Total, time, failed,
current are currently supported. All pipelines will now show all
top level processors that belong to it. Failure processors are not
displayed, however, the time taken to execute the failure chain is part
of the stats for the top level processor.

The processor name is the type of the processor, ordered as defined in
the pipeline. If a tag for the processor is found, then the tag is
appended to the type.

Pipeline processors will have the pipeline name appended to the name of
the name of the processors (before the tag if one exists). If more
then one pipeline is used to process the document, then each pipeline
will carry its own stats. The outer most pipeline will also include the
inner most pipeline stats.

Conditional processors will only included in the stats if the condition evaluates
to true.

This change introduces stats per processors. Total, time, failed, current are currently supported. All pipelines will now show all top level processors that belong to it. Failure processors are not displayed, however, the time taken to execute the failure chain is part of the stats for the top level processor. The processor name is the type of the processor, ordered as defined in the pipeline. If a tag for the processor is found, then the tag is appended to the type. Pipeline processors will have the pipeline name appended to the name of the name of the processors (before the tag if one exists). If more then one pipeline is used to process the document, then each pipeline will carry its own stats. The outer most pipeline will also include the inner most pipeline stats. Conditional processors will only included in the stats if the condition evaluates to true.

elasticmachine · 2018-10-22T21:43:29Z

Pinging @elastic/es-core-infra

rjernst

This looks good, but the bwc behavior still needs a slight change, and as discussed offline, I think a unit test for the bwc behavior of IngestStats would be good (a simple round trip test with pre 6.5 version).

rjernst · 2018-10-22T21:43:44Z

server/src/main/java/org/elasticsearch/ingest/IngestStats.java

+            String pipelineId = in.readString();
+            Stats pipelineStat = new Stats(in);
+            this.pipelineStats.add(new PipelineStat(pipelineId, pipelineStat));
+            if (in.getVersion().onOrAfter(Version.V_7_0_0_alpha1)) {


This should be the version we are going back to, so 6.5. In this PR, disable bwc tests in the root build.gradle file. Then re-enable in the backport, and do a followup to master to re-enable there as well. This minimizes the changes necessary to followup to master (instead of needing to remember and change all the places that have this version constant).

jasontedor

Thanks so much @jakelandis. Given the code was previously reviewed, I reviewed the changes need to address the BWC failures. I am comfortable with those changes. LGTM.

rjernst

Thanks @jakelandis. LGTM.

This change introduces stats per processors. Total, time, failed, current are currently supported. All pipelines will now show all top level processors that belong to it. Failure processors are not displayed, however, the time taken to execute the failure chain is part of the stats for the top level processor. The processor name is the type of the processor, ordered as defined in the pipeline. If a tag for the processor is found, then the tag is appended to the type. Pipeline processors will have the pipeline name appended to the name of the name of the processors (before the tag if one exists). If more then one pipeline is used to process the document, then each pipeline will carry its own stats. The outer most pipeline will also include the inner most pipeline stats. Conditional processors will only included in the stats if the condition evaluates to true.

jakelandis · 2018-10-23T12:31:03Z

merge this PR
backport PR but with BWC enabled (a3c9423)
re-enable BWC tests (re-enable bwc tests #34743)

This change introduces stats per processors. Total, time, failed, current are currently supported. All pipelines will now show all top level processors that belong to it. Failure processors are not displayed, however, the time taken to execute the failure chain is part of the stats for the top level processor. The processor name is the type of the processor, ordered as defined in the pipeline. If a tag for the processor is found, then the tag is appended to the type. Pipeline processors will have the pipeline name appended to the name of the name of the processors (before the tag if one exists). If more then one pipeline is used to process the document, then each pipeline will carry its own stats. The outer most pipeline will also include the inner most pipeline stats. Conditional processors will only included in the stats if the condition evaluates to true.

* master: (24 commits) ingest: better support for conditionals with simulate?verbose (elastic#34155) [Rollup] Job deletion should be invoked on the allocated task (elastic#34574) [DOCS] .Security index is never auto created (elastic#34589) CCR: Requires soft-deletes on the follower (elastic#34725) re-enable bwc tests (elastic#34743) Empty GetAliases authorization fix (elastic#34444) INGEST: Document Processor Conditional (elastic#33388) [CCR] Add total fetch time leader stat (elastic#34577) SQL: Support pattern against compatible indices (elastic#34718) [CCR] Auto follow pattern APIs adjustments (elastic#34518) [Test] Remove dead code from ExceptionSerializationTests (elastic#34713) A small typo in migration-assistance doc (elastic#34704) ingest: processor stats (elastic#34724) SQL: Implement IN(value1, value2, ...) expression. (elastic#34581) Tests: Add checks to GeoDistanceQueryBuilderTests (elastic#34273) INGEST: Rename Pipeline Processor Param. (elastic#34733) Core: Move IndexNameExpressionResolver to java time (elastic#34507) [DOCS] Force Merge: clarify execution and storage requirements (elastic#33882) TESTING.asciidoc fix examples using forbidden annotation (elastic#34515) SQL: Implement `CONVERT`, an alternative to `CAST` (elastic#34660) ...

This change introduces stats per processors. Total, time, failed, current are currently supported. All pipelines will now show all top level processors that belong to it. Failure processors are not displayed, however, the time taken to execute the failure chain is part of the stats for the top level processor. The processor name is the type of the processor, ordered as defined in the pipeline. If a tag for the processor is found, then the tag is appended to the type. Pipeline processors will have the pipeline name appended to the name of the name of the processors (before the tag if one exists). If more then one pipeline is used to process the document, then each pipeline will carry its own stats. The outer most pipeline will also include the inner most pipeline stats. Conditional processors will only included in the stats if the condition evaluates to true.

jakelandis added 2 commits October 22, 2018 16:12

fix bwc issue

dbc6a77

jakelandis added :Data Management/Ingest Node Execution or management of Ingest Pipelines including GeoIP v7.0.0 v6.5.0 labels Oct 22, 2018

jakelandis requested a review from rjernst October 22, 2018 21:17

This was referenced Oct 22, 2018

ingest: processor stats #34202

Merged

ingest: better support for conditionals with simulate?verbose #34155

Merged

rjernst reviewed Oct 22, 2018

View reviewed changes

ingest: disable bwc tests, add bwc for 6.5 with tests

27c7077

jasontedor approved these changes Oct 23, 2018

View reviewed changes

rjernst approved these changes Oct 23, 2018

View reviewed changes

jakelandis merged commit ad94e79 into elastic:master Oct 23, 2018

jakelandis mentioned this pull request Oct 23, 2018

re-enable bwc tests #34743

Merged

colings86 added the >enhancement label Oct 25, 2018

Mpdreamz mentioned this pull request Dec 13, 2018

[meta] 6.5.0 Release elastic/elasticsearch-net#3457

Closed

Mpdreamz mentioned this pull request Jan 2, 2019

Add processor stats to node ingest stats elastic/elasticsearch-net#3529

Merged

colings86 added v7.0.0-beta1 and removed v7.0.0 labels Feb 7, 2019

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

ingest: processor stats #34724

ingest: processor stats #34724

jakelandis commented Oct 22, 2018 •

edited

Loading

elasticmachine commented Oct 22, 2018

rjernst left a comment

rjernst Oct 22, 2018

jasontedor left a comment

rjernst left a comment

jakelandis commented Oct 23, 2018 •

edited

Loading

ingest: processor stats #34724

ingest: processor stats #34724

Conversation

jakelandis commented Oct 22, 2018 • edited Loading

elasticmachine commented Oct 22, 2018

rjernst left a comment

Choose a reason for hiding this comment

rjernst Oct 22, 2018

Choose a reason for hiding this comment

jasontedor left a comment

Choose a reason for hiding this comment

rjernst left a comment

Choose a reason for hiding this comment

jakelandis commented Oct 23, 2018 • edited Loading

jakelandis commented Oct 22, 2018 •

edited

Loading

jakelandis commented Oct 23, 2018 •

edited

Loading