Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Infrastructure UI][Rules] Add logging to Metric Threshold Rule #132343

Merged

Conversation

simianhacker
Copy link
Member

Summary

This PR adds logging for the execution time (debug) along with the Elasticsearch request/responses (trace) to the Metric Threshold Rule.

To enable the logging for the Metric Threshold rule, add the following snippet to your config/kibana.dev.yml:

logging.loggers:
  - name: plugins.infra.metricThresholdRule
    level: trace

Example Log Format

[2022-05-17T09:58:20.209-06:00][TRACE][plugins.infra.metricThresholdRule] [AlertId: cbf86ed0-d5f6-11ec-8e5e-efda79ee6822][ExecutionId: 33745d86-4f21-4cf2-805a-4ca2b9528805] Request: {"index":"metrics-*,metricbeat-*","allow_no_indices":true,"ignore_unavailable":true,"body":{"track_total_hits":true,"query":{"bool":{"filter":[{"range":{"@timestamp":{"gte":"2022-05-17T15:56:17.079Z","lte":"2022-05-17T15:58:20.084Z"}}},{"exists":{"field":"system.cpu.total.norm.pct"}}]}},"size":0,"aggs":{"groupings":{"composite":{"size":10000,"sources":[{"groupBy0":{"terms":{"field":"host.name"}}}],"after":{"groupBy0":"host-4"}},"aggs":{"currentPeriod":{"filter":{"range":{"@timestamp":{"gte":"2022-05-17T15:57:20.084Z","lte":"2022-05-17T15:58:20.084Z"}}},"aggs":{"aggregatedValue":{"avg":{"field":"system.cpu.total.norm.pct"}}}},"shouldWarn":{"bucket_script":{"buckets_path":{},"script":"0"}},"shouldTrigger":{"bucket_script":{"buckets_path":{"value":"currentPeriod>aggregatedValue"},"script":"params.value > 0.01 ? 1 : 0"}},"lastPeriod":{"filter":{"range":{"@timestamp":{"gte":"2022-05-17T15:56:17.079Z","lte":"2022-05-17T15:57:17.079Z"}}}},"missingGroup":{"bucket_script":{"buckets_path":{"lastPeriod":"lastPeriod>_count","currentPeriod":"currentPeriod>_count"},"script":"params.lastPeriod > 0 && params.currentPeriod < 1 ? 1 : 0"}},"newOrRecoveredGroup":{"bucket_script":{"buckets_path":{"lastPeriod":"lastPeriod>_count","currentPeriod":"currentPeriod>_count"},"script":"params.lastPeriod < 1 && params.currentPeriod > 0 ? 1 : 0"}},"evaluation":{"bucket_selector":{"buckets_path":{"shouldWarn":"shouldWarn","shouldTrigger":"shouldTrigger","missingGroup":"missingGroup","newOrRecoveredGroup":"newOrRecoveredGroup"},"script":"(params.missingGroup != null && params.missingGroup > 0) || (params.shouldWarn != null && params.shouldWarn > 0) || (params.shouldTrigger != null && params.shouldTrigger > 0) || (params.newOrRecoveredGroup != null && params.newOrRecoveredGroup > 0)"}}}}}}}
[2022-05-17T09:58:20.211-06:00][TRACE][plugins.infra.metricThresholdRule] [AlertId: cbf86ed0-d5f6-11ec-8e5e-efda79ee6822][ExecutionId: 33745d86-4f21-4cf2-805a-4ca2b9528805] Response: {"took":0,"timed_out":false,"_shards":{"total":1,"successful":1,"skipped":0,"failed":0},"hits":{"total":{"value":60,"relation":"eq"},"max_score":null,"hits":[]},"aggregations":{"groupings":{"buckets":[]}}}
[2022-05-17T09:58:20.213-06:00][DEBUG][plugins.infra.metricThresholdRule] [AlertId: cbf86ed0-d5f6-11ec-8e5e-efda79ee6822][ExecutionId: 33745d86-4f21-4cf2-805a-4ca2b9528805] Scheduled 5 actions in 12ms

Checklist

@simianhacker simianhacker requested a review from a team as a code owner May 17, 2022 16:00
@simianhacker simianhacker marked this pull request as draft May 17, 2022 16:00
@simianhacker simianhacker force-pushed the add-logging-to-metric-threshold-rule branch 4 times, most recently from e846af4 to 5461779 Compare May 17, 2022 16:20
@simianhacker simianhacker added Feature:Metrics UI Metrics UI feature Team:Infra Monitoring UI - DEPRECATED DEPRECATED - Label for the Infra Monitoring UI team. Use Team:obs-ux-infra_services Team: Actionable Observability - DEPRECATED For Observability Alerting and SLOs use "Team:obs-ux-management", for AIops "Team:obs-knowledge" v8.3.0 release_note:enhancement labels May 17, 2022
@simianhacker simianhacker force-pushed the add-logging-to-metric-threshold-rule branch from 5461779 to 5b274a3 Compare May 17, 2022 16:24
@simianhacker simianhacker force-pushed the add-logging-to-metric-threshold-rule branch from 5b274a3 to d537560 Compare May 17, 2022 20:54
@kibana-ci
Copy link
Collaborator

💚 Build Succeeded

Metrics [docs]

✅ unchanged

History

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

@simianhacker simianhacker marked this pull request as ready for review May 17, 2022 21:56
@elasticmachine
Copy link
Contributor

Pinging @elastic/infra-monitoring-ui (Team:Infra Monitoring UI)

Copy link
Contributor

@matschaffer matschaffer left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Got some logs LGTM

[2022-05-18T10:32:46.473+09:00][TRACE][plugins.infra.metricThresholdRule] [AlertId: 69402c20-d64a-11ec-840b-fdd156179a03][ExecutionId: 8005ed8e-8c4f-46df-a4e6-cf9e8c65e14e] Request: {"index":"metrics-*,metricbeat-*","allow_no_indices":true,"ignore_unavailable":true,"body":{"track_total_hits":true,"query":{"bool":{"filter":[{"range":{"@timestamp":{"gte":"2022-05-18T01:31:46.339Z","lte":"2022-05-18T01:32:46.339Z"}}},{"exists":{"field":"system.cpu.total.pct"}}]}},"size":0,"aggs":{"all":{"filters":{"filters":{"all":{"match_all":{}}}},"aggs":{"currentPeriod":{"filter":{"range":{"@timestamp":{"gte":"2022-05-18T01:31:46.339Z","lte":"2022-05-18T01:32:46.339Z"}}},"aggs":{"aggregatedValue":{"avg":{"field":"system.cpu.total.pct"}}}},"shouldWarn":{"bucket_script":{"buckets_path":{},"script":"0"}},"shouldTrigger":{"bucket_script":{"buckets_path":{"value":"currentPeriod>aggregatedValue"},"script":"params.value > 0.3 ? 1 : 0"}}}}}}}
[2022-05-18T10:32:46.485+09:00][TRACE][plugins.infra.metricThresholdRule] [AlertId: 69402c20-d64a-11ec-840b-fdd156179a03][ExecutionId: 8005ed8e-8c4f-46df-a4e6-cf9e8c65e14e] Response: {"took":8,"timed_out":false,"_shards":{"total":1,"successful":1,"skipped":0,"failed":0},"hits":{"total":{"value":6,"relation":"eq"},"max_score":null,"hits":[]},"aggregations":{"all":{"buckets":{"all":{"doc_count":6,"currentPeriod":{"doc_count":6,"aggregatedValue":{"value":0.11966666666666666}},"shouldWarn":{"value":0},"shouldTrigger":{"value":0}}}}}}

error: fakeLogger,
fatal: fakeLogger,
log: sinon.stub(),
get: sinon.stub(),
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I've been trying to avoid sinon in favor of jest.fn in my tests but looks like this isn't really net-new code so LGTM.

@@ -240,6 +246,8 @@ export const createMetricThresholdExecutor = (libs: InfraBackendLibs) =>
});
}
}
const stopTime = Date.now();
logger.debug(`Scheduled ${scheduledActionsCount} actions in ${stopTime - startTime}ms`);
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This makes me think about APM tracing :) Guess logging is probably a more critical base feature though.

@simianhacker simianhacker merged commit 3e837ca into elastic:main May 18, 2022
@kibanamachine kibanamachine added the backport:skip This commit does not require backporting label May 18, 2022
@simianhacker simianhacker deleted the add-logging-to-metric-threshold-rule branch April 17, 2024 15:30
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
backport:skip This commit does not require backporting Feature:Metrics UI Metrics UI feature release_note:enhancement Team: Actionable Observability - DEPRECATED For Observability Alerting and SLOs use "Team:obs-ux-management", for AIops "Team:obs-knowledge" Team:Infra Monitoring UI - DEPRECATED DEPRECATED - Label for the Infra Monitoring UI team. Use Team:obs-ux-infra_services v8.3.0
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

5 participants