Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add refresh list of perf counters at every fetch #13344

Merged
merged 5 commits into from Sep 26, 2019

Conversation

narph
Copy link
Contributor

@narph narph commented Aug 26, 2019

In the perfmon metricset the list of performance counters is calculated when the metricset starts, then cached.
If users use wildcards in their counter queries they expect to see counter values for the processes/instances that match those queries as well.
For this a refresh of the list should be added (maybe at every Fetch).

Should fix #13091

How to test:

  • Add wildcard counter query (ex process)
  • While the perfmon metricset is running and collecting counter values start a new process instance
  • Observe if there are any new counter values returned for the just started process
  • Stop process
  • Observe if no more counter values related to the stopped process are coming through

@narph narph requested a review from a team as a code owner August 26, 2019 13:03
@narph narph self-assigned this Aug 26, 2019
@narph narph added Team:Integrations Label for the Integrations team :Windows [zube]: In Progress labels Aug 26, 2019
@prakash1243
Copy link

Hi @narph ,
Thanks so much for the PR, Could you please push this to the master, I'll take a nightly snapshot build and test it and let you know my findings. Thanks again. :)

metricbeat/module/windows/perfmon/pdh_query_windows.go Outdated Show resolved Hide resolved
metricbeat/module/windows/perfmon/perfmon.go Outdated Show resolved Hide resolved
metricbeat/module/windows/perfmon/reader.go Outdated Show resolved Hide resolved
metricbeat/module/windows/perfmon/perfmon.go Show resolved Hide resolved
childQueries, err := r.query.ExpandWildCardPath(counter.Query)
if err == nil && len(childQueries) >= 1 && !strings.Contains(childQueries[0], "*") {
for _, v := range childQueries {
if err := r.query.AddCounter(v, counter, len(childQueries) > 1); err != nil {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should we also remove counters at some moment to avoid leaks?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In the line below query.Close() I am running the PdhCloseQuery function which should close all counters contained in the specified query, closes all handles related to the query, and should free all memory associated with the query.
https://docs.microsoft.com/en-us/windows/win32/api/pdh/nf-pdh-pdhclosequery

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

But query.Close() is only executed if AddCounter fails. Shouldn't we do some cleanup of old counters during the normal operation?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I have removed the close query and added the option to compare counters and remove older ones.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks 👍

@narph narph requested a review from jsoriano September 23, 2019 16:18
metricbeat/module/windows/perfmon/perfmon.go Outdated Show resolved Hide resolved
metricbeat/module/windows/perfmon/perfmon.go Show resolved Hide resolved
childQueries, err := r.query.ExpandWildCardPath(counter.Query)
if err == nil && len(childQueries) >= 1 && !strings.Contains(childQueries[0], "*") {
for _, v := range childQueries {
if err := r.query.AddCounter(v, counter, len(childQueries) > 1); err != nil {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

But query.Close() is only executed if AddCounter fails. Shouldn't we do some cleanup of old counters during the normal operation?

if err == nil && len(childQueries) >= 1 && !strings.Contains(childQueries[0], "*") {
for _, v := range childQueries {
if err := r.query.AddCounter(v, counter, len(childQueries) > 1); err != nil {
r.query.Close()
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

RefreshCounterPaths() is run on every fetch after the first one. If AddCounter fails, r.query will be closed and never reopened. Will then all the subsequent fetches fail because the query is closed?
I think that we shouldn't close here, or if we do it, we should have some reopening logic during fetch.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good catch @jsoriano , I used to call OpenQuery at each refresh but moved it out, I have removed the close option.

@ruflin
Copy link
Member

ruflin commented Sep 24, 2019

@narph Sorry, missed this one. I'll leave the review to @jsoriano

@narph narph requested a review from jsoriano September 25, 2019 10:22
childQueries, err := r.query.ExpandWildCardPath(counter.Query)
if err == nil && len(childQueries) >= 1 && !strings.Contains(childQueries[0], "*") {
for _, v := range childQueries {
if err := r.query.AddCounter(v, counter, len(childQueries) > 1); err != nil {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks 👍

@narph
Copy link
Contributor Author

narph commented Sep 26, 2019

jenkins test this

@narph narph merged commit 6b9039a into elastic:master Sep 26, 2019
@narph narph deleted the perfmon-refresh-list branch September 26, 2019 10:23
@narph narph added v7.5.0 test-plan Add this PR to be manual test plan labels Oct 1, 2019
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Metricbeat Metricbeat module needs testing notes review Team:Integrations Label for the Integrations team test-plan Add this PR to be manual test plan v7.5.0 :Windows
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Couldn't get all the processes data running in Metricbeats Windows Module
5 participants