Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Update host metrics receiver to use scraper controller #1949

Conversation

james-bebbington
Copy link
Member

@james-bebbington james-bebbington commented Oct 13, 2020

Update host metrics receiver to use scraper controller removing around 400 lines of code. There is some "scraper factory" code left over that might be able to be moved into the receiverhelper (or scraper) library. Will think about that for next phase.

Resolves #918


Sample Output:

Note in these examples we see quite a lot of errors scraping process data due to not running the Collector in "Administrative mode" on Windows:

Traces (from http://localhost:55679/debug/tracez?zspanname=scraper%2fprocess%2fMetricsScraped&ztype=2&zsubtype=0):

image

Metrics (from http://localhost:8888/metrics):

image

@project-bot project-bot bot added this to In progress in Collector Oct 13, 2020
@codecov
Copy link

codecov bot commented Oct 13, 2020

Codecov Report

Merging #1949 into master will increase coverage by 0.09%.
The diff coverage is 100.00%.

Impacted file tree graph

@@            Coverage Diff             @@
##           master    #1949      +/-   ##
==========================================
+ Coverage   91.92%   92.01%   +0.09%     
==========================================
  Files         282      278       -4     
  Lines       16797    16746      -51     
==========================================
- Hits        15440    15409      -31     
+ Misses        933      921      -12     
+ Partials      424      416       -8     
Impacted Files Coverage Δ
...rnal/scraper/processesscraper/processes_scraper.go 77.77% <ø> (-2.23%) ⬇️
receiver/hostmetricsreceiver/internal/utils.go 100.00% <ø> (ø)
receiver/hostmetricsreceiver/factory.go 86.66% <100.00%> (+11.66%) ⬆️
...eceiver/internal/scraper/cpuscraper/cpu_scraper.go 100.00% <100.00%> (ø)
...icsreceiver/internal/scraper/cpuscraper/factory.go 100.00% <100.00%> (ø)
...nternal/scraper/diskscraper/disk_scraper_others.go 100.00% <100.00%> (ø)
...csreceiver/internal/scraper/diskscraper/factory.go 100.00% <100.00%> (ø)
...iver/internal/scraper/filesystemscraper/factory.go 88.88% <100.00%> (+3.17%) ⬆️
...al/scraper/filesystemscraper/filesystem_scraper.go 100.00% <100.00%> (ø)
...csreceiver/internal/scraper/loadscraper/factory.go 100.00% <100.00%> (ø)
... and 13 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 07d7a1f...98a76e5. Read the comment docs.

@james-bebbington james-bebbington force-pushed the hostmetrics-receiverhelpe- branch 2 times, most recently from 53f5f9f to 86d5ea2 Compare October 14, 2020 07:58
@james-bebbington james-bebbington marked this pull request as ready for review October 15, 2020 04:45
@james-bebbington james-bebbington force-pushed the hostmetrics-receiverhelpe- branch 6 times, most recently from a90bbee to 3a7f609 Compare October 15, 2020 11:19
Collector automation moved this from In progress to Done Oct 15, 2020
Collector automation moved this from Done to In progress Oct 15, 2020
@asuresh4
Copy link
Member

@james-bebbington do you think it makes sense to split this PR? Asking since there are changes in componenterror and obsreport apart from updates to host metrics receiver.

@james-bebbington
Copy link
Member Author

james-bebbington commented Oct 15, 2020

@james-bebbington do you think it makes sense to split this PR? Asking since there are changes in componenterror and obsreport apart from updates to host metrics receiver.

Yes definitely.

This PR actually depends on #1961 (the first commit of this PR with changes to componenterror, obsreport, etc is in #1961). If you want you can just review the second commit of this PR, but otherwise I would recommend reviewing #1961 first and waiting for that to be merged before reviewing this.

@github-actions
Copy link
Contributor

This PR was marked stale due to lack of activity. It will be closed in 7 days.

@github-actions github-actions bot added the Stale label Oct 23, 2020
@james-bebbington
Copy link
Member Author

james-bebbington commented Oct 29, 2020

This is now unblocked and ready for review again

@@ -135,7 +144,7 @@ func (s *scraper) getProcessMetadata() ([]*processMetadata, error) {

executable, err := getProcessExecutable(handle)
if err != nil {
errs = append(errs, fmt.Errorf("error reading process name for pid %v: %w", pid, err))
errs = append(errs, consumererror.NewPartialScrapeError(fmt.Errorf("error reading process name for pid %v: %w", pid, err), 1))
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Didn't think of this during the previous review but seeing it now a less verbose version might be

var errs consumererror.ScrapeErrors
errs.Add(fmt.Errorf("error reading process name for pid %v: %w", pid, err), 1))
errs.Add(...)
return errs.Combine()

Or something. Just an idea.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It is a bit ugly yea.

Another option might be to just make a fmt-like constructor for PartialScrapeError so we can do something like

errs = append(errs, consumererror.NewScrapeError(1, "error reading process name for pid %v: %w", pid, err))

WDYT?

In any case I will probably do in a follow up PR as this one is very large already

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Another option might be to just make a fmt-like constructor for PartialScrapeError so we can do something like

Yeah I think that'd be good. I was worried go vet might not catch fmt string errors but looks like it does from a quick test:

package main

import (
	"fmt"
)

func f(s string, args... interface{}) string {
   return fmt.Sprintf(s, args...)
  
}

func main() {
//	fmt.Printf("Hello, playground %s %s", "foo", 5)
	f("hello, playground %s %s", "foo", 3)
}

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

But I still think it'd be nice to have a struct to accumulate errors into instead of having to do all the appends. Could give it an Addf for formatted strings as well.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yea that seems like a good idea. Note we would also want a way to include regular unexpected errors that don't necessarily make sense to be represented as a scrape error.

Moved this discussion to an issue: #2046

Collector automation moved this from In progress to Reviewer approved Oct 29, 2020
@tigrannajaryan
Copy link
Member

OK to merge this?

@jrcamp
Copy link
Contributor

jrcamp commented Oct 30, 2020

OK to merge this?

@tigrannajaryan yes

@bogdandrutu bogdandrutu merged commit 5414c55 into open-telemetry:master Nov 1, 2020
Collector automation moved this from Reviewer approved to Done Nov 1, 2020
MovieStoreGuy pushed a commit to atlassian-forks/opentelemetry-collector that referenced this pull request Nov 11, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
No open projects
Collector
  
Done
Development

Successfully merging this pull request may close these issues.

Use obsreport.StartMetricsReceiveOp/EndMetricsReceiveOp in host scrapers
5 participants