some profiles fail scraping #46

povilasv · 2020-04-28T04:23:48Z

conprof-0 conprof level=debug ts=2020-04-28T04:21:13.165040571Z caller=scrape.go:370 component=scrape-manager scrape_pool=conprof msg="scraping profile" url=http://10.200.2.38:8080/debug/pprof/trace
conprof-0 conprof level=debug ts=2020-04-28T04:21:14.18123509Z caller=scrape.go:505 component=scrape-manager scrape_pool=conprof target=http://10.200.2.38:8080/debug/pprof/trace msg="Scrape failed" err="failed to parse target's pprof profile: parsing profile: unrecognized profile format"

Same for cmdline profile:

conprof-0 conprof level=debug ts=2020-04-28T04:23:02.305811774Z caller=scrape.go:370 component=scrape-manager scrape_pool=conprof msg="scraping profile" url=http://10.200.0.177:8080/debug/pprof/cmdline
conprof-0 conprof level=debug ts=2020-04-28T04:23:02.310718078Z caller=scrape.go:505 component=scrape-manager scrape_pool=conprof target=http://10.200.0.177:8080/debug/pprof/cmdline msg="Scrape failed" err="failed to parse target's pprof profile: parsing profile: unrecognized profile format"

Might be a consequence of running on arm

The text was updated successfully, but these errors were encountered:

brancz · 2020-04-28T05:59:02Z

I assume subsequent scrapes are working?

povilasv · 2020-04-28T14:50:41Z

Nope, all of them fail

I'm currently just collecting pprof's from conprof

rl=http://10.200.0.177:8080/debug/pprof/cmdline
level=debug ts=2020-04-28T14:50:02.307342886Z caller=scrape.go:505 component=scrape-manager scrape_pool=conprof target=http://10.200.0.177:8080/debug/pprof/cmdline msg="Scrape failed" err="failed to parse target's pprof profile: parsing profile: unrecognized profile format"
level=debug ts=2020-04-28T14:50:03.419355382Z caller=scrape.go:370 component=scrape-manager scrape_pool=conprof msg="scraping profile" url=http://10.200.0.177:8080/debug/pprof/mutex
level=debug ts=2020-04-28T14:50:03.927836756Z caller=scrape.go:370 component=scrape-manager scrape_pool=conprof msg="scraping profile" url=http://10.200.0.177:8080/debug/pprof/threadcreate
level=debug ts=2020-04-28T14:50:04.271020553Z caller=scrape.go:370 component=scrape-manager scrape_pool=conprof msg="scraping profile" url=http://10.200.2.38:8080/debug/pprof/profile
level=debug ts=2020-04-28T14:50:13.165194934Z caller=scrape.go:370 component=scrape-manager scrape_pool=conprof msg="scraping profile" url=http://10.200.2.38:8080/debug/pprof/trace
level=debug ts=2020-04-28T14:50:14.171548533Z caller=scrape.go:505 component=scrape-manager scrape_pool=conprof target=http://10.200.2.38:8080/debug/pprof/trace msg="Scrape failed" err="failed to parse target's pprof profile: parsing profile: unrecognized profile format"

level=debug ts=2020-04-28T14:49:28.549548302Z caller=scrape.go:370 component=scrape-manager scrape_pool=conprof msg="scraping profile" url=http://10.200.0.177:8080/debug/pprof/profile
level=debug ts=2020-04-28T14:49:35.007156332Z caller=scrape.go:370 component=scrape-manager scrape_pool=conprof msg="scraping profile" url=http://10.200.0.177:8080/debug/pprof/block
level=debug ts=2020-04-28T14:49:35.862504319Z caller=scrape.go:370 component=scrape-manager scrape_pool=conprof msg="scraping profile" url=http://10.200.0.177:8080/debug/pprof/heap
level=debug ts=2020-04-28T14:49:43.345292018Z caller=scrape.go:370 component=scrape-manager scrape_pool=conprof msg="scraping profile" url=http://10.200.2.38:8080/debug/pprof/cmdline
level=debug ts=2020-04-28T14:49:43.348384784Z caller=scrape.go:505 component=scrape-manager scrape_pool=conprof target=http://10.200.2.38:8080/debug/pprof/cmdline msg="Scrape failed" err="failed to parse target's pprof profile: parsing profile: unrecognized profile format"
level=debug ts=2020-04-28T14:49:44.295231256Z caller=scrape.go:370 component=scrape-manager scrape_pool=conprof msg="scraping profile" url=http://10.200.2.38:8080/debug/pprof/block
level=debug ts=2020-04-28T14:49:51.409794607Z caller=scrape.go:370 component=scrape-manager scrape_pool=conprof msg="scraping profile" url=http://10.200.0.177:8080/debug/pprof/allocs
level=debug ts=2020-04-28T14:49:54.582743287Z caller=scrape.go:370 component=scrape-manager scrape_pool=conprof msg="scraping profile" url=http://10.200.0.177:8080/debug/pprof/trace
level=debug ts=2020-04-28T14:49:55.588346168Z caller=scrape.go:505 component=scrape-manager scrape_pool=conprof target=http://10.200.0.177:8080/debug/pprof/trace msg="Scrape failed" err="failed to parse target's pprof profile: parsing profile: unrecognized profile format"
level=debug ts=2020-04-28T14:49:59.116544328Z caller=scrape.go:370 component=scrape-manager scrape_pool=conprof msg="scraping profile" url=http://10.200.2.38:8080/debug/pprof/mutex
level=debug ts=2020-04-28T14:50:01.184449711Z caller=scrape.go:370 component=scrape-manager scrape_pool=conprof msg="scraping profile" url=http://10.200.2.38:8080/debug/pprof/allocs
level=debug ts=2020-04-28T14:50:02.305631715Z caller=scrape.go:370 component=scrape-manager scrape_pool=conprof msg="scraping profile" url=http://10.200.0.177:8080/debug/pprof/cmdline
level=debug ts=2020-04-28T14:50:02.307342886Z caller=scrape.go:505 component=scrape-manager scrape_pool=conprof target=http://10.200.0.177:8080/debug/pprof/cmdline msg="Scrape failed" err="failed to parse target's pprof profile: parsing profile: unrecognized profile format"
level=debug ts=2020-04-28T14:50:03.419355382Z caller=scrape.go:370 component=scrape-manager scrape_pool=conprof msg="scraping profile" url=http://10.200.0.177:8080/debug/pprof/mutex
level=debug ts=2020-04-28T14:50:03.927836756Z caller=scrape.go:370 component=scrape-manager scrape_pool=conprof msg="scraping profile" url=http://10.200.0.177:8080/debug/pprof/threadcreate
level=debug ts=2020-04-28T14:50:04.271020553Z caller=scrape.go:370 component=scrape-manager scrape_pool=conprof msg="scraping profile" url=http://10.200.2.38:8080/debug/pprof/profile
level=debug ts=2020-04-28T14:50:13.165194934Z caller=scrape.go:370 component=scrape-manager scrape_pool=conprof msg="scraping profile" url=http://10.200.2.38:8080/debug/pprof/trace
level=debug ts=2020-04-28T14:50:14.171548533Z caller=scrape.go:505 component=scrape-manager scrape_pool=conprof target=http://10.200.2.38:8080/debug/pprof/trace msg="Scrape failed" err="failed to parse target's pprof profile: parsing profile: unrecognized profile format"
level=debug ts=2020-04-28T14:50:16.881053375Z caller=scrape.go:370 component=scrape-manager scrape_pool=conprof msg="scraping profile" url=http://10.200.2.38:8080/debug/pprof/heap
level=debug ts=2020-04-28T14:50:19.178319771Z caller=scrape.go:370 component=scrape-manager scrape_pool=conprof msg="scraping profile" url=http://10.200.2.38:8080/debug/pprof/goroutine
level=debug ts=2020-04-28T14:50:19.324823635Z caller=scrape.go:370 component=scrape-manager scrape_pool=conprof msg="scraping profile" url=http://10.200.0.177:8080/debug/pprof/goroutine
level=debug ts=2020-04-28T14:50:25.937183977Z caller=scrape.go:370 component=scrape-manager scrape_pool=conprof msg="scraping profile" url=http://10.200.2.38:8080/debug/pprof/threadcreate
level=debug ts=2020-04-28T14:50:28.549563304Z caller=scrape.go:370 component=scrape-manager scrape_pool=conprof msg="scraping profile" ur

povilasv · 2020-04-28T14:54:04Z

This is the contents of http://localhost:8080/debug/pprof/cmdline from conprof pod:

cmdline

and trace http://localhost:8080/debug/pprof/trace:
trace

syepes · 2020-04-28T22:28:35Z

I have also just build the project from master and it not able to scrap itself nor any of my projects built with 1.14.2

Is there currently an issue/fix for this?

level=debug ts=2020-04-28T22:23:29.358497874Z caller=scrape.go:505 component=scrape-manager scrape_pool=conprof target=http://localhost:8080/debug/pprof/trace msg="Scrape failed" err=
"failed to parse target's pprof profile: parsing profile: unrecognized profile format"
level=debug ts=2020-04-28T22:23:31.105672907Z caller=scrape.go:370 component=scrape-manager scrape_pool=conprof msg="scraping profile" url=http://localhost:8080/debug/pprof/block
level=debug ts=2020-04-28T22:23:31.309165448Z caller=scrape.go:370 component=scrape-manager scrape_pool=conprof msg="scraping profile" url=http://localhost:8080/debug/pprof/allocs
level=debug ts=2020-04-28T22:23:36.404392335Z caller=scrape.go:370 component=scrape-manager scrape_pool=conprof msg="scraping profile" url=http://localhost:8080/debug/pprof/profile
level=debug ts=2020-04-28T22:23:51.891763991Z caller=scrape.go:370 component=scrape-manager scrape_pool=conprof msg="scraping profile" url=http://localhost:8080/debug/pprof/goroutine
level=debug ts=2020-04-28T22:23:52.290836201Z caller=scrape.go:370 component=scrape-manager scrape_pool=conprof msg="scraping profile" url=http://localhost:8080/debug/pprof/cmdline
level=debug ts=2020-04-28T22:23:52.291465509Z caller=scrape.go:505 component=scrape-manager scrape_pool=conprof target=http://localhost:8080/debug/pprof/cmdline msg="Scrape failed" er
r="failed to parse target's pprof profile: parsing profile: unrecognized profile format"
level=debug ts=2020-04-28T22:24:15.571826594Z caller=scrape.go:370 component=scrape-manager scrape_pool=conprof msg="scraping profile" url=http://localhost:8080/debug/pprof/mutex
level=debug ts=2020-04-28T22:24:18.492315323Z caller=scrape.go:370 component=scrape-manager scrape_pool=conprof msg="scraping profile" url=http://localhost:8080/debug/pprof/heap
level=debug ts=2020-04-28T22:24:26.875040226Z caller=scrape.go:370 component=scrape-manager scrape_pool=conprof msg="scraping profile" url=http://localhost:8080/debug/pprof/threadcrea
te
level=debug ts=2020-04-28T22:24:28.357037155Z caller=scrape.go:370 component=scrape-manager scrape_pool=conprof msg="scraping profile" url=http://localhost:8080/debug/pprof/trace
level=debug ts=2020-04-28T22:24:29.358824086Z caller=scrape.go:505 component=scrape-manager scrape_pool=conprof target=http://localhost:8080/debug/pprof/trace msg="Scrape failed" err=
"failed to parse target's pprof profile: parsing profile: unrecognized profile format"

povilasv · 2020-04-29T03:58:51Z

So basically there is no support for traces in google/pprof#529 and won't be.

There is a way to convert traces to pprof using go tool trace -pprof=TYPE trace.out > TYPE.pprof.
Types:

- net: network blocking profile
- sync: synchronization blocking profile
- syscall: syscall blocking profile
- sched: scheduler latency profile

One option is to convert trace to pprof all 4 types, which would mean roughly copy paste & adapt code https://golang.org/src/cmd/trace/pprof.go here.

Another option actually support golang traces and build UI on top of that? (The way go tool does it now).

Also question what we can do with cmdline?

povilasv · 2020-04-29T04:06:36Z

IMO supporting go tool trace is a better option as the go tool trace is quite useful:

So cmdline isn't really a profile, it's command line arguments, so maybe we just get rid of it? I.e:
/bin/conprof^@ALL^@--log.level=debug^@--storage.tsdb.path=/conprof^@--config.file=/etc/conprof/conprof.yaml

syepes · 2020-04-29T06:32:20Z

@povilasv
Interesting, but did you get it working on any of the profiles? I have been actively spacing 4 apps and in the UI I don't see anything at all

povilasv · 2020-04-29T07:19:31Z

you need to add query expression, like {job="conprof"}, as the landing page doesn't show you anything without expression.

brancz · 2020-04-29T07:39:36Z

@povilasv both of those suggestions sound good to me (remove cmdline, embed trace UI).

syepes · 2020-04-29T09:17:28Z

@povilasv Thanks for the tip! it's now working :-)

povilasv · 2020-05-10T03:46:24Z

I've made some progress 🕺

My proposal to move this forward is:

We copy go's internal/trace package to internal/trace, leave the License headers as is.
We add a new profile_type label to all profiles. This is need to figure out which UI to show + new you can query for specific type of profile :)
We add a new internal/traceui package which roughly copies /usr/lib/go/src/cmd/trace, but instead of using global variable, which stores parsed profile we change it to actually take it as an argument.

Still needs some work:

Network blocking profile
Synchronization blocking profile
Syscall blocking profile
Scheduler latency profile

As internally go tool just converts them to pprof and outputs a svg. Maybe I can just use pprof UI instead.

My approach would be we add those internal packages to the repo incrementally and expose UIs to users little by little.

Thoughts?

brancz · 2020-05-10T06:26:29Z

I don't mind copying the internal code, I've started doing this on a branch as well which convert profile nodes into Prometheus remote read.

We add a new internal/traceui package which roughly copies /usr/lib/go/src/cmd/trace, but instead of using global variable, which stores parsed profile we change it to actually take it as an argument.

I would say even if we copy the internal package, let's try contributing this upstream as we're going to want to stay as close to upstream as possible, making copying of internal essentially vendoring.

povilasv · 2020-05-10T08:26:34Z

I don't mind copying the internal code, I've started doing this on a branch as well which convert profile nodes into Prometheus remote read.

We add a new internal/traceui package which roughly copies /usr/lib/go/src/cmd/trace, but instead of using global variable, which stores parsed profile we change it to actually take it as an argument.

I would say even if we copy the internal package, let's try contributing this upstream as we're going to want to stay as close to upstream as possible, making copying of internal essentially vendoring.

Agree, I've created an issue golang/go#38971 if Go maintainers agree, I would try to help them push this forward.

povilasv · 2020-05-10T08:28:56Z

BTW I actually made everything work 👯

brancz · 2020-05-10T08:30:43Z

Nice! I don't even think it needs to necessarily be a non internal package, as long as we don't modify the "manually vendored" one. That said if it was public then even better! (but I would understand that it's quite a large API to give any sort of guarantees on)

brancz · 2020-05-10T08:36:33Z

In an attempt to not duplicate work, I opened a WIP for the remote read work I had laying around as that also did the internal manual vendoring: #54

brancz · 2020-05-11T15:28:13Z

Cycling back to the profile type proposal. I think something like type metadata in Prometheus would be more appropriate here. I think I'm ok with a stop gap (maybe analysing the profile scrape path) to heuristically decide what profile it probably is (kind of like Prometheus interpreting _bytes suffix metric names).

povilasv · 2020-05-12T02:38:01Z

What about users wanting to query for specific profiles? I can definitely see myself asking conprof "show me memory profiles for jobs conprof,traefik" . Right now we could only do it via paths.

But what if my path is different for different types of apps? For example I can can easily configure HeapConfig job to scrape /pprof/heap for one application and /debug/heap for another. (FYI this is actually the case in the company I work for, as we have internal conventions for app debug endpoints and non custome made apps like traefik, etc have different conventions )

We already have specific types of config for each different profile, so it makes sense to me to have a type also.

For Prometheus it's a bit different as you have a metric name and you can query just that metric.

Maybe it could even be a profile name?
Example:

{__address__="10.200.3.81:8080",profile_type="allocs",__scheme__="http",container="conprof",instance="10.200.3.81:8080",job="conprof",namespace="sys-mon",pod="conprof-0",profile_path="/debug/pprof/allocs",}

vs

allocs{__address__="10.200.3.81:8080",__scheme__="http",container="conprof",instance="10.200.3.81:8080",job="conprof",namespace="sys-mon",pod="conprof-0",profile_path="/debug/pprof/allocs"}

Thoughts?

syepes · 2020-05-12T06:07:48Z

IMHO, I kind of like the profile name / type solution as the queries won't change from app to app.

brancz · 2020-05-12T07:54:56Z

I think I like the ergonomics of the profile type/name being the entrypiont for querying. I think we're going to have to adjust the query language/API for this slightly.

povilasv · 2020-05-16T02:59:34Z

managed to make it work :)

povilasv · 2020-05-20T08:43:56Z

As Add Profile Type got merged, we can now scrape traces, so closing this. I've opened a seperate issue for UI #61

*: Split pprof into individual profiles by sample index

povilasv mentioned this issue Apr 28, 2020

Go 1.14 cmdline & trace profiles parse error on arm/arm64 google/pprof#529

Closed

This was referenced May 1, 2020

Remove cmdline profile scraping #48

Merged

[WIP] Add scrape go trace profile as bytes #50

Closed

povilasv mentioned this issue May 9, 2020

proposal: runtime: expose internal/trace package golang/go#38971

Closed

This was referenced May 10, 2020

Add internal/trace from golang #55

Merged

[WIP] Add traceui package from go cmd/trace #56

Closed

povilasv mentioned this issue May 16, 2020

Add Profile Type #59

Merged

povilasv closed this as completed May 20, 2020

brancz added a commit that referenced this issue Oct 5, 2021

Merge pull request #46 from parca-dev/split-pprof

1a20864

*: Split pprof into individual profiles by sample index

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

some profiles fail scraping #46

some profiles fail scraping #46

povilasv commented Apr 28, 2020 •

edited

brancz commented Apr 28, 2020

povilasv commented Apr 28, 2020 •

edited

povilasv commented Apr 28, 2020 •

edited

syepes commented Apr 28, 2020

povilasv commented Apr 29, 2020 •

edited

povilasv commented Apr 29, 2020 •

edited

syepes commented Apr 29, 2020

povilasv commented Apr 29, 2020

brancz commented Apr 29, 2020

syepes commented Apr 29, 2020

povilasv commented May 10, 2020

brancz commented May 10, 2020

povilasv commented May 10, 2020 •

edited

povilasv commented May 10, 2020

brancz commented May 10, 2020 •

edited

brancz commented May 10, 2020

brancz commented May 11, 2020 •

edited

povilasv commented May 12, 2020 •

edited

syepes commented May 12, 2020

brancz commented May 12, 2020

povilasv commented May 16, 2020

povilasv commented May 20, 2020

some profiles fail scraping #46

some profiles fail scraping #46

Comments

povilasv commented Apr 28, 2020 • edited

brancz commented Apr 28, 2020

povilasv commented Apr 28, 2020 • edited

povilasv commented Apr 28, 2020 • edited

syepes commented Apr 28, 2020

povilasv commented Apr 29, 2020 • edited

povilasv commented Apr 29, 2020 • edited

syepes commented Apr 29, 2020

povilasv commented Apr 29, 2020

brancz commented Apr 29, 2020

syepes commented Apr 29, 2020

povilasv commented May 10, 2020

brancz commented May 10, 2020

povilasv commented May 10, 2020 • edited

povilasv commented May 10, 2020

brancz commented May 10, 2020 • edited

brancz commented May 10, 2020

brancz commented May 11, 2020 • edited

povilasv commented May 12, 2020 • edited

syepes commented May 12, 2020

brancz commented May 12, 2020

povilasv commented May 16, 2020

povilasv commented May 20, 2020

povilasv commented Apr 28, 2020 •

edited

povilasv commented Apr 28, 2020 •

edited

povilasv commented Apr 28, 2020 •

edited

povilasv commented Apr 29, 2020 •

edited

povilasv commented Apr 29, 2020 •

edited

povilasv commented May 10, 2020 •

edited

brancz commented May 10, 2020 •

edited

brancz commented May 11, 2020 •

edited

povilasv commented May 12, 2020 •

edited