Hidden flag to disable address normalization for debugging #2955

marselester · 2023-04-07T15:27:07Z

This flag mirrors parca-dev/parca-agent#1534 so we should be able to test the changes without interfering with release cycles on both sides.

marselester · 2023-04-07T15:37:51Z

As a next step (not in this PR) I consider adding isNormalized flag to the table, though I haven't looked at the storage layer yet. Maybe my assumptions are naive and changes to schema could ruin existing Parca setups. Another option is to write non-normalized samples into a separate table. Please let me know what you think since you have a better understanding how Parca works. Here is a PR introducing Location.is_raw_address flag #2978.

pkg/parca/parca.go

Co-authored-by: Kemal Akkoyun <kakkoyun@users.noreply.github.com>

kakkoyun

This looks good to me.

I have commented.

kakkoyun · 2023-05-16T10:43:08Z

pkg/parcacol/normalizer.go

@@ -301,7 +305,7 @@ func (n *MetastoreNormalizer) NormalizeLocations(
 			mappingIndex := location.MappingId - 1
 			mappingNormalizationInfo := mappings[mappingIndex]

-			if !normalizedAddress {
+			if n.isAddrNormEnabled && !normalizedAddress {


My only concern with this PR is that we still need to test this path.
Have you tested it with a modified agent where we disable normalization, and is this execution path actually called without an issue? And everything symbolized? If so, could you add some screenshots or logs?

Sure, I will prepare that.

Thanks 🙏

I've added a print statement in the branch in question to see if the addresses get normalized on Parca side.
As you can see in the Parca logs, the addresses are printed.

Parca logs

./parca --debug-normalize-addresses=true ooooooooo. `888 `Y88. 888 .d88' .oooo. oooo d8b .ooooo. .oooo. 888ooo88P' `P )88b `888""8P d88' `"Y8 `P )88b 888 .oP"888 888 888 .oP"888 888 d8( 888 888 888 .o8 d8( 888 o888o `Y888""8o d888b `Y8bod8P' `Y888""8o level=info name=parca ts=2023-05-16T14:21:25.835301051Z caller=factory.go:52 msg="loading bucket configuration" level=info name=parca ts=2023-05-16T14:21:25.844258186Z caller=badger.go:53 msg="Set nextTxnTs to 0" level=info name=parca ts=2023-05-16T14:21:25.848055958Z caller=server.go:93 msg="starting server" addr=:7070 normalizing address 18446744072327227653 normalizing address 18446744072314052922 normalizing address 18446744072314053552 normalizing address 18446744072324153195 normalizing address 18446744072325169889 normalizing address 18446744072325173648 normalizing address 18446744072325231847 normalizing address 18446744072325244862 normalizing address 18446744072325020508 normalizing address 18446744072325021112 normalizing address 18446744072325021245 normalizing address 18446744072325022623 normalizing address 18446744072324333674 normalizing address 18446744072324333765 normalizing address 18446744072324334318 normalizing address 18446744072324328867 normalizing address 18446744072324329862 normalizing address 18446744072329625817 normalizing address 18446744072313656941 normalizing address 18446744072313657076 normalizing address 18446744072325037201 normalizing address 18446744072325040711 normalizing address 18446744072325040958 normalizing address 18446744072325041224 normalizing address 18446744072325045726 normalizing address 18446744072325046132 normalizing address 18446744072325046853 normalizing address 18446744072325195264 normalizing address 18446744072325201049 normalizing address 18446744072325202983 normalizing address 18446744072325099545 normalizing address 18446744072325103178 normalizing address 18446744072325104237 normalizing address 18446744072325368771 normalizing address 18446744072324129570 normalizing address 18446744072324129731 normalizing address 18446744072316572960 normalizing address 18446744072316575893 normalizing address 18446744072316586229 normalizing address 18446744072316586313 normalizing address 18446744072327150332 normalizing address 18446744072327528601 normalizing address 4212494 normalizing address 4754504 normalizing address 4747013 normalizing address 5432549 normalizing address 6141481 normalizing address 6214277 normalizing address 6290121 normalizing address 9878498 normalizing address 9878257 normalizing address 9596583 normalizing address 9601509 normalizing address 9747687 normalizing address 9739166 normalizing address 9779173 normalizing address 4659393 normalizing address 4469945 normalizing address 4467133 normalizing address 4472689 normalizing address 4473997 ...

Parca Agent logs

sudo ./dist/parca-agent --debug-normalize-addresses=false --remote-store-address=:7070 --remote-store-insecure ooooooooo. .o. . `888 `Y88. .888. .o8 888 .d88' .oooo. oooo d8b .ooooo. .oooo. .8"888. .oooooooo .ooooo. ooo. .oo. .o888oo 888ooo88P' `P )88b `888""8P d88' `"Y8 `P )88b .8' `888. 888' `88b d88' `88b `888P"Y88b 888 888 .oP"888 888 888 .oP"888 .88ooo8888. 888 888 888ooo888 888 888 888 888 d8( 888 888 888 .o8 d8( 888 .8' `888. `88bod8P' 888 .o 888 888 888 . o888o `Y888""8o d888b `Y8bod8P' `Y888""8o o88o o8888o `8oooooo. `Y8bod8P' o888o o888o "888" d" YD "Y88888P' level=info name=parca-agent ts=2023-05-16T14:30:08.026438748Z caller=main.go:292 msg="maxprocs: Leaving GOMAXPROCS=2: CPU quota undefined" level=info name=parca-agent ts=2023-05-16T14:30:08.041058238Z caller=main.go:334 msg="eBPF is supported and enabled by the host kernel" name=parca-agent ts=2023-05-16T14:30:08.041674708Z caller=main.go:396 msg=starting... node=ubuntu-jammy store=:7070 level=info name=parca-agent ts=2023-05-16T14:30:08.041984627Z caller=main.go:499 msg=rlimit cur=1048576 max=1048575 level=info name=parca-agent ts=2023-05-16T14:30:08.04832702Z caller=cpu.go:224 msg="Attempting to create unwind shards" count=50

I also stumbled on a panic in the agent (it's built from the main branch). I will have a look what's going on there.

agent panic

$ sudo ./dist/parca-agent --debug-normalize-addresses=false --remote-store-address=:7070 --remote-store-insecure ooooooooo. .o. . `888 `Y88. .888. .o8 888 .d88' .oooo. oooo d8b .ooooo. .oooo. .8"888. .oooooooo .ooooo. ooo. .oo. .o888oo 888ooo88P' `P )88b `888""8P d88' `"Y8 `P )88b .8' `888. 888' `88b d88' `88b `888P"Y88b 888 888 .oP"888 888 888 .oP"888 .88ooo8888. 888 888 888ooo888 888 888 888 888 d8( 888 888 888 .o8 d8( 888 .8' `888. `88bod8P' 888 .o 888 888 888 . o888o `Y888""8o d888b `Y8bod8P' `Y888""8o o88o o8888o `8oooooo. `Y8bod8P' o888o o888o "888" d" YD "Y88888P' level=info name=parca-agent ts=2023-05-16T14:30:08.026438748Z caller=main.go:292 msg="maxprocs: Leaving GOMAXPROCS=2: CPU quota undefined" level=info name=parca-agent ts=2023-05-16T14:30:08.041058238Z caller=main.go:334 msg="eBPF is supported and enabled by the host kernel" name=parca-agent ts=2023-05-16T14:30:08.041674708Z caller=main.go:396 msg=starting... node=ubuntu-jammy store=:7070 level=info name=parca-agent ts=2023-05-16T14:30:08.041984627Z caller=main.go:499 msg=rlimit cur=1048576 max=1048575 level=info name=parca-agent ts=2023-05-16T14:30:08.04832702Z caller=cpu.go:224 msg="Attempting to create unwind shards" count=50 level=warn name=parca-agent ts=2023-05-16T14:31:48.542377961Z caller=cpu.go:324 msg="lost events" count=0 level=warn name=parca-agent ts=2023-05-16T14:31:48.542643804Z caller=cpu.go:324 msg="lost events" count=0 level=warn name=parca-agent ts=2023-05-16T14:31:48.54265841Z caller=cpu.go:324 msg="lost events" count=0 level=warn name=parca-agent ts=2023-05-16T14:31:48.542762117Z caller=cpu.go:324 msg="lost events" count=0 level=warn name=parca-agent ts=2023-05-16T14:31:48.542814513Z caller=cpu.go:324 msg="lost events" count=0 level=warn name=parca-agent ts=2023-05-16T14:31:48.542858404Z caller=cpu.go:324 msg="lost events" count=0 level=warn name=parca-agent ts=2023-05-16T14:31:48.54287753Z caller=cpu.go:324 msg="lost events" count=0 level=warn name=parca-agent ts=2023-05-16T14:31:48.542886802Z caller=cpu.go:324 msg="lost events" count=0 level=warn name=parca-agent ts=2023-05-16T14:31:48.542894348Z caller=cpu.go:324 msg="lost events" count=0 level=warn name=parca-agent ts=2023-05-16T14:31:48.542904881Z caller=cpu.go:324 msg="lost events" count=0 level=warn name=parca-agent ts=2023-05-16T14:31:48.542953281Z caller=cpu.go:324 msg="lost events" count=0 level=warn name=parca-agent ts=2023-05-16T14:31:48.542967188Z caller=cpu.go:324 msg="lost events" count=0 panic: runtime error: invalid memory address or nil pointer dereference [signal SIGSEGV: segmentation violation code=0x1 addr=0x20 pc=0x1a98c56] goroutine 135 [running]: github.com/parca-dev/parca-agent/pkg/symbol.(*Symbolizer).Symbolize(0xc00036fb30, 0xc00071fa40) github.com/parca-dev/parca-agent/pkg/symbol/symbol.go:82 +0x536 github.com/parca-dev/parca-agent/pkg/profiler/cpu.(*CPU).Run(0xc0004c7180, {0x23bdcd8, 0xc0007e7740}) github.com/parca-dev/parca-agent/pkg/profiler/cpu/cpu.go:556 +0x1788 main.run.func11.1({0x23bdcd8?, 0xc0007e7740?}) github.com/parca-dev/parca-agent/cmd/parca-agent/main.go:741 +0x42 runtime/pprof.Do({0x23bdc30?, 0xc00036fdb0?}, {{0xc00039b7c0?, 0x2?, 0x0?}}, 0xc00053fea0) runtime/pprof/runtime.go:44 +0xa3 main.run.func11() github.com/parca-dev/parca-agent/cmd/parca-agent/main.go:740 +0x565 github.com/oklog/run.(*Group).Run.func1({0xc00033fb40?, 0xc00039b4c0?}) github.com/oklog/run@v1.1.0/group.go:38 +0x2f created by github.com/oklog/run.(*Group).Run github.com/oklog/run@v1.1.0/group.go:37 +0x69

Awesome 🎉

Thanks for taking the time and further testing this.

Also, good catch with the panic. I wonder what just happened there.

@kakkoyun thank you for always reviewing the PRs promptly!

Regarding the panic, it occurs at if l.Location.Mapping.File == "[vdso]" { ... } https://github.com/parca-dev/parca-agent/blob/main/pkg/symbol/symbol.go#L82. The mapping is nil for some reason &{ID:24 Mapping:<nil> Address:85858880 Line:[] IsFolded:false mappingIDX:0}.

Let's open an issue for this. I can have a look at it.

https://github.com/parca-dev/parca/issues/3135

marselester requested review from a team as code owners April 7, 2023 15:27

kakkoyun reviewed Apr 7, 2023

View reviewed changes

pkg/parca/parca.go Outdated Show resolved Hide resolved

marselester force-pushed the addr-norm-flag branch from 7fa835e to b791533 Compare April 7, 2023 21:26

marselester and others added 3 commits April 10, 2023 21:33

Add a flag to disable address normalization for debugging

2b80b4a

Remove "hidden" attritbute from embedded flag

2a90456

Co-authored-by: Kemal Akkoyun <kakkoyun@users.noreply.github.com>

Pass address normalization flag to fix benchmarkSetup

6ca3439

marselester force-pushed the addr-norm-flag branch 2 times, most recently from af45e52 to 567498d Compare April 11, 2023 13:51

Update NormalizeLocations to omit address normalization

a7c8a73

marselester force-pushed the addr-norm-flag branch from 567498d to a7c8a73 Compare April 11, 2023 15:26

marselester mentioned this pull request Apr 12, 2023

Store address normalization flag within Location entity #2978

Closed

kakkoyun reviewed May 16, 2023

View reviewed changes

vercel bot had a problem deploying to Preview – parca-ui May 16, 2023 10:44 Failure

kakkoyun merged commit 4a3ce86 into parca-dev:main May 16, 2023
32 of 33 checks passed

kakkoyun mentioned this pull request May 16, 2023

Panic with vdso mapping parca-dev/parca-agent#1654

Closed

metalmatze mentioned this pull request May 16, 2023

pkg/parcacol: Fix NewNormalizer call in test #3137

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Hidden flag to disable address normalization for debugging #2955

Hidden flag to disable address normalization for debugging #2955

marselester commented Apr 7, 2023

marselester commented Apr 7, 2023 •

edited

kakkoyun left a comment

kakkoyun May 16, 2023

marselester May 16, 2023

kakkoyun May 16, 2023

marselester May 16, 2023

kakkoyun May 16, 2023

kakkoyun May 16, 2023

marselester May 16, 2023

kakkoyun May 16, 2023

kakkoyun May 16, 2023

Hidden flag to disable address normalization for debugging #2955

Hidden flag to disable address normalization for debugging #2955

Conversation

marselester commented Apr 7, 2023

marselester commented Apr 7, 2023 • edited

kakkoyun left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

marselester commented Apr 7, 2023 •

edited