After version 0.23 is unable to resolve symbols #2096
Comments
Sorry just to make sure, you meant v0.19 here, right? |
|
yeah sorry :), I'll fix the report |
|
The 0.23 version that works that you tried is 0.23.3? |
|
Could you also turn on debug logging on the v0.25.1 agent and look for log lines that say |
|
I don't see any line with The dump of |
yes |
|
more information: the addresses seem wrong, attaching the debugger I see they don't correspond with the running process (a part for the least significant 3 nibbles): this is a couple of addresses from the UI of parca
meanwhile from the debugger with some mappings |
|
I'm trying with a very simple executable and If I patch with the following I obtain the following log note that in this case the I have a question, |
|
The addresses are not normalized (anymore), we just send enough information for the backend to normalize. |
|
Maybe we're not handling Amazing debugging by the way! It would make a ton of sense that this is the culprit as this is exactly one of the things that changed from 0.23.3 to 0.24. |
|
and the server how knows if the address is normalized or not? |
|
The client sends whether the addresses are normalized, which Parca-Agent since 0.24 always sets to false. |
|
bd65f33#diff-7276397f96515c11ca49f4f72de2237d63936ab985888a5a4fe07ccf8c371d42L124 always sets normalized to false (since it's the default value) |
|
Maybe we should add some debug logging here: https://github.com/parca-dev/parca/blob/d29477a629bfdc95ed3cfd7ebece9e0cd9be3e6f/pkg/parcacol/normalizer.go#L352-L359 |
|
Do you have your example binary that you can share? That would make debugging on our side much easier. It's odd that we haven't noticed this with our customers who have long been on 0.24.0+ (though if it ends up being a backend bug then maybe not so surprising since symbolization is entirely different in Polar Signals Cloud). |
|
sure, but I think whatever binary will work (zipped for uploading), in particular the problem it's in the libraries, |
|
Another update, with the debugger I see at the end of with for example |
|
What I don't understand is the concept of normalized mapping, with to respect to what is normalized? |
|
with a position independent executable not even Were you able to reproduce from your side? |
|
Sorry we had a big launch yesterday which ate up most of our time, I haven't found the time to reproduce yet, but we're gonna be on top of this today! The mapping normalization is to account for address space layout randomization (ASLR), basically even if the layout is different between two processes on two different machines, if the same binaries were used we want to only save the mapping once. If that's the case though we need to normalize all addresses we store to the first layout we've ever observed. Now that you say it though, not sure it can make a difference, but maybe we need to do the part where we normalize the address to the normalized mapping first, before we perform the second normalization which is to the start of the mapping and offset. |
|
Ok I can also reproduce this with our PIE test binary. |
|
I think I found at least one problem, we'll see if that fixes things. |
|
Ok found it. There were two reasons why we might double normalize:
This all didn't matter with non-PIEs because all offsets are 0 therefore subtracting/adding 0s didn't make a difference. Opened parca-dev/parca#3903 to fix |
|
Can you test that and let me know if it works for you? If it does then I'll cut a patch release. |
|
It seems to work, there is a couple of |
|
That should be fine, that just means that we couldn't symbolize, we'll be improving the symbolization quality over the next weeks and months. If some are symbolized that means normalization is done correctly again though. Thank you so much for reporting this and bearing with us! |
|
After having upgraded to 0.20 the problem with symbol resolution persists, there are a lot more of functions that are not resolved but I don't see a specific pattern. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Describe the bug
previous versions resolved symbols correctly
To Reproduce
in our test environment I switched between the version
0.25and0.23and the first it's not showing any simbol.Also tried version
0.24but it's the same as0.25Screenshots
you can see 0.25 (left) and 0.23 (right), by luck the flamegraphs are aligned with the version
Logs
no obvious error is coming out from the log (by the application and by ebpf)
Software (please complete the following information):
parca-agent --version] 0.25.1parca --version] 0.19 (but it's the same for 0.18)Workload (please complete the following information):
Environment (please complete the following information):
cat /etc/*-release] rhel9uname -r] 5.14.0-162.6.1.el9_1.x86_64uname -m] x86_64kubectl --version]Additional context
The text was updated successfully, but these errors were encountered: