New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
pkg/stack/unwind: Add unwind tables support for Arm64 #1953
Conversation
This is still a WIP - some work still needs to be done for unit tests and testing a few other binaries. Apart from that, comments and a lot of |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM ❤️ 💚 💙 💛 💜
Just one readability suggestion.
I think if you could clean it up, we can merge it.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Looking good so far! A bit more work is needed so we can get this landed, but it's getting dangerously close 😄 .
One thing to think about as a follow-up (let's make a note, perhaps an issue or add a line in the tracking issue) about DWARF expressions, not sure if arm64 emits the same 2 PLT expressions we support and definitely the registers will be off, so we need to add special cases for them. Perhaps in the meantime, we can say that we don't support expressions at all and then change this later on?
pkg/stack/unwind/unwind_table.go
Outdated
@@ -29,6 +29,7 @@ import ( | |||
var ( | |||
ErrNoFDEsFound = errors.New("no FDEs found") | |||
ErrEhFrameSectionNotFound = errors.New("failed to find .eh_frame section") | |||
ErrNoRegisterFound = errors.New("failed to find register for architecture") |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This error is not handled anywhere, it's used more like a "default error" state, what about having that as a string directly rather than an error?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Also now that I think about it it should be more like "architecture not supported", no?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Good catch! I do feel we should capture an error here and just log it for easier debugging later but open to other suggestions :D
9103705
to
d9289d1
Compare
d9289d1
to
44ac7ac
Compare
Add support for extracting unwind table information from the ".eh-frame" section of Arm64 binaries. So far, only X86 binaries were supported. - Arm64 majorly differs from X86 in having a "Link Register(x30)" which saves the Return Address - The changes are only limited to userspace to generate "compact-unwind-tables"; we do not send the content of the Link Register to the BPF side- this will be done in follow-ups. - Enable DWARF-unwinding for Arm64 in "main.go" to enable building unwind-tables. - `CompactUnwindTableRow` has a new field `lrOffset` which holds the offset for the Link Register. This is initiialised to 0 for X86 architectures. TEST PLAN ========== - tested for `libc.so.6`, `basic-cpp-no-fp`, `systemd` on both Arm64 and X86 using `make-test-dwarf-unwind-tables` - tables generated using "dist/eh-frame" verified against `readelf -wF` output 1. Run `make build` in `parca-agent/testdata` `cd testdata && make build` 2. Run `make build && make-test-dwarf-unwind-tables` from parca-agent root on the main branch. `cd .. && make build && make-test-dwarf-unwind-tables` 3. Copy the output in `testdata/tables` and `testdata/compact_tables` in a separate untracked folder `mkdir ~/testdata-arm64-main && cp -r testdata/tables testdata/compact_tables ~/testdata-arm64-main` 4. Switch to current `dwarf-unwinding-arm64` branch 5. Run `make build` in parca-agent root. (Make sure you don't do this in the `testdata` submodule) 6. Run `make-test-dwarf-unwind-tables`. 7. Compare the output in `testdata/tables` and `testdata/compact_tables` to the results saved earlier in `~/testdata-arm64-main` 8. Verify all tables against `readelf -wF "required-binary"` 9. Repeat above for x86 binaries. Signed-off-by: Sumera Priyadarsini <sylphrenadin@gmail.com> pkg/stack/unwind: fix tests TEST PLAN ========== $ go test -v ./pkg/stack/unwind -count=1 ``` === RUN TestIsEndOfFDEMarkerWorks --- PASS: TestIsEndOfFDEMarkerWorks (0.00s) === RUN TestCompactUnwindTableX86 === RUN TestCompactUnwindTableX86/CFA_with_Offset_on_x86_64_stack_pointer === RUN TestCompactUnwindTableX86/CFA_with_Offset_on_x86_64_frame_pointer === RUN TestCompactUnwindTableX86/CFA_known_expression_PLT_1 === RUN TestCompactUnwindTableX86/CFA_known_expression_PLT_2 === RUN TestCompactUnwindTableX86/CFA_not_known_expression === RUN TestCompactUnwindTableX86/RBP_offset === RUN TestCompactUnwindTableX86/RBP_register === RUN TestCompactUnwindTableX86/RBP_expression === RUN TestCompactUnwindTableX86/Invalid_CFA_rule_returns_error --- PASS: TestCompactUnwindTableX86 (0.00s) --- PASS: TestCompactUnwindTableX86/CFA_with_Offset_on_x86_64_stack_pointer (0.00s) --- PASS: TestCompactUnwindTableX86/CFA_with_Offset_on_x86_64_frame_pointer (0.00s) --- PASS: TestCompactUnwindTableX86/CFA_known_expression_PLT_1 (0.00s) --- PASS: TestCompactUnwindTableX86/CFA_known_expression_PLT_2 (0.00s) --- PASS: TestCompactUnwindTableX86/CFA_not_known_expression (0.00s) --- PASS: TestCompactUnwindTableX86/RBP_offset (0.00s) --- PASS: TestCompactUnwindTableX86/RBP_register (0.00s) --- PASS: TestCompactUnwindTableX86/RBP_expression (0.00s) --- PASS: TestCompactUnwindTableX86/Invalid_CFA_rule_returns_error (0.00s) === RUN TestCompactUnwindTableArm64 === RUN TestCompactUnwindTableArm64/CFA_with_Offset_on_Arm64_stack_pointer === RUN TestCompactUnwindTableArm64/CFA_with_Offset_on_Arm64_frame_pointer === RUN TestCompactUnwindTableArm64/CFA_known_expression_PLT_1 === RUN TestCompactUnwindTableArm64/CFA_known_expression_PLT_2 === RUN TestCompactUnwindTableArm64/CFA_not_known_expression === RUN TestCompactUnwindTableArm64/RBP_offset === RUN TestCompactUnwindTableArm64/RBP_register === RUN TestCompactUnwindTableArm64/RBP_expression === RUN TestCompactUnwindTableArm64/Invalid_CFA_rule_returns_error --- PASS: TestCompactUnwindTableArm64 (0.00s) --- PASS: TestCompactUnwindTableArm64/CFA_with_Offset_on_Arm64_stack_pointer (0.00s) --- PASS: TestCompactUnwindTableArm64/CFA_with_Offset_on_Arm64_frame_pointer (0.00s) --- PASS: TestCompactUnwindTableArm64/CFA_known_expression_PLT_1 (0.00s) --- PASS: TestCompactUnwindTableArm64/CFA_known_expression_PLT_2 (0.00s) --- PASS: TestCompactUnwindTableArm64/CFA_not_known_expression (0.00s) --- PASS: TestCompactUnwindTableArm64/RBP_offset (0.00s) --- PASS: TestCompactUnwindTableArm64/RBP_register (0.00s) --- PASS: TestCompactUnwindTableArm64/RBP_expression (0.00s) --- PASS: TestCompactUnwindTableArm64/Invalid_CFA_rule_returns_error (0.00s) === RUN TestHasFramePointersInModernGolang --- PASS: TestHasFramePointersInModernGolang (0.00s) === RUN TestHasFramePointersInCApplication --- PASS: TestHasFramePointersInCApplication (0.00s) === RUN TestHasFramePointersCache --- PASS: TestHasFramePointersCache (0.00s) === RUN TestEmptyMappingsWorks --- PASS: TestEmptyMappingsWorks (0.00s) === RUN TestMappingsWorks --- PASS: TestMappingsWorks (0.00s) === RUN TestMappingsWithSplitSectionsWorks --- PASS: TestMappingsWithSplitSectionsWorks (0.00s) === RUN TestMappingsWithJittedSectionsWorks --- PASS: TestMappingsWithJittedSectionsWorks (0.00s) === RUN TestMappingsJitSectionDetectionWorks --- PASS: TestMappingsJitSectionDetectionWorks (0.00s) === RUN TestMappingsIsNotFileBackedWorks --- PASS: TestMappingsIsNotFileBackedWorks (0.00s) === RUN TestMappingJitDetectionWorks --- PASS: TestMappingJitDetectionWorks (0.00s) === RUN TestMappingSpecialSectionDetectionWorks --- PASS: TestMappingSpecialSectionDetectionWorks (0.00s) === RUN TestMappingJitDumpDetectionWorks --- PASS: TestMappingJitDumpDetectionWorks (0.00s) === RUN TestExecutableMappingCountWorks --- PASS: TestExecutableMappingCountWorks (0.00s) === RUN TestExecutableHashWorks --- PASS: TestExecutableHashWorks (0.00s) === RUN TestAllProcesses maps_test.go:157: This test requires root --- SKIP: TestAllProcesses (0.00s) === RUN TestBuildUnwindTable --- PASS: TestBuildUnwindTable (0.00s) PASS ok github.com/parca-dev/parca-agent/pkg/stack/unwind 0.007s ``` Signed-off-by: Sumera Priyadarsini <sylphrenadin@gmail.com>
Signed-off-by: Sumera Priyadarsini <sylphrenadin@gmail.com>
44ac7ac
to
3c73e24
Compare
… Arm64 `rbpType` is not set correctly in case of Undefined Return Address in Link Register causing unwinding an extra frame despite reaching bottom frame(main()). This was not caught by any tests in #1953(#1953) so add tests as well. TEST PLAN ========= Before: ``` === RUN TestCompactUnwindTableArm64/RA_Rule_undefined compact_unwind_table_test.go:217: Error Trace: /Users/icebear/Snow/polar/parca-agent/pkg/stack/unwind/compact_unwind_table_test.go:217 Error: Not equal: expected: unwind.CompactUnwindTable{unwind.CompactUnwindTableRow{pc:0x7b, lrOffset:0, cfaType :0x1, rbpType:0x0, cfaOffset:0, rbpOffset:0}} actual : unwind.CompactUnwindTable{unwind.CompactUnwindTableRow{pc:0x7b, lrOffset:0, cfaType :0x2, rbpType:0x4, cfaOffset:0, rbpOffset:0}} --- FAIL: TestCompactUnwindTableArm64/RA_Rule_undefined (0.00s) === RUN TestCompactUnwindTableX86/RA_Rule_undefined compact_unwind_table_test.go:407: Error Trace: /Users/icebear/Snow/polar/parca-agent/pkg/stack/unwind/compact_unwind_table_test.go:407 Error: Not equal: expected: unwind.CompactUnwindTable{unwind.CompactUnwindTableRow{pc:0x7b, lrOffset:-16, cfaType :0x2, rbpType:0x4, cfaOffset:0, rbpOffset:0}} actual : unwind.CompactUnwindTable{unwind.CompactUnwindTableRow{pc:0x7b, lrOffset:0, cfaType :0x1, rbpType:0x0, cfaOffset:0, rbpOffset:0}} --- FAIL: TestCompactUnwindTableX86/RA_Rule_undefined (0.00s) ``` After: ``` - PASS: TestCompactUnwindTableX86 (0.00s) - PASS: TestCompactUnwindTableX86/CFA_with_Offset_on_x86_64_stack_pointer (0.00s) - PASS: TestCompactUnwindTableX86/CFA_with_Offset_on_x86_64_frame_pointer (0.00s) - PASS: TestCompactUnwindTableX86/CFA_known_expression_PLT_1 (0.00s) - PASS: TestCompactUnwindTableX86/CFA_known_expression_PLT_2 (0.00s) - PASS: TestCompactUnwindTableX86/CFA_not_known_expression (0.00s) - PASS: TestCompactUnwindTableX86/RA_Rule_undefined (0.00s) - PASS: TestCompactUnwindTableX86/RBP_offset (0.00s) - PASS: TestCompactUnwindTableX86/RBP_register (0.00s) - PASS: TestCompactUnwindTableX86/RBP_expression (0.00s) - PASS: TestCompactUnwindTableX86/Invalid_CFA_rule_returns_error (0.00s) -- PASS: TestCompactUnwindTableArm64 (0.00s) - PASS: TestCompactUnwindTableArm64/CFA_with_Offset_on_Arm64_stack_pointer (0.00s) - PASS: TestCompactUnwindTableArm64/CFA_with_Offset_on_Arm64_frame_pointer (0.00s) - PASS: TestCompactUnwindTableArm64/CFA_known_expression_PLT_1 (0.00s) - PASS: TestCompactUnwindTableArm64/CFA_known_expression_PLT_2 (0.00s) - PASS: TestCompactUnwindTableArm64/CFA_not_known_expression (0.00s) - PASS: TestCompactUnwindTableArm64/RA_Rule_undefined (0.00s) - PASS: TestCompactUnwindTableArm64/RBP_offset (0.00s) - PASS: TestCompactUnwindTableArm64/RBP_register (0.00s) - PASS: TestCompactUnwindTableArm64/RBP_expression (0.00s) - PASS: TestCompactUnwindTableArm64/Invalid_CFA_rule_returns_error (0.00s) ```
… Arm64 `rbpType` is not set correctly in case of Undefined Return Address in Link Register causing unwinding an extra frame despite reaching bottom frame(main()). This was not caught by any tests in #1953(#1953) so add tests as well. TEST PLAN ========= Before: ``` === RUN TestCompactUnwindTableArm64/RA_Rule_undefined compact_unwind_table_test.go:217: Error Trace: /Users/icebear/Snow/polar/parca-agent/pkg/stack/unwind/compact_unwind_table_test.go:217 Error: Not equal: expected: unwind.CompactUnwindTable{unwind.CompactUnwindTableRow{pc:0x7b, lrOffset:0, cfaType :0x1, rbpType:0x0, cfaOffset:0, rbpOffset:0}} actual : unwind.CompactUnwindTable{unwind.CompactUnwindTableRow{pc:0x7b, lrOffset:0, cfaType :0x2, rbpType:0x4, cfaOffset:0, rbpOffset:0}} --- FAIL: TestCompactUnwindTableArm64/RA_Rule_undefined (0.00s) === RUN TestCompactUnwindTableX86/RA_Rule_undefined compact_unwind_table_test.go:407: Error Trace: /Users/icebear/Snow/polar/parca-agent/pkg/stack/unwind/compact_unwind_table_test.go:407 Error: Not equal: expected: unwind.CompactUnwindTable{unwind.CompactUnwindTableRow{pc:0x7b, lrOffset:-16, cfaType :0x2, rbpType:0x4, cfaOffset:0, rbpOffset:0}} actual : unwind.CompactUnwindTable{unwind.CompactUnwindTableRow{pc:0x7b, lrOffset:0, cfaType :0x1, rbpType:0x0, cfaOffset:0, rbpOffset:0}} --- FAIL: TestCompactUnwindTableX86/RA_Rule_undefined (0.00s) ``` After: ``` - PASS: TestCompactUnwindTableX86 (0.00s) - PASS: TestCompactUnwindTableX86/CFA_with_Offset_on_x86_64_stack_pointer (0.00s) - PASS: TestCompactUnwindTableX86/CFA_with_Offset_on_x86_64_frame_pointer (0.00s) - PASS: TestCompactUnwindTableX86/CFA_known_expression_PLT_1 (0.00s) - PASS: TestCompactUnwindTableX86/CFA_known_expression_PLT_2 (0.00s) - PASS: TestCompactUnwindTableX86/CFA_not_known_expression (0.00s) - PASS: TestCompactUnwindTableX86/RA_Rule_undefined (0.00s) - PASS: TestCompactUnwindTableX86/RBP_offset (0.00s) - PASS: TestCompactUnwindTableX86/RBP_register (0.00s) - PASS: TestCompactUnwindTableX86/RBP_expression (0.00s) - PASS: TestCompactUnwindTableX86/Invalid_CFA_rule_returns_error (0.00s) -- PASS: TestCompactUnwindTableArm64 (0.00s) - PASS: TestCompactUnwindTableArm64/CFA_with_Offset_on_Arm64_stack_pointer (0.00s) - PASS: TestCompactUnwindTableArm64/CFA_with_Offset_on_Arm64_frame_pointer (0.00s) - PASS: TestCompactUnwindTableArm64/CFA_known_expression_PLT_1 (0.00s) - PASS: TestCompactUnwindTableArm64/CFA_known_expression_PLT_2 (0.00s) - PASS: TestCompactUnwindTableArm64/CFA_not_known_expression (0.00s) - PASS: TestCompactUnwindTableArm64/RA_Rule_undefined (0.00s) - PASS: TestCompactUnwindTableArm64/RBP_offset (0.00s) - PASS: TestCompactUnwindTableArm64/RBP_register (0.00s) - PASS: TestCompactUnwindTableArm64/RBP_expression (0.00s) - PASS: TestCompactUnwindTableArm64/Invalid_CFA_rule_returns_error (0.00s) ``` Signed-off-by: Sumera Priyadarsini <sylphrenadin@gmail.com>
…unwind table (#2015) pkg/stack/unwind: Handle undefined return addresses in arm64 compact unwind table `rbpType` is not set correctly in case of Undefined Return Address in Link Register causing unwinding an extra frame despite reaching bottom frame(main()). This was not caught by any tests in #1953(#1953) so add tests as well. TEST PLAN ========= Before: ``` === RUN TestCompactUnwindTableArm64/RA_Rule_undefined compact_unwind_table_test.go:217: Error Trace: /Users/icebear/Snow/polar/parca-agent/pkg/stack/unwind/compact_unwind_table_test.go:217 Error: Not equal: expected: unwind.CompactUnwindTable{unwind.CompactUnwindTableRow{pc:0x7b, lrOffset:0, cfaType :0x1, rbpType:0x0, cfaOffset:0, rbpOffset:0}} actual : unwind.CompactUnwindTable{unwind.CompactUnwindTableRow{pc:0x7b, lrOffset:0, cfaType :0x2, rbpType:0x4, cfaOffset:0, rbpOffset:0}} --- FAIL: TestCompactUnwindTableArm64/RA_Rule_undefined (0.00s) === RUN TestCompactUnwindTableX86/RA_Rule_undefined compact_unwind_table_test.go:407: Error Trace: /Users/icebear/Snow/polar/parca-agent/pkg/stack/unwind/compact_unwind_table_test.go:407 Error: Not equal: expected: unwind.CompactUnwindTable{unwind.CompactUnwindTableRow{pc:0x7b, lrOffset:-16, cfaType :0x2, rbpType:0x4, cfaOffset:0, rbpOffset:0}} actual : unwind.CompactUnwindTable{unwind.CompactUnwindTableRow{pc:0x7b, lrOffset:0, cfaType :0x1, rbpType:0x0, cfaOffset:0, rbpOffset:0}} --- FAIL: TestCompactUnwindTableX86/RA_Rule_undefined (0.00s) ``` After: ``` - PASS: TestCompactUnwindTableX86 (0.00s) - PASS: TestCompactUnwindTableX86/CFA_with_Offset_on_x86_64_stack_pointer (0.00s) - PASS: TestCompactUnwindTableX86/CFA_with_Offset_on_x86_64_frame_pointer (0.00s) - PASS: TestCompactUnwindTableX86/CFA_known_expression_PLT_1 (0.00s) - PASS: TestCompactUnwindTableX86/CFA_known_expression_PLT_2 (0.00s) - PASS: TestCompactUnwindTableX86/CFA_not_known_expression (0.00s) - PASS: TestCompactUnwindTableX86/RA_Rule_undefined (0.00s) - PASS: TestCompactUnwindTableX86/RBP_offset (0.00s) - PASS: TestCompactUnwindTableX86/RBP_register (0.00s) - PASS: TestCompactUnwindTableX86/RBP_expression (0.00s) - PASS: TestCompactUnwindTableX86/Invalid_CFA_rule_returns_error (0.00s) -- PASS: TestCompactUnwindTableArm64 (0.00s) - PASS: TestCompactUnwindTableArm64/CFA_with_Offset_on_Arm64_stack_pointer (0.00s) - PASS: TestCompactUnwindTableArm64/CFA_with_Offset_on_Arm64_frame_pointer (0.00s) - PASS: TestCompactUnwindTableArm64/CFA_known_expression_PLT_1 (0.00s) - PASS: TestCompactUnwindTableArm64/CFA_known_expression_PLT_2 (0.00s) - PASS: TestCompactUnwindTableArm64/CFA_not_known_expression (0.00s) - PASS: TestCompactUnwindTableArm64/RA_Rule_undefined (0.00s) - PASS: TestCompactUnwindTableArm64/RBP_offset (0.00s) - PASS: TestCompactUnwindTableArm64/RBP_register (0.00s) - PASS: TestCompactUnwindTableArm64/RBP_expression (0.00s) - PASS: TestCompactUnwindTableArm64/Invalid_CFA_rule_returns_error (0.00s) ``` Signed-off-by: Sumera Priyadarsini <sylphrenadin@gmail.com>
What?
Add support for extracting unwind table information from the ".eh-frame"
section of Arm64 binaries. So far, only X86 binaries were supported.
not send the content of the Link Register to the BPF side- this will be done in follow-ups.
CompactUnwindTableRow
has a new fieldlrOffset
which holds the offset for the Link Register.This is initiialised to 0 for X86 architectures.
Test Plan
tested for
libc.so.6
,basic-cpp-no-fp
,systemd
on both Arm64 and X86using
make-test-dwarf-unwind-tables
tables generated using "dist/eh-frame" verified against
readelf -wF
outputCheck #1797 for Arm64 support roadmap