Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

unwinding/dwarf: arm64 support #1797

Open
13 of 18 tasks
javierhonduco opened this issue Jun 26, 2023 · 1 comment
Open
13 of 18 tasks

unwinding/dwarf: arm64 support #1797

javierhonduco opened this issue Jun 26, 2023 · 1 comment
Assignees
@Sylfrena
Labels
area/stack-unwinding Something on stack unwinding

Comments

@javierhonduco
Copy link
Contributor

javierhonduco commented Jun 26, 2023 •
edited by Sylfrena

Context

While we currently build arm64 containers for the Agent, we don't truly support BPF-based profiling anymore. This is due to a complete rework of the core BPF bits to unwind through code without frame pointers that focused on supporting x86_64.

We want to change this, as there's an ever-growing amount of developers using this architecture locally, see Apple Silicon, as well as in production (Graviton et al).

Roadmap

Enabling work

  • 9f6546a Per-architecture vmlinux.h. As of this time, we only have the one for x86_64, we need to have one per architecture, and choose the right one at build time. This is needed CO-RE and BTF support ;
  • abdaf10 Build the agent in arm64 (we only release it now, but we don't build it in normal CI runs AFAIK);
  • Run CI tests in arm64 to ensure further changes (build tags, unwinder changes, etc) don't break our code under this architecture;

Adding frame pointer unwinding

  • 9f6546a Frame pointer unwinding is significantly simpler than DWARF-based unwinding. Very small changes are required both in the BPF code and userspace to make it work, so we'll start here, focusing on Go apps;

DWARF-based unwinding (#1209)

  • Ensure our unwind info reader and parser can deal with arm64 binaries (e.g. the info being in the same ELF section, always being present according to its ABI, etc)
  • Adapt our unwind info parser/interpreter to work with multiple architectures, particularly, arm64;
  • Adapting the compact unwind row, as well as the unwinder code for arm64's saved return address (link register);

Other things to look out for:

  • Hardcoded DWARF expressions (plt1 and plt2);
  • Pointer authentication;
  • Split stacks;
  • Port JIT test program to arm64;

Testing

  • The profiler's integration tests must pass;
  • Checking that things work fine on a VM in an Apple Silicon machine;
  • Test on Graviton machines;
  • vmtests for arm64 as a smoke test that we can load the BPF program;
  • Given that BPF on arm64 could be less mature, as a stretch goal, we should try
    • multiple distros, as they might have different patches, kernel versions, configs etc;
    • kernel with lockdep enabled

Related issues: #1211
@maxbrunet
Copy link
Member

Maybe tests could run on Oracle's Free Tier? https://blogs.oracle.com/cloud-infrastructure/post/announcing-github-actions-arm-runners-for-the-arm-compute-platform-on-oracle-cloud-infrastructure

I also found BuildJet supports ARM if you have a small budget for it: https://buildjet.com/for-github-actions

@Sylfrena Sylfrena mentioned this issue Aug 28, 2023
Merged
@javierhonduco javierhonduco added the area/stack-unwinding Something on stack unwinding label Aug 29, 2023
kakkoyun added a commit that referenced this issue Sep 4, 2023
@kakkoyun
pkg/stack/unwind: Add unwind tables support for Arm64 (#1953)
79dd98a 
### What?
Add support for extracting unwind table information from the ".eh-frame"
section of Arm64 binaries. So far, only X86 binaries were supported.
    
- Arm64 majorly differs from X86 in having a "Link Register(x30)" which
saves the Return Address
- The changes are only limited to userspace to generate
"compact-unwind-tables"; we do
not send the content of the Link Register to the BPF side- this will be
done in follow-ups.
- Enable DWARF-unwinding for Arm64 in "main.go" to enable building
unwind-tables.
- `CompactUnwindTableRow` has a new field `lrOffset` which holds the
offset for the Link Register.
    This is initiialised to 0 for X86 architectures.

### Test Plan

```
$ go test  ./pkg/stack/unwind -count=1
ok      github.com/parca-dev/parca-agent/pkg/stack/unwind       0.007s
```

- tested for `libc.so.6`, `basic-cpp-no-fp`, `systemd` on both Arm64 and
X86
    using `make-test-dwarf-unwind-tables`

- tables generated using "dist/eh-frame" verified against `readelf -wF`
output

``` 
 1. Run `make build` in `parca-agent/testdata`
    `cd testdata && make build`
 2. Run `make build && make-test-dwarf-unwind-tables` from parca-agent root on the main branch.
    `cd .. && make build && make-test-dwarf-unwind-tables`
 3. Copy the output in `testdata/tables` and `testdata/compact_tables` in a separate untracked folder
    `mkdir ~/testdata-arm64-main && cp -r testdata/tables testdata/compact_tables ~/testdata-arm64-main`
 4. Switch to current `dwarf-unwinding-arm64` branch
 5. Run `make build` in parca-agent root. (Make sure you don't do this in the `testdata` submodule)
 6. Run `make-test-dwarf-unwind-tables`.
 7. Compare the output in `testdata/tables` and `testdata/compact_tables` to the results saved earlier in
    `~/testdata-arm64-main`
 8. Verify all tables against `readelf -wF "required-binary"`
 9. Repeat above for x86 binaries.
```

Check #1797 for Arm64 support roadmap

<!--

copilot:poem

-->
@kakkoyun kakkoyun changed the title unwinder/native: arm64 support unwinding/dwarf: arm64 support Feb 24, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@Sylfrena Sylfrena

Labels
area/stack-unwinding Something on stack unwinding
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@javierhonduco @maxbrunet @Sylfrena