Improve trace callback performance

[pschanely]

As described here, CrossHair spends a lot of its execution time in a pure-python sys.settrace handler. Making this faster should speed up CrossHair quite a bit.

It would be great to benchmark how much would be saved by moving some of that logic to an extension module written in C; it wouldn't be possible to move all the logic, but many tracing invocations are no-ops, and avoiding those at least could help a lot. Cython might also be worth investigating.

[IssueHuntBot]

@amacfie has funded $100.00 to this issue.

---

• Submit pull request via IssueHunt to receive this reward.
• Want to contribute? Chip in to this issue via IssueHunt.
• Checkout the IssueHunt Issue Explorer to see more funded issues.
• Need help from developers? Add your repository on IssueHunt to raise funds.

[pschanely]

Thank you for the bounty, @amacfie!
To make the task a little more concrete, the logic in this function is, more or less, what we'd want to implement at the C level.

Honestly, the biggest challenge here might be to preserve the easy, fast, and frustration-free packaging and installation process that CrossHair currently enjoys.

We use github actions to package/publish when a release is tagged. Once we add this extension module, we'll need to build and publish binary wheels. I'd want to support the same platforms that z3 does, and retain our current level of automation (probably with GitHub actions). I believe GitHub actions has runners for windows and mac, so I think this is achievable, maybe with some help from cibuildwheel. (though the last time I investigated this, it wasn't as trivial to get working as I'd hoped)

[pschanely]

Much of this is now implemented, but there are two more valuable changes:
(1) We should port the many different ways to do a stack read and write to C. In theory, the resulting code should be faster, smaller, and more correct/portable, because the ctypes approach we have now is overly dependent on the byte-level layout of the frame struct.
(2) CompositeTracer is now largely just a wrapper around CTracer. Let's try to remove the CompositeTracer layer entirely.