Stack overflow: stack trace and nix debugger support

[roberth]

Motivation

note
This is distinct from infinite recursion errors, which are already handled properly.

Try it with for example:

nix run github:hercules-ci/nix/stack-overflow -- build .#my-package

Provide means to solve stack overflows in Nix code.

• Stack trace
• Launch the debugger

TODO:

• solve the TODOs in the code. (Sorry for the low effort, as I had to write this from my not so great backup laptop for unrelated reasons)
• figure out what's the deal with frameOnly. If I don't comment it out, the trace just stops! Understand frame traces #8906
• add test
• support coroutines and test with overflow in source filter

Context

• A problem with the virtual memory-based stack overflow detection. Not the reason I wrote this, but this alternate implementation should sidestep the problem in Stack overflow detection is broken except on x86_64-linux #8281 (which isn't truly solved as long as we keep the virtual memory based solution around)
  • Fixes Stack overflow detection is broken except on x86_64-linux #8281
• Fixes Impossible to trace error: stack overflow (possible infinite recursion) #6361

Checklist for maintainers

Maintainers: tick if completed or explain if not relevant

• agreed on idea
• agreed on implementation strategy
• tests, as appropriate
  • functional tests - tests/**.sh
  • unit tests - src/*/tests
  • integration tests - tests/nixos/*
• documentation in the manual
• documentation in the internal API docs
• code and comments are self-explanatory
• commit message explains why the change was made
• new feature or incompatible change: updated release notes

Priorities

Add 👍 to pull requests you find important.

[Fuuzetsu]

I just want to say "thank you"! I managed to get a stack overflow using this patch. I had to set doCheck = false but other than that it seems to work for me!

[yajo]

I'm still not being able to diagnose my endless recursion. I set doCheck = false in flake.nix and ran eval against the flake where I have the problem, using nix from this PR. Exit code 1. Logs: nix-log.txt (I think those aren't very helpful).

[yajo]

I also tested with a publicly reproducible case: #6361 (comment)

Still unhelpful: nix-log.txt

[Fuuzetsu]

I also tested with a publicly reproducible case: #6361 (comment)

Still unhelpful: nix-log.txt

I'm able to confirm that it indeed doesn't seem to work for this case. Maybe something about import from derivation..? Who knows.

[roberth]

That probably happens because source filtering happens on a coroutine stack instead of the main thread stack.
Their stack protection pages should also be included. Added to TODO.