Reimplement Wasmtime's DWARF transform and debugging support

[fitzgen]

We should reimplement the DWARF_wasm to DWARF_native
transformation pass that implements the GDB/LLDB debugging support in Wasmtime
by separating DWARF translation from DWARF traversal. We could do this by
defining a generic DWARF transformation pass that takes a generic visitor
implementation, walks the read-only input DWARF, calls the corresponding visitor
method for each DIE/attribute/value/line-table entry/etc... in the DWARF to
produce a new DWARF entity, and writes that new DWARF entity into the output
DWARF that is being built up. We would then implement a DWARF_wasm to
DWARF_native visitor.

I think this approach would be much easier to implement, maintain, and ensure
correctness of than our current open-coded transformation.

Assuming this interface works out well and we prove it out, it could be worth
upstreaming the generic transformation pass and visitor trait into gimli
itself (cc @philipc).

Potential hiccups could be that, for our purposes here, the visitor might not be
exactly a simple map over the input DWARF (or "functor-ish") in that one
DWARF_wasm entity might become multiple DWARF_native
entities (making it more "monad-ish", apologies if I'm just muddying the waters
with this nomenclature). One example is that what might be a location list entry
in Wasm could become multiple location list entries in native code due to
register allocation, live range splitting, and spilling.

Testing

Our testing story for debugging support is very poor at the moment and the
debugging support is correspondingly buggy. As part of this reimplementation, we
should take the opportunity to improve our approach to testing.

I think we can do something like this, in a loop:

• generate a random C program with C-Smith
• compile the program twice:
  1. to wasm32-wasi
  2. to the host target
• attach gdb and/or lldb to
  1. wasmtime running the wasm version
  2. the native binary
• single step N times (or until main exits) and at each point assert that:
  • the native and wasm programs are paused at the same location
  • the same variables are in scope
  • the variables in scope have the same values (at least for non-pointer
    scalars, we can tune the C-Smith flags we use to generate test programs as
    necessary)

I think this should give us fairly high confidence in the correctness of the new
DWARF transform.

Unfortunately, this won't fit into OSS-Fuzz's paradigm super well. It involves a
lot of wrangling external processes. I think we can do N iterations under normal
cargo test with a fixed corpus of seeds, so that running cargo test twice
runs the same set of test programs each time. And then in CI perhaps we can have
a job that runs more iterations, or a nightly CI job that does a bunch of
iterations, or something like that. To some degree, we can kick this can down
the road and figure things out once we have the test infrastructure set up (even
just running it manually whenever we touch this code would be a huge improvement
over our current debugging testing strategy).

cc @cfallin as this is something we have talked about together in the past.

[cfallin]

Admittedly a slightly wild idea, if we wanted to try to fuzz: if the Wasmtime function-call API had a mode in which we could "single-step call" (maybe an async fn that yields at each new srcloc), and a way to introspect via DWARF the Wasm-level state at each step (so, a built-in programmatic gdb), and if we similarly instrumented an interpreter (wasmi?) to let us single-step and introspect state, then we could fuzz in-process as rapidly as we do differential execution tests today.

I wonder how we might be able to leverage existing Rust debugger libraries; I see Headcrab, and e.g. its DWARF support may be usable for this on the native-code side.

This is probably at least three months of developer time but we'd have best-in-class assurances of debuggability and the correctness of the provided program state if we had something like this, and a "debugger API" on Wasmtime could be the basis for a lot of other useful tooling too.