Find strategy for evolving debuginfo format without breaking everything

[michaelwoerister]

It's a well-known fact that the Rust compiler emits debuginfo in a format that is neither very complete nor very elegant. The main reason for this is that so far we've targeted debuginfo consumers that do not know about Rust and would treat it more or less like C or C++. Going forward though, we need to make changes to the debuginfo format if we want to allow Rust-aware debuggers to provide much better support than is possible now. This raises the question how we can start evolving the format without breaking support by the current line of debuggers.

There are two scenarios here:
(1) The new debuginfo format is just a backwards compatible extension to what we have now.
(2) The new format contains constructs that current debuggers will not be able to interpret.

While scenario (1) would provide some desirable, short-term benefits, my personal opinion is that it is (a) likely not feasible to achieve, and (b) that we should strive not to set in stone some of the ugliness of the current format (e.g. enum encoding). That being said, it might make sense to identify any points where a backwards-compatible format would fall short in functionality.

If we assume scenario (2) it seems unavoidable to me that we support emitting two debuginfo encodings for at least the next few years because:

• the GDB versions distributed with common Linux distributions will still not reliably know about Rust for a while, and
• there is no effort to implement native Rust support in LLDB yet, at least as far as I know.

The question is: how can we implement this in a way that is ergonomic and not confusing to the user? Or in other words, how do we avoid that for part of the user-base the debugging story is worse than it could be -- either because they don't use their Rust-aware debugger to its full potential, or because their C++ debugger chokes on debuginfo it cannot handle -- without them knowing that a simple compiler flag would solve their problems, or it being too cumbersome to set that compiler flag.

cc @rust-lang/tools @tromey @Manishearth

[Manishearth]

there is no effort to implement native Rust support in LLDB yet, at least as far as I know.

I am willing to join such an effort if it exists, but this is not something I can do on my own.

The question is: how can we implement this in a way that is ergonomic and not confusing to the user?

shipping gdb/lldb trunk with rust was sort of the solution to this (#34457) 😄

how do we avoid that for part of the user-base the debugging story is worse than it could be -- either because they don't use their Rust-aware debugger to its full potential

If we have magic-new-dwarf off by default, we can have debuggers aware of this new format throw a warning when they load, and suggest the users use on rustc to get the magic new format.

I am unsure how easy it would be to keep it backcompat. Backcompat is ideal, but cumbersome.

@tromey, what things are hard to represent in current DWARF? I can think of two:

• Traits (who implements what). We can perhaps emit an extra table here or something, backwards compatibly.
• Enums -- we already have a solution for this, which is hacky, but ... works. And is backcompat, yay.

[tromey]

@tromey, what things are hard to represent in current DWARF?

In addition to what you mentioned, also trait objects and closures; also a few constructs are currently represented but would be done more cleanly with some new DWARF tags. @michaelwoerister and I have a google doc where we're writing up a plan for all this. My long term goals here are (1) put the doc into the tree to document how DWARF generation ought to work; (2) fix rustc to conform; and (3) submit DWARF issues to extend the spec, joining the committee if necessary.

I think it's on the whole better not to add special legacy DWARF output code to rustc. That effort would be better spent writing a Rust plugin for lldb. So how about we start that instead?

There are some deployment issues here but those, too, I think are surmountable; e.g., for Mac, build and ship the lldb plugin; for Linux, well, it just isn't that hard to build one's own gdb, and anyway the distros will all be shipping the Rust-enabled one soon (and sooner if the Rust toolchain gets packaged).

[Manishearth]

Could you add me to this doc? The plan looks good so far.

[michaelwoerister]

I think it's on the whole better not to add special legacy DWARF output code to rustc. That effort would be better spent writing a Rust plugin for lldb. So how about we start that instead?

I suspect that that would be a lot more work than just extending rustc while leaving in the existing stuff via a commandline switch. I think this would only be an option if we kept things strictly backwards compatible until the LLDB plugin has reached production quality.

Supporting two different formats of debuginfo for a transition period is more of a usability issue than one of limited resources, in my mind.

[Manishearth]

Oh, since you asked on IRC and logged off later: either email is fine. I usually use my gmail for open source projects (except for GNU stuff, where ironically I use mozilla.com, because copyright).

[Manishearth]

Are gdb and lldb the only consumers of DWARF? I guess windows uses PDB only?

[tromey]

Are gdb and lldb the only consumers of DWARF?

There are others on Linux: SystemTap, abigail (ABI checker), 7 dwarves, dwgrep, dwz.

[nrc]

Profilers and some other tools also use DWARF, although I expect only the line info rather than type info and the latter is the real issue for us.

[DemiMarie]

Here is a thought that might work, at least for debuggers: have the code that is upstreamed be only stub code. Rust would then provide the real interface code as a plug-in (a shared object loaded at runtime).

This has the added bonus of allowing the debugger support to be written in Rust, instead of C/C++.

[Manishearth]

That works for lldb, but not gdb

[tromey]

I've started work on the Rust language plugin for lldb. My plan to solve this bug is to first make that work, then evolve Rust debuginfo in lockstep with changes to llvm, lldb, and gdb.

I think this bug ought to be closed as the work is in progress and there isn't much here that is actionable (apart from the individual bits of work that will be done elsewhere).

[michaelwoerister]

Thanks for the update, @tromey!