Package gdb with Rust

[brson]

gdb has first-class support for Rust now, and two active maintainers (@tromey @Manishearth). Let's package it with Rust. This will have lots of advantages:

• All major platforms will get the same debugging experience
• Debuging will be more reliable because users have the exact debugger we've actually tested with our toolchain
• By controlling the producer and consumer of our DWARF we can evolve it quickly, move to Rust-specific DWARF

Steps:

• Set up a fork in rust-lang-nursery
• Modify it to produce a gdb with a Rust-specific name
• Set up CI and packaging for gdb
• Modify rust-buildbot / rust-packaging to add gdb to the manifest as an optional component
• Figure out the best user experience for obtaining gdb via rustup

cc @michaelwoerister @rust-lang/tools What you think?

[alexcrichton]

Sounds great to me, I'm all for it!

[vadimcn]

I can see where you are coming from, but long term, I'd rather see LLDB standardized for Rust. I've investigated both while developing my vscode debugger front-end and here's why I chose to go with LLDB:

• GDB only really works on Linux and even then not always (try interrupting a process via the MI API on OSX or Windows, for instance).
• LLDB is a part of LLVM toolchain, so they are much easier to co-develop.
• LLDB supports plugins, so there's a chance to use the actual Rust parser for expressions, rather than some hand-rolled approximation.
• LLDB has MSVC .pdb debug info support in the works; I haven't seen any plans to do that for GDB.
• LLDB has an actual API for embedding.

[Manishearth]

LLDB being standardized for rust can happen independently of this. I was thinking of working on it; but not sure of I will (won't have time; C++ is draining).

[Manishearth]

Also, until we figure out a story for DWARF in rust things are unstable anyway. We already hit the situation once where marking our binaries as DW_LANG_RUST broke lldb but was necessary for gdb to work.

[michaelwoerister]

I agree with the things that @vadimcn says but I don't think that these observations necessarily speak against providing a Rust-specific GDB version.

Here are some more questions:

1. What's the advantage of providing our own version of GDB versus just relying on Rust support getting merged upstream eventually?
2. Can we provide an ergonomic enough experience for people to actually use this?
3. How much maintenance effort would this mean?
4. And, more specifically regarding the co-evolution of DWARF and GDB, will we need a "debugger-tuning" flag in the compiler? (since, at least in the beginning, LLDB will still not know about Rust)

Here are some thoughts, but I'd also like to hear how other people would answer these questions:

1. (a) The main advantage I see is that more people would be able to profit from Rust support sooner. It can take quite a while until a change ends up in package managers and some people (like me) tend to keep using LTS versions and only upgrade when they have to.
   
   (b) @brson's point about providing one specific and well-tested version per platform also very good! That's something I had not thought about before.
2. This boils down to the questions if we can make something like rustup install rust-gdb just work on Linux and OSX and hopefully on Windows too, I think. I would hope that that is not too hard on Linux, but I don't know about OSX, and GDB on Windows is always a rather brittle thing in my experience.
3. I have no idea :)
4. We might not quite get around that. On the other hand, there is precedent for such compiler flags.

So, I think, I'm generally in favor of this if we can pull it off with reasonable effort.

[brson]

I can see where you are coming from, but long term, I'd rather see LLDB standardized for Rust.

I agree LLDB is more promising in the long-run, but think we should take the opportunity to do what we can now with gdb.

It's true though that if we run ahead with our own DWARF formats while favoring GDB then the LLDB support will get worse. Perhaps we'll need to leave some option to stay LLDB compatible.

What's the advantage of providing our own version of GDB versus just relying on Rust support getting merged upstream eventually?

• We can turn on Rust-DWARF by default if we have confidence that all our users can read it. Otherwise we need to use C-compatible DWARF until we have confidence the Rust support has trickled out.
• Even once the Rust-DWARF support gets into the world, it will be very hard to evolve because of the lag time between us changing the definition, upstream landing, and upstream's downstreams distributing.

Can we provide an ergonomic enough experience for people to actually use this?

I'd expect we can do something where you type rust-gdb (or some new command), and it says "rust-gdb is not installed, run rustup component install rust-gdb", then you're ready.

How much maintenance effort would this mean?

For us it would mean updating the gdb build we produce at least once per quarter and remediating any issues that reveals.

And, more specifically regarding the co-evolution of DWARF and GDB, will we need a "debugger-tuning" flag in the compiler? (since, at least in the beginning, LLDB will still not know about Rust)

Yeah, seems like we would need to be able to emit at least two variations of DWARF. This would present some problems with the debuginfo in std.

This boils down to the questions if we can make something like rustup install rust-gdb just work on Linux and OSX and hopefully on Windows too, I think.

The quality of Windows support worries me too. It would be a shame if we couldn't support gdb there.

[brson]

My primary motivation for doing this is to be able to make more radical changes to our DWARF output. I'm only assuming that we need to make such changes, or have the resources to do it.

[tromey]

What's the advantage of providing our own version of GDB versus just relying on Rust support getting merged upstream eventually?

It's already merged.

And, more specifically regarding the co-evolution of DWARF and GDB, will we need a "debugger-tuning" flag in the compiler?

Probably not for gdb, but maybe for lldb.

My primary motivation for doing this is to be able to make more radical changes to our DWARF output. I'm only assuming that we need to make such changes, or have the resources to do it.

Yes, there's a need. There are a number of issues open about both errors in the generated debuginfo and also spots where more information is needed. Importing gdb into the tree isn't needed to make progress here (though you may want to do this for other reasons). I'm happy to write the gdb side upstream in parallel with rustc fixes.

The main issue I see is that there are few guards in place against rustc debuginfo regressions. 1.9 already regressed gdb's Rust support.

[Manishearth]

We could set up a build step that runs the gdb rust tests.

Build steps are straightforward and shouldn't take long:

• Ensure DejaGnu (runtest) is installed, built rustc is in the path
• Clone git://sourceware.org/git/binutils-gdb.git
• make all-gdb
• cd testsuite && make check RUNTESTFLAGS="gdb.rust/*.exp"

[Manishearth]

Alternatively we could get a gdb fork to use http://nightli.es/ with a rust-specific travis test

[vadimcn]

My primary motivation for doing this is to be able to make more radical changes to our DWARF output. I'm only assuming that we need to make such changes, or have the resources to do it.

Yes, there's a need. There are a number of issues open about both errors in the generated debuginfo and also spots where more information is needed.

Are these things spelled out anywhere?

[tromey]

Are these things spelled out anywhere?

Yes, I filed bugs for all the problems I found. Looking at open debuginfo bugs will show them all.

Looking at this, though, I see that the most important regression was closed (for some reason I thought the patch hadn't landed); so I'll have to re-test gdb with nightly and see if it's doing better now.

Some of the remaining holes require some work on DWARF or on detailing the Rust/DWARF binding. I've been planning to write a proposal for this, though I wasn't sure if this should just be a discourse post or a full-fledged RFC. Alternatively if someone gets to this before I do, I'd be happy to comment on it.

[tromey]

Still some failures with nightly -- better than 1.9, but 1.8 was clean. I'll diagnose soon, you can follow along here.

[Manishearth]

The methods failures are due to {{impl}} finding its way into method paths

[michaelwoerister]

Yeah, seems like we would need to be able to emit at least two variations of DWARF. This would present some problems with the debuginfo in std.

That is a problem. Can we somehow provide two versions of the debuginfo and make the wrapper scripts add the right one to the search path?

The main issue I see is that there are few guards in place against rustc debuginfo regressions.

That's a good point. We definitely would have to include the Rust version of GDB into autotests. But that's also something that should actually become easier if we distribute our own version, right?

My primary motivation for doing this is to be able to make more radical changes to our DWARF output. I'm only assuming that we need to make such changes, or have the resources to do it.

Yes, there's a need.

I absolutely agree with @tromey. We are basically stuck if we don't co-evolve the compiler and debuginfo consumers.

Some of the remaining holes require some work on DWARF or on detailing the Rust/DWARF binding. I've been planning to write a proposal for this, though I wasn't sure if this should just be a discourse post or a full-fledged RFC. Alternatively if someone gets to this before I do, I'd be happy to comment on it.

@tromey I'd be interested in this too. What would be the best place to allow multiple people working on it?

[michaelwoerister]

BTW, there might be a pretty big shift in the way LLVM handles debuginfo:
http://lists.llvm.org/pipermail/llvm-dev/2016-March/097773.html

TL;DR; DWARF or any other debuginfo might be emitted directly to LLVM IR instead of as LLVM metadata. The good thing about that is that we would have total control about what we emit and would not have to rely on LLVM merging any custom LLVM metadata => DWARF lowerings we might need.

[nrc]

I'm in favour of doing this. I think lowering the bar for getting good debugger support for Rust on at least one platform is very worthwhile. There is too much friction right now for a user to get to a good debugging experience.

My major worry is that we don't have the long term resources to keep this in a good place. Debuginfo in the compiler is already under-resourced and slowly bit-rotting, I fear that our gdb fork would end up in the same place, or worse.

[Manishearth]

The gdb fork probably won't stay a fork, patches will keep getting upstreamed. We just maintain a fork so that we can iterate faster, and the only commits that live in the fork would be things like this special function (which could also live in rust-gdb)

On the note of rust-gdb, what is the proposed future for rust-gdb? I left a comment here, but that seems to be something better discussed in this thread.

[tromey]

@tromey I'd be interested in this too. What would be the best place to allow multiple people working on it?

I have been working on a google doc. Contact me in email and I'll share it with you. It's not nearly done yet.

[blanham]

One issue with packaging GDB for macOS is that the binary needs to be codesigned in order to function.

[alexcrichton]

The tools team discussed this issue during triage yesterday and the main conclusion was that we're probably not quite in a position just yet to actively ship gdb. That being said, we're very willing to actively test newer versions of gdb to ensure that our support does not regress. In other words, we're always willing to have more automation!

The major blockers we think need to be overcome before making progress on this are:

• We need to ensure the experience is good on almost all platforms Rust supports. Right now it sounds like gdb and Rust works best on Linux, but on OSX there's worries about having to sign the binary and on Windows there's worries about gdb in general on MinGW and then also how it may interoperate with the MSVC target.
• Blessing one debugger will likely require at least two components, effort to ensure that the Rust compiler continues to make progress in terms of what it emits, as well as effort on the debugger side to ensure that bugs are fixed and such.
• Issues like #34560 come up if we start having Rust-specific DWARF information.

[DemiMarie]

Do we need to worry about man-in-the-middle attacks on the insecure git:// clone? I would definitely worry about them. If someone can do the clones who is a GDB committer they should use their ssh:// clone instead.

[Manishearth]

Presumably we'd have a fork on github, so no.

[DemiMarie]

@Manishearth how will that fork be cloned/updated?

[Manishearth]

There are two of us with access to ssh gdb sources.

Though this is a very narrow threat model which I don't expect to happen, really.

[DemiMarie]

Update: HTTPS access to GDB was broken due to a bug; this has now been fixed (at least I managed to check out the sources).

[locka99]

I'm trying to debug on Windows and I still can't do it well. TDM GDB sort of works in VSCode but step debugging is broken. At this point its easier to debug code by plastering logging all over the place and trying to pinpoint the issue.

I think shipping gdb with the stable-gcc would take a lot of guesswork out of making it work. TDM's gdb standalone is a 12MB zip so that's we're talking about sizewise

[tromey]

As part of my work on the Rust language plugin, I talked about shipping the plugin on the upstream list. There are two threads:

1. The one I started. This is specifically about the Rust plugin. Somewhat interesting on its own, but a reply pointed out a different discussion:

2. Thread about removing some other language plugins. Here the most interesting message proposes a standard for accepting new languages.

My conclusion from all of this is that we'll need some way to ship some lldb-related thing via rustup. In the short- and medium-term, this is likely to be all of lldb. In the longer term, maybe we can either ship just the Rust plugin (if lldb ever provides a way to build a language plugin out-of-tree), or just the Rust expression parser helper library/executable.

[tromey]

I've come around on this and I think it would make sense to get gdb into rustup. This would let us ship Rust improvements more quickly to users.

The main issues are around library dependencies. There's some discussion in #48168. There may also be some path issues, like finding system separate debuginfo or system gdb scripts.

[tromey]

I'm looking into this a bit. A couple possible issues:

• gdb optionally depends on a number of libraries; the main ones are Python, expat, LZMA, and termcap. (There are a few more that can be used, but I think we can comfortably omit them.) I think most of these can be linked statically (maybe some gdb build tweak will be required). For Python my current plan is to depend on Python 2.7. Maybe we can write some kind of wrapper that will check that it is installed either at rustup time or when trying to run gdb.

• gdb allows a few compile-time paths, like --with-separate-debug-dir. (I also saw the JIT reader dir and the system gdbinit; and possibly the default auto-load and safe paths.) A potential issue here is that different distros may put these in different places; if so, a possible fix would be to patch our gdb to adapt; or just use a path that encompasses all the common approaches.

[nrc]

Maybe we can write some kind of wrapper that will check that it is installed either at rustup time or when trying to run gdb.

I think a runtime wrapper will be better - especially if/when we move rustup into Cargo, we won't want to support that kind of check (although there are also vague plans for things like post-install scripts which could do this).

cc @joshtriplett for any potential distro interactions.

[tromey]

Some state here:

The PR may not be merged, as it puts gdb into the overall rust build; and this build is already quite slow, etc. Instead Alex said he'd be investigating whether gdb could be built separately and then the artifacts still distributed via rustup. This would definitely be nicer.

The PR is still of use though. It shows how to make a Dockerfile that installs the needed static dependencies. And, it patches the rust-gdb script to prefer a rustup-managed gdb if one is available; this patch should be landed if/when gdb is shipped through rustup.

There's a fork of gdb here: https://github.com/rust-dev-tools/gdb. My plan for this was to treat it like we do LLVM: keep some Rust-specific patches here (e.g., there is one to deal with the Python issue there), but otherwise keep it close to upstream. In particular, anything that would be generically useful should go upstream first, with the only divergences being things needed for the peculiarities of rustup and/or the build setup.