Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Ship a custom LLDB with Rust support #48168

Open
alexcrichton opened this issue Feb 12, 2018 · 33 comments
Open

Ship a custom LLDB with Rust support #48168

alexcrichton opened this issue Feb 12, 2018 · 33 comments

Comments

@alexcrichton
Copy link
Member

@alexcrichton alexcrichton commented Feb 12, 2018

I hear that @tromey has been doing some awesome work with Rust-specific support in LLDB, and we should enable easily shipping that work to users! This is intended to be a checklist/tracking issue of sorts of the work needed to be done to enable this.

  • One of the first things we'll need to do is to add it to the Rust source tree (off by default). This will probably involve adding a submodule and adding rustbuild logic to build LLDB. This will currently, as of this writing, require syncing the LLDB work to get based off the same LLVM version we have in tree, presently the release_60 branch.
  • Next up we'll want to confirm that LLDB works with sccache and is being cached properly when using sccache. The change here would basically be ensuring that --enable-ccache works for LLDB locally.
  • After that we'll want to build LLDB on the dist bots, but not upload it yet. This'll help us evaluate impacts on cycle time, if any. The change here should mostly just be tweaking flags in travis/appveyor/dockerfiles
  • Next we'd add rustbuild support for creating a new LLDB component that we'd publish on the dist bots. This would probably look similar to the support for RLS (ish).
  • And finally we'd teach rustup.rs about the LLDB component, allowing rustup component add lldb (or whatever the name is), but this is much farther down the line!
@michaelwoerister

This comment was marked as resolved.

Copy link
Contributor

@michaelwoerister michaelwoerister commented Feb 21, 2018

cc me

@tromey

This comment has been minimized.

Copy link
Contributor

@tromey tromey commented Aug 13, 2018

@tromey

This comment has been minimized.

Copy link
Contributor

@tromey tromey commented Aug 15, 2018

The first item is done. I'm testing the second item now. I had thought that the third item would be covered by the .travis.yml change in the patch for the first item -- is that not so? I think the rustbuild bits are done too (?). I haven't looked into rustup.rs yet.

@alexcrichton

This comment has been minimized.

Copy link
Member Author

@alexcrichton alexcrichton commented Aug 15, 2018

@tromey I think we'll need to wait for a new nightly to see whether or not rustup works, at that point we should have a look at the manifest and it ay already work!

@tromey

This comment has been minimized.

Copy link
Contributor

@tromey tromey commented Aug 15, 2018

Thanks. Also, a local --enable-ccache build worked fine.

@michaelwoerister

This comment has been minimized.

Copy link
Contributor

@michaelwoerister michaelwoerister commented Aug 16, 2018

@tromey, I saw that dynamically linking to LLVM is disabled when shipping LLDB:

!want_lldb {

Is that a hard requirement? Because we might want to enable dynamic linking when compiling LLVM with ThinLTO (see #53245).

@tromey

This comment has been minimized.

Copy link
Contributor

@tromey tromey commented Aug 16, 2018

Is that a hard requirement?

I don't think so.

@tromey

This comment has been minimized.

Copy link
Contributor

@tromey tromey commented Aug 17, 2018

This reports that it didn't work: https://internals.rust-lang.org/t/rust-debugging-quest/6753/13?u=tromey

I haven't looked into it yet.

@tromey

This comment has been minimized.

Copy link
Contributor

@tromey tromey commented Aug 17, 2018

lldb isn't mentioned in channel-rust-nightly.toml, which I guess it should be. I don't know how this is made.

@cynecx

This comment has been minimized.

Copy link
Contributor

@cynecx cynecx commented Aug 17, 2018

@tromey I am not familiar with the code but if I follow the code correctly, there might be an issue here:

let version = match *self.cached_version(pkgname) {
Some(ref version) => version.clone(),
None => {
println!("Skipping package {}", pkgname);
return;
}
};

cached_version(...) is called with "lldb-preview":

} else if component == "lldb" || component == "lldb-preview" {
&self.lldb_version

lldb_version is initialized here:

self.lldb_version = self.version("lldb", "x86_64-unknown-linux-gnu");

So I think the problem here is that call to version with x86_64-unknown-linux-gnu as target, it basically tries to extract the version of that particular target archive. And according to #52716, lldb is only built for macOS on nightly. Therefore the call version returns None and lldb_version is then None.

@tromey

This comment has been minimized.

Copy link
Contributor

@tromey tromey commented Aug 17, 2018

Thanks for your note. That's a good find!

@tromey

This comment has been minimized.

Copy link
Contributor

@tromey tromey commented Aug 17, 2018

I have the obvious patch here: https://github.com/tromey/rust/tree/lldb-manifest-fix - but I don't know how to test it locally.

@japaric

This comment has been minimized.

Copy link
Member

@japaric japaric commented Aug 17, 2018

I see that #52716 is building LLDB for macOS. What's the (long term) plan for Windows and Linux? Will we also ship LLDB on those two OSes? At some point I heard that we might ship GDB on Linux (because GDB is better, or maybe easier to ship, than LLDB on Linux?) and then that we'll probably not do that (because system GDB is good enough?).

For the embedded use case having a debugger with support for architectures other than x86 (notably ARM and RISCV) shipped with the compiler would be godsend as the user would have one less tool to install (e.g. arm-none-eabi-gdb) or at least the installation would be straightforward (rustup component add lldb-preview). Remote debugging of microcontrollers, which have non-x86 processors, is a must have for professional embedded development; making the setup process simpler would be a big win for embedded Rust.

In that regard, an lldb component would fit the bill as it has multiarch support by default, and a gdb component would too but only if it's compiled with --enable-targets=all or similar (Most (?) distros ship GDB with support only for the host architecture; that's why you usually need to install a second, prefixed GDB for embedded development). IME (from several months ago) GDB integrates much better with embedded debugging tools (e.g. OpenOCD) but I expect that LLDB will eventually catch up (this PR indicates progress in that area).

cc @dvc94ch

@tromey

This comment has been minimized.

Copy link
Contributor

@tromey tromey commented Aug 17, 2018

The main reason lldb is only built for macOS is that on other platforms, the Python dependency is hard to satisfy. On Windows I guess we'd have to ship Python. On Linux, ideally you'd use the system Python, but I think there is no compatibility across distros, due to the Python 2/3 schism, and perhaps also due to differences in filesystem layout (? I can't recall).

On Linux, there may be an additional difficulty setting the path for the system's separate debuginfo. However I haven't really looked to see what distros other than Fedora do here. It's possible there are also other dependency-related issues.

The upstream situations with gdb and lldb are different. gdb accepted all the Rust code, and I'm the maintainer for that, so it is very easy for us to add rust-related improvements there. The main issue with gdb is that it is on a twice-per-year release schedule, so there is some lag relative to rustc. This is one reason that it would be good to ship a gdb -- we could synchronize rust improvements better.

lldb, on the other hand, did not want the rust plugin until it has been proven. I put this info in the rust lldb wiki but here's the message: http://lists.llvm.org/pipermail/lldb-dev/2018-January/013213.html (IIRC there are some other notes that month including some discussion of rust specifically). So, we have to ship a fork and I have no concrete plans to upstream anything. In a sense this is good because it avoids the synchronization problem.

@tromey

This comment has been minimized.

Copy link
Contributor

@tromey tromey commented Aug 17, 2018

I sent the PR: #53452

@dvc94ch

This comment has been minimized.

Copy link
Contributor

@dvc94ch dvc94ch commented Aug 17, 2018

The main reason lldb is only built for macOS is that on other platforms, the Python dependency is hard to satisfy. On Windows I guess we'd have to ship Python. On Linux, ideally you'd use the system Python, but I think there is no compatibility across distros, due to the Python 2/3 schism, and perhaps also due to differences in filesystem layout (? I can't recall).

I think every distro supports both python2 and python3. What python simlinks to may be different. If you have a specific issue can you provide more info so that I can help debug it?

On Linux, there may be an additional difficulty setting the path for the system's separate debuginfo. However I haven't really looked to see what distros other than Fedora do here. It's possible there are also other dependency-related issues.

I think users can set this in their .gdbinit in their home directory. I'm sure we can figure this one out too...

Are there any concrete steps that can be taken to get lldb distributed with rust on linux?

@tromey

This comment has been minimized.

Copy link
Contributor

@tromey tromey commented Aug 17, 2018

If you have a specific issue can you provide more info so that I can help debug it?

If we ship an lldb that is linked against some system Python, then that version of Python has to exist on the user's machine. But AFAIK we can't influence whether this is true or not. Perhaps it's even possible to have an install without Python. Also, library paths differ between distros (IIRC Debian and Fedora handle multi-arch differently), though perhaps this could be worked around in rustup.

If the dependency problems can be solved somehow then we should also ship gdb on Linux. Less sure about shipping either debugger on Windows.

@dvc94ch

This comment has been minimized.

Copy link
Contributor

@dvc94ch dvc94ch commented Aug 17, 2018

How about building gdb statically? It seems to be supported https://stackoverflow.com/questions/9364685/how-i-can-static-build-gdb-from-source

@tromey

This comment has been minimized.

Copy link
Contributor

@tromey tromey commented Aug 17, 2018

How about building gdb statically?

Will that work if Python code loads a C extension? I wouldn't think so but I am not 100% certain.

@dvc94ch

This comment has been minimized.

Copy link
Contributor

@dvc94ch dvc94ch commented Aug 19, 2018

Will that work if Python code loads a C extension? I wouldn't think so but I am not 100% certain.

Is that necessary for the rust gdb part? I think that 99% of users will use a ide for debugging. I currently use a python gdb gui which is a huge improvement over just using the cli, but I'll be adding lldb/gdb support to https://github.com/theia-ide/theia-rust-extension after eclipse-theia/theia#2447 lands.

@tromey

This comment has been minimized.

Copy link
Contributor

@tromey tromey commented Aug 20, 2018

Is that necessary for the rust gdb part?

It isn't necessary in the sense that gdb will still work without it. However, I'd rather not limit what users can do. For example I use this facility all the time.

I currently use a python gdb gui

OOC what is it?

One idea I had is that maybe we could have rustup check for required dependencies before installing. Then, if the dependencies were missing, it could point users to a wiki page explaining how to install the dependencies for their distro. It isn't just Python, for best results gdb also needs zlib, expat, curses, and lzma (and maybe mpfr as well).

The second part of this idea would be to pick sufficiently old versions of these libraries to build against so that the dependencies can be satisfied on all the target distros, whatever those are. Also, we'd modify gdb to find the distro's separate debug directory and installed gdb python code.

At runtime the dependencies would be satisfied by having rustup (continue to) set LD_LIBRARY_PATH in a distro-specific way.

lldb would work similarly, though I think there are fewer dependencies, since it doesn't support some of the Linux-specific things that gdb does.

@dvc94ch

This comment has been minimized.

Copy link
Contributor

@dvc94ch dvc94ch commented Aug 20, 2018

I currently use a python gdb gui

OOC what is it?

https://github.com/cyrus-and/gdb-dashboard

@dvc94ch

This comment has been minimized.

Copy link
Contributor

@dvc94ch dvc94ch commented Aug 20, 2018

One idea I had is that maybe we could have rustup check for required dependencies before installing. Then, if the dependencies were missing, it could point users to a wiki page explaining how to install the dependencies for their distro. It isn't just Python, for best results gdb also needs zlib, expat, curses, and lzma (and maybe mpfr as well).

One problem with that is that some distros (fedora) don't provide symlinks to old libs. So if I compile a lib with a dependency on libreadline with travis which uses libreadline.so.6 the dep isn't found on fedora which only has libreadline.so.7. A possible solution would be to change the lib names to libreadline.so in the binary before distributing. I currently am working around this on my machine by adding the appropriate symlinks. Distributing software on Linux in a distro independent way is a PITA.

@tromey

This comment has been minimized.

Copy link
Contributor

@tromey tromey commented Aug 20, 2018

We'd have to survey the dependencies on the distros we care about to see if it's possible. Note that readline in particular needn't be an issue because gdb vendors it (distros tend to disable this but we don't have to).

@tromey

This comment has been minimized.

Copy link
Contributor

@tromey tromey commented Aug 29, 2018

Let's move the gdb discussion to #34457.

@tromey

This comment has been minimized.

Copy link
Contributor

@tromey tromey commented Aug 29, 2018

rustup PR is here: rust-lang/rustup#1492

tromey added a commit to tromey/rust that referenced this issue Sep 5, 2018
We're shipping a rust-enabled lldb, but the "lldb" executable is not
installed into the "bin" directory by rustup.  See the discussion in
rust-lang/rustup#1492 for
background on this decision.  There, we agreed to have rust-lldb
prefer the rust-enabled lldb if it is installed.  This patch changes
rust-lldb to look in the sysroot and use the lldb found there, if any.

See issue rust-lang#48168
tromey added a commit to tromey/rust that referenced this issue Sep 7, 2018
We're shipping a rust-enabled lldb, but the "lldb" executable is not
installed into the "bin" directory by rustup.  See the discussion in
rust-lang/rustup#1492 for
background on this decision.  There, we agreed to have rust-lldb
prefer the rust-enabled lldb if it is installed.

This patch changes dist.rs to put lldb into rustlib, following what
was done for the other LLVM tools in rust-lang#53955, and then fixes rust-lldb
to prefer that lldb, if it exists.

See issue rust-lang#48168
kennytm added a commit to kennytm/rust that referenced this issue Sep 7, 2018
…alexcrichton

Have rust-lldb look for the rust-enabled lldb

We're shipping a rust-enabled lldb, but the "lldb" executable is not
installed into the "bin" directory by rustup.  See the discussion in
rust-lang/rustup#1492 for
background on this decision.  There, we agreed to have rust-lldb
prefer the rust-enabled lldb if it is installed.  This patch changes
rust-lldb to look in the sysroot and use the lldb found there, if any.

See issue rust-lang#48168
kennytm added a commit to kennytm/rust that referenced this issue Sep 7, 2018
…alexcrichton

Have rust-lldb look for the rust-enabled lldb

We're shipping a rust-enabled lldb, but the "lldb" executable is not
installed into the "bin" directory by rustup.  See the discussion in
rust-lang/rustup#1492 for
background on this decision.  There, we agreed to have rust-lldb
prefer the rust-enabled lldb if it is installed.  This patch changes
rust-lldb to look in the sysroot and use the lldb found there, if any.

See issue rust-lang#48168
kennytm added a commit to kennytm/rust that referenced this issue Sep 8, 2018
…alexcrichton

Have rust-lldb look for the rust-enabled lldb

We're shipping a rust-enabled lldb, but the "lldb" executable is not
installed into the "bin" directory by rustup.  See the discussion in
rust-lang/rustup#1492 for
background on this decision.  There, we agreed to have rust-lldb
prefer the rust-enabled lldb if it is installed.  This patch changes
rust-lldb to look in the sysroot and use the lldb found there, if any.

See issue rust-lang#48168
kennytm added a commit to kennytm/rust that referenced this issue Sep 8, 2018
…alexcrichton

Have rust-lldb look for the rust-enabled lldb

We're shipping a rust-enabled lldb, but the "lldb" executable is not
installed into the "bin" directory by rustup.  See the discussion in
rust-lang/rustup#1492 for
background on this decision.  There, we agreed to have rust-lldb
prefer the rust-enabled lldb if it is installed.  This patch changes
rust-lldb to look in the sysroot and use the lldb found there, if any.

See issue rust-lang#48168
@tromey

This comment has been minimized.

Copy link
Contributor

@tromey tromey commented Sep 14, 2018

Latest problem is #54126.

@vadimcn

This comment has been minimized.

Copy link
Contributor

@vadimcn vadimcn commented Sep 26, 2018

Regarding Python dependency on Linux: I doubt that any distro will drop Python 2.7 before 2020, even if it might not be the default. So for the next year we could ship LLDB linked to Python 2.7.

Some time after that, Python 2 will start going extinct, so we'll need to switch to Python 3. Which is going to be a headache, because a new version comes every once in a while. At that point, one of the following needs to happen:

  • LLDB adopts Rust plugin in the main tree,
  • We provide several versions of rust-lldb, compatible with latest 2 or 3 Python versions,
  • We bundle Python with rust-lldb,
  • SWIG finally supports stable Python ABI, which will make it possible to use any Python version past 3.2 (some changes will also be needed on LLDB side as well, but those should be minimal).
@tromey

This comment has been minimized.

Copy link
Contributor

@tromey tromey commented Sep 27, 2018

I doubt that any distro will drop Python 2.7 before 2020, even if it might not be the default

One question is whether we can rely on it being installed. But maybe we could teach rustup to check, if it can't already (I don't know).

LLDB adopts Rust plugin in the main tree,

I don't think this will change anything here, because even if lldb accepts the Rust plugin, we will probably still want to ship lldb, so that we aren't blocked on making debuginfo improvements. There's an analogous case for shipping gdb, see #34457.

I'm generally agreed on the other points.

@crlf0710

This comment has been minimized.

Copy link
Contributor

@crlf0710 crlf0710 commented Oct 13, 2018

On http://lldb.llvm.org/build.html it mentions that python dependency(and some other dependencies) can be opt-out by adding -DLLDB_DISABLE_PYTHON=1 but i'm not sure how much it hurts functionality. Maybe someone can give it a try?

The full list:

-DLLDB_DISABLE_LIBEDIT=1
-DLLDB_DISABLE_CURSES=1
-DLLDB_DISABLE_PYTHON=1
-DLLVM_ENABLE_TERMINFO=0
@tromey

This comment has been minimized.

Copy link
Contributor

@tromey tromey commented Oct 25, 2018

but i'm not sure how much it hurts functionality

I think we want Python, because it is how the pretty-printers in rust-lldb work; and also Python support in lldb is required by the debuginfo tests.

@ncihnegn

This comment has been minimized.

Copy link

@ncihnegn ncihnegn commented Aug 22, 2019

Is lldb still shipped as a component on macOS?

@mati865

This comment has been minimized.

Copy link
Contributor

@mati865 mati865 commented Aug 22, 2019

@ncihnegn it was dropped during migration to LLVM 9: rust-lang/llvm-project#19

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

None yet
You can’t perform that action at this time.