introduce the ability to browse translation units & inclusions instead of context-free header files

[JamesWidman]

Hi all!

In many projects (e.g. clang, neovim, the mold linker, and many others), there are headers that are not self-contained. Browsing any these header files with clangd results in many spurious errors, apparently because there is no attempt at processing that header within the preprocessor context of a particular inclusion in a translation unit.*

To solve this problem, I'd like to see features like the following:

1. If clangd is invoked with a compile_commands.json, then for each translation unit, it will build the tree of all executions of the #include directive, and, as an LSP extension, and upon request from the editor, clangd will send this tree to the editor.

2. When the cursor is on a translation unit in compile_commands.json and i invoke "goto definition", the language server will jump to the primary source file and select its include-tree as the include-tree associated with the buffer for the primary source file.

3. In that TU, whenever I jump-to-def, I jump not just to a file, but to a particular entry in the include-tree ("ITE") of some translation unit that contains the definition. So from the editor's perspective, a cursor location is no longer just "file+line_number+column_number", but instead "TU+ITE+line_number+column_number"

4. When I use the editor to open a header file without indicating a TU and/or without indicating an ITE, the editor will ask clangd to pick an entry in the include tree of some TU that includes that file, and clangd will report the TU & ITE back to the editor. (The editor might optionally specify a list of TUs that the user recently browsed, and clangd will search those include trees first.) If the header is not used in any translation unit indicated by compile_commands.json, clangd could issue a warning diagnostic (which will be a helpful surprise if you assumed it was being included somewhere).

5. When I'm browsing a TU+ITE, I can:
   5.1. request the editor to toggle the display of the include tree, with a cursor/focus initially on the inclusion that I was initially browsing.
   5.2. "Step out" from that inclusion to its #include directive from the parent inclusion. (When the cursor is in the primary source file and I then step-out, the editor will jump to the entry for the TU in compile_commands.json.)
   5.3. request a list of other ITEs of the same header within the same TU (to which you can then jump) or all TUs. (This would be helpful when trying to make sense of uses of an xmacro header, as we often see in e.g. clang's source code.) The editor could also provide commands to simply jump to the next/previous inclusion of the same file without displaying the list.

(Note, headers included as a consequence of the clang option -include <file> should appear at the top of the include-tree. This means that the "primary" source file (the .c/.cpp file named on the command line) is really a sibling of these headers. And even if there are no uses of -include <file> for a TU, it might be nice if the "primary" source file were treated as a child of the root of the include tree, because we might want to support browsing a virtual file that begins with the #define directives for all of the pre-defined macros and ends with an #include that names the primary source file. If an editor were to support this, it would be a lot more comfortable than hunting down compile flags for the purpose of executing clang -E -dM! So, maybe amend item (2) above to jump to this virtual file, but the editor could provide an option to indicate whether it should bypass the virtual file & jump directly to the "primary" file.)

This could also pair nicely with other UI enhancements; e.g. the editor could support toggling the inline display of an inclusion, with a border-line showing where an inclusion stops and the including-file resumes.

[*] But even when a header is self-contained, when I browse diagnostics, I generally want the editor+LSP_server to think in terms of diagnostics against the translation unit, and not in terms of diagnostics against a [context-free] file.

[HighCommander4]

I agree that this would be a super nice set of features to have.

I believe the features described here are a superset of #45. If anyone is interested in making progress on this, I think tackling the implementation challenges and design decisions discussed in the following comments in particular would be a good starting point:

• Support non-self-contained files #45 (comment)
• Support non-self-contained files #45 (comment)