x/tools/gopls: support something like compile_commands.json

[paulthomson]

Build systems such as CMake and Ninja can output a compilation database -- a compile_commands.json file that contains every command that was executed to compile each C++ source file.

Bear intercepts process forking/creation to get the compile_commands.json from any build system, even those that do not natively support outputting a compile_commands.json file.

A compile_commands.json file can be used by Clang tooling, such as clangd to get autocompletion, go-to-definition, etc. in editors like Vim, Emacs, and VSCode. The CLion IDE supports opening a compile_commands.json file as a C++ project to get the usual autocompletion and refactoring features.

The nice thing about compile_commands.json is that, regardless of your build system, you know that it must be possible to log each invocation of the compiler, and thus get a compile_commands.json file, one way or another. Furthermore, regardless of the various project models or abstractions that various tools/IDEs/build systems come up with, you know it must be possible to get all the required information from each invocation of the compiler, and thus any tool should be able to import a compile_commands.json file to create its project model.

Is there any way that gopls could support using a compile_commands.json file for golang (or a similar type of file that essentially is a log of every invocation of the go compiler)?

Note: this could also be a good "source-of-truth" and useful for cross-checking other ways of getting the project model. I.e. log the actual invocations of the go compiler to build a project model; we should get the same project model when obtained via other means, like when relying on the standard GOPATH directory structure.

Note: the compilation database / compile_commands.json spec currently does not have fields for storing environment variables, which I assume would be required for go.

[heschi]

I don't think this is feasible in the current architecture. We use the output of go list -json via go/packages to get much more than just a list of files. What are you actually trying to accomplish with this request? Is there some other build tool you want to use? Is it possible to implement a go/packages driver for it?

[paulthomson]

We use the output of go list -json via go/packages to get much more than just a list of files.

To clarify, the compile_commands.json file contains entries like this:

(for C++)

{
    "directory": "/git/mytool",
    "command": "external/androidndk/ndk/toolchains/llvm/prebuilt/linux-x86_64/bin/clang -gcc-toolchain external/androidndk/ndk/toolchains/aarch64-linux-android-4.9/prebuilt/linux-x86_64 -target aarch64-none-linux-android -fpic -isystemexternal/androidndk/ndk/sysroot/usr/include/aarch64-linux-android '-D__ANDROID_API__=23' -no-canonical-prefixes -Wno-invalid-command-line-argument -Wno-unused-command-line-argument -funwind-tables -fstack-protector-strong -fno-addrsig '-Werror=return-type' '-Werror=int-to-pointer-cast' '-Werror=pointer-to-int-cast' '-Werror=implicit-function-declaration' -O0 -g -UNDEBUG -MD -MF bazel-out/android-arm64-v8a-dbg/bin/external/com_google_protobuf/_objs/protobuf_lite/generated_message_table_driven_lite.pic.d '-frandom-seed=bazel-out/android-arm64-v8a-dbg/bin/external/com_google_protobuf/_objs/protobuf_lite/generated_message_table_driven_lite.pic.o' -fPIC -iquote external/com_google_protobuf -iquote bazel-out/android-arm64-v8a-dbg/bin/external/com_google_protobuf -isystem external/com_google_protobuf/src -isystem bazel-out/android-arm64-v8a-dbg/bin/external/com_google_protobuf/src '-std=c++11' -DHAVE_PTHREAD -DHAVE_ZLIB -Woverloaded-virtual -Wno-sign-compare -Wno-unused-function -Wno-write-strings '--sysroot=external/androidndk/ndk/platforms/android-23/arch-arm64' -isystem external/androidndk/ndk/sources/cxx-stl/llvm-libc++/include -isystem external/androidndk/ndk/sources/cxx-stl/llvm-libc++abi/include -isystem external/androidndk/ndk/sources/android/support/include -isystemexternal/androidndk/ndk/sysroot/usr/include -c external/com_google_protobuf/src/google/protobuf/generated_message_table_driven_lite.cc -o bazel-out/android-arm64-v8a-dbg/bin/external/com_google_protobuf/_objs/protobuf_lite/generated_message_table_driven_lite.pic.o",
    "file": "external/com_google_protobuf/src/google/protobuf/generated_message_table_driven_lite.cc"
},

Thus it is more than just a list of files. It is all information needed for full semantic analysis of all code that was compiled. For go, perhaps it would contain entries made up of things like the following:

directory: /usr/local/google/home/paulthomson/.cache/bazel/_bazel_paulthomson/fa056b76b9eefba3521939bad4da8c7e/execroot/project

env:

CGO_ENABLED=1
GOARCH=amd64
GOOS=linux
GOROOT=external/go_sdk
GOROOT_FINAL=GOROOT

package: external/org_golang_google_grpc/codes

command: bazel-out/host/bin/external/go_sdk/builder compile -sdk external/go_sdk -installsuffix linux_amd64 -tags analytics,crashreporting -src external/org_golang_google_grpc/codes/code_string.go -src external/org_golang_google_grpc/codes/codes.go -o bazel-out/host/bin/external/org_golang_google_grpc/codes/linux_amd64_stripped/go_default_library%/google.golang.org/grpc/codes.a -package_list bazel-out/host/bin/external/go_sdk/packages.txt -p google.golang.org/grpc/codes -- -trimpath .

From there, I assume it would be possible to derive further information that exactly matches the way the files were built by the build system.

What are you actually trying to accomplish with this request? Is there some other build tool you want to use?

In reality, our team is using Bazel :-) So eventually, I am sure gopls will work. My fear though is that there are other build systems. And that, even for Bazel, there may be quirks (like updates to Bazel, build system hacks, weird edge cases, etc.) that cause the support to be imperfect. In contrast, the above approach (in theory) cannot fail; the tool (e.g. clangd or gopls) has exactly the same information as the compiler. By definition, it cannot be missing any information or have the wrong information.

Is it possible to implement a go/packages driver for it?

Yes, perhaps it would be possible to make a driver that reads a "compile_commands-like" file (for all go files that were compiled) and returns the relevant JSON. This issue was mainly just to ask whether the team has considered relying (directly or indirectly) on the actual compilation commands executed, and whether this seems like it may have advantages over relying on a driver for each build system? Could these drivers get details incorrect? Etc.

[KantarBruceAdams]

Just to add a note to this. I ran into a hitch with sonarcloud not understanding cgo. A "compilation database" file would be one way to work around it.

[hyangah]

Is it possible to implement a go/packages driver for it?

Yes, perhaps it would be possible to make a driver that reads a "compile_commands-like" file (for all go files that were compiled) and returns the relevant JSON. This issue was mainly just to ask whether the team has considered relying (directly or indirectly) on the actual compilation commands executed, and whether this seems like it may have advantages over relying on a driver for each build system? Could these drivers get details incorrect? Etc.

Almost three years had passed. I doubt gopls wants to change its architecture to support this now - cc @findleyr
If the goal is to consume compile_commands.json-like input, it sounds like writing a go/packages driver is the way to achieve this :-(. There is already a go/packages driver for bazel.

I am afraidcompile_commands.json itself is sufficient or abstract enough for gopls to operate with. Gopls needs info more than what compile_commands.json or an extension of it may carry (e.g. info available outside compilation stage such as go module information in gc).

Just to add a note to this. I ran into a hitch with sonarcloud not understanding cgo. A "compilation database" file would be one way to work around it.

If you are looking for a way to produce compilation info, I am afraid that gopls is not the right place. (I am not sure if this is sufficient but go build -x prints underlying tool invocations and go list -json also outputs CgoFiles and CFiles)

[KantarBruceAdams]

I think I misunderstood this ticket to be about generating rather than consuming a compile_commands.json. I have moved the sonar discussion to a new ticket - #56706 . That said surely the purpose of a language server protocol is to interface with other tools so have it cause something to generate something like a compile_commands.json would not be out of place.

[hyangah]

the purpose of a language server protocol is to interface with other tools

the scope of LSP is determined in its spec and I don't see how generation of compile_commands.json can be in the scope. Even it looks like clangd that adopts compile_commands.json is only a consumer, not a compilation database generator.

I hope the discussion in the other issue helps you find a solution. I see clang commands appearing in my go build -x command output, but I guess that's not complete.