Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

x/tools/gopls: support something like `compile_commands.json` #36995

paulthomson opened this issue Feb 3, 2020 · 2 comments

x/tools/gopls: support something like `compile_commands.json` #36995

paulthomson opened this issue Feb 3, 2020 · 2 comments


Copy link

@paulthomson paulthomson commented Feb 3, 2020

Build systems such as CMake and Ninja can output a compilation database -- a compile_commands.json file that contains every command that was executed to compile each C++ source file.

Bear intercepts process forking/creation to get the compile_commands.json from any build system, even those that do not natively support outputting a compile_commands.json file.

A compile_commands.json file can be used by Clang tooling, such as clangd to get autocompletion, go-to-definition, etc. in editors like Vim, Emacs, and VSCode. The CLion IDE supports opening a compile_commands.json file as a C++ project to get the usual autocompletion and refactoring features.

The nice thing about compile_commands.json is that, regardless of your build system, you know that it must be possible to log each invocation of the compiler, and thus get a compile_commands.json file, one way or another. Furthermore, regardless of the various project models or abstractions that various tools/IDEs/build systems come up with, you know it must be possible to get all the required information from each invocation of the compiler, and thus any tool should be able to import a compile_commands.json file to create its project model.

Is there any way that gopls could support using a compile_commands.json file for golang (or a similar type of file that essentially is a log of every invocation of the go compiler)?

Note: this could also be a good "source-of-truth" and useful for cross-checking other ways of getting the project model. I.e. log the actual invocations of the go compiler to build a project model; we should get the same project model when obtained via other means, like when relying on the standard GOPATH directory structure.

Note: the compilation database / compile_commands.json spec currently does not have fields for storing environment variables, which I assume would be required for go.

@gopherbot gopherbot added this to the Unreleased milestone Feb 3, 2020
Copy link

@heschik heschik commented Feb 3, 2020

I don't think this is feasible in the current architecture. We use the output of go list -json via go/packages to get much more than just a list of files. What are you actually trying to accomplish with this request? Is there some other build tool you want to use? Is it possible to implement a go/packages driver for it?

Copy link

@paulthomson paulthomson commented Feb 4, 2020

We use the output of go list -json via go/packages to get much more than just a list of files.

To clarify, the compile_commands.json file contains entries like this:

(for C++)

    "directory": "/git/mytool",
    "command": "external/androidndk/ndk/toolchains/llvm/prebuilt/linux-x86_64/bin/clang -gcc-toolchain external/androidndk/ndk/toolchains/aarch64-linux-android-4.9/prebuilt/linux-x86_64 -target aarch64-none-linux-android -fpic -isystemexternal/androidndk/ndk/sysroot/usr/include/aarch64-linux-android '-D__ANDROID_API__=23' -no-canonical-prefixes -Wno-invalid-command-line-argument -Wno-unused-command-line-argument -funwind-tables -fstack-protector-strong -fno-addrsig '-Werror=return-type' '-Werror=int-to-pointer-cast' '-Werror=pointer-to-int-cast' '-Werror=implicit-function-declaration' -O0 -g -UNDEBUG -MD -MF bazel-out/android-arm64-v8a-dbg/bin/external/com_google_protobuf/_objs/protobuf_lite/generated_message_table_driven_lite.pic.d '-frandom-seed=bazel-out/android-arm64-v8a-dbg/bin/external/com_google_protobuf/_objs/protobuf_lite/generated_message_table_driven_lite.pic.o' -fPIC -iquote external/com_google_protobuf -iquote bazel-out/android-arm64-v8a-dbg/bin/external/com_google_protobuf -isystem external/com_google_protobuf/src -isystem bazel-out/android-arm64-v8a-dbg/bin/external/com_google_protobuf/src '-std=c++11' -DHAVE_PTHREAD -DHAVE_ZLIB -Woverloaded-virtual -Wno-sign-compare -Wno-unused-function -Wno-write-strings '--sysroot=external/androidndk/ndk/platforms/android-23/arch-arm64' -isystem external/androidndk/ndk/sources/cxx-stl/llvm-libc++/include -isystem external/androidndk/ndk/sources/cxx-stl/llvm-libc++abi/include -isystem external/androidndk/ndk/sources/android/support/include -isystemexternal/androidndk/ndk/sysroot/usr/include -c external/com_google_protobuf/src/google/protobuf/ -o bazel-out/android-arm64-v8a-dbg/bin/external/com_google_protobuf/_objs/protobuf_lite/generated_message_table_driven_lite.pic.o",
    "file": "external/com_google_protobuf/src/google/protobuf/"

Thus it is more than just a list of files. It is all information needed for full semantic analysis of all code that was compiled. For go, perhaps it would contain entries made up of things like the following:

directory: /usr/local/google/home/paulthomson/.cache/bazel/_bazel_paulthomson/fa056b76b9eefba3521939bad4da8c7e/execroot/project



package: external/org_golang_google_grpc/codes

command: bazel-out/host/bin/external/go_sdk/builder compile -sdk external/go_sdk -installsuffix linux_amd64 -tags analytics,crashreporting -src external/org_golang_google_grpc/codes/code_string.go -src external/org_golang_google_grpc/codes/codes.go -o bazel-out/host/bin/external/org_golang_google_grpc/codes/linux_amd64_stripped/go_default_library%/ -package_list bazel-out/host/bin/external/go_sdk/packages.txt -p -- -trimpath .

From there, I assume it would be possible to derive further information that exactly matches the way the files were built by the build system.

What are you actually trying to accomplish with this request? Is there some other build tool you want to use?

In reality, our team is using Bazel :-) So eventually, I am sure gopls will work. My fear though is that there are other build systems. And that, even for Bazel, there may be quirks (like updates to Bazel, build system hacks, weird edge cases, etc.) that cause the support to be imperfect. In contrast, the above approach (in theory) cannot fail; the tool (e.g. clangd or gopls) has exactly the same information as the compiler. By definition, it cannot be missing any information or have the wrong information.

Is it possible to implement a go/packages driver for it?

Yes, perhaps it would be possible to make a driver that reads a "compile_commands-like" file (for all go files that were compiled) and returns the relevant JSON. This issue was mainly just to ask whether the team has considered relying (directly or indirectly) on the actual compilation commands executed, and whether this seems like it may have advantages over relying on a driver for each build system? Could these drivers get details incorrect? Etc.

@golang golang deleted a comment from gopherbot Feb 4, 2020
@stamblerre stamblerre modified the milestones: Unreleased, gopls/v1.0.0 Feb 5, 2020
@stamblerre stamblerre modified the milestones: gopls/v1.0.0, Unreleased Mar 12, 2020
@stamblerre stamblerre removed this from the Unreleased milestone Jul 23, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

None yet
4 participants
You can’t perform that action at this time.