Expose generated Go files to editors and IDEs

[jayconrod]

When a library is created with go_proto_library or some other mechanism that generates Go code (.pb.go files in this case), the generated sources files are not exposed to editors and IDEs. Consequently, code completion and refactoring tools don't work, lowering developer productivity.

The Go team is developing a workspace abstraction mechanism that will allow editors and Bazel (and other build systems) to communicate with each other without the need for direct integration. This is the long-term solution to this problem.

This issue will track progress on the workspace abstraction and will track editor awareness of code generated with Bazel.

[jayconrod]

This issue is in response to Bazel build, protobuf and code completion.

TODO: update answer when this issue is resolved.

[excavador]

What the status of the task?

[excavador]

So, I have proposal how to close this issue.

Suppose you generated code:
bazel-genfiles/somedir/somefile.go
bazel-genfiles/somedir/anotherfile.go

Let's add symlink
src/somedir => ../bazel-genfiles/somedir

Use in your files
import "somedir"

As result you will get workable code-completition AND workable build.

[jayconrod]

@excavador We're still developing a more complete solution to this (the workspace abstraction I mentioned above). Ideally, tools won't need to be aware of the project layout in use. I don't have any ETA for that yet, but we are making progress.

As you showed, it's possible to work around this with symlinks. You can also copy or symlink generated files into your source directory. Bazel will prefer a generated file if a source file of the same name is present. I'd recommend against checking generated files into VCS though.

[excavador]

Thank you for notify! Will wait. You can consider me as closed beta-tester :)

[raliste]

Any updates?

[excavador]

@raliste it works for me in IDEA Ultimate + Bazel plugin with following limitation: autocomplete works only if all files inside golang package are auto-generated. If mix together auto-generated and manually written file, autogenerated file would not be inspected

[jayconrod]

The workspace abstraction mentioned above is being actively worked on. We've gone through a lot of internal prototypes, and we're in the process of reviewing, testing, and merging one that we're pretty happy with. It's a huge body of code though, and it will take time. Following that, a large number of Go tools will need to be updated to use the workspace abstraction so they can function in "go build", vgo, Bazel, and other kinds of workspaces.

I know it seems like progress is slow, but it's an interface that many, many tools will interact with, and we want to be sure we get it right. It's a huge amount of work, but it is our highest priority task.

[burdiyan]

@jayconrod Are there any public discussions about that? Design docs perhaps?

[jayconrod]

Not yet, but soon. We're still working on simplifying and narrowing the interface.

[vitarb]

Can you briefly explain how this workspaces feature is going to work?
Are there any upcoming changes in the 1.11 related to this?

[jayconrod]

The first public piece of this is golang.org/x/tools/go/packages. At the moment, that is based on go list and is intended to be used by tools that support go build and vgo. We're also working on an workspace abstraction layer that go/packages may eventually be built on. That workspace abstraction would be extensible and could include Bazel. Lot of work still happening there.

[akshayjshah]

@jayconrod Any updates on the proposed workspace abstraction layer?

[jayconrod]

@akshayjshah We've made some progress. The golang.org/x/tools/go/packages API is nearing completion. Our team has migrated a few tools to use it already, but the effort is ongoing.

If you set the GOPACKAGESDRIVER environment variable, or if you have a binary named gopackagesdriver in PATH, go/packages will shell out to that binary for queries. I have an implementation of that binary about 70% done. It works in most situations, but it needs polish, documentation, and tests.

So ideally, once that change is in, you'd be able to build and install the driver into your PATH, and tools that use go/packages will just work in Bazel workspaces (and they'll fall back to the default behavior if there's no WORKSPACE file).

[akshayjshah]

Thanks for the update! Would you be willing to push your driver to a branch so I can take a look? I'm not in a huge rush for you to officially release it, but I'd love to use your rough draft to better understand how these pieces should fit together.

[jayconrod]

@akshayjshah Sure, it's at https://github.com/jayconrod/rules_go/tree/dev-gopackagesdriver if you'd like to take a look. As I said, it's very rough right now.

[pierreis]

That looks awesome, thanks! Is there any update on this?

[jayconrod]

@pierreis Sorry, no updates since my last comment. I'm still planning to work on this this quarter. There are a lot of things going on, but this is one of the higher priority projects.

[jayconrod]

Generated .pb.go files are now available in the go_generated_srcs output group since #1827.

Leaving this issue open though since it's not documented yet.

[vitarb]

@jayconrod is package driver still planned as a feature or are we going to rely on the aspect-based approach mentioned above?

[jayconrod]

A driver is still planned, but I don't have any update to share yet. Implementation-wise, the data will be produced by the builder and accessed through an output group (but the driver will hide those details).

[c4milo]

@jayconrod, has there been any progress on the workspace abstraction?

[steeve]

I have a useable GOPACKAGESDRIVER at #2858

[akshayjshah]

I have a useable GOPACKAGESDRIVER at #2858

This is amazing - thanks, @steeve! 🎉

Hopefully a working packages driver partially unblocks this issue.

[robfig]

What is left to do here?

[gabroo]

Is there an explanation of $GOPACKAGESDRIVER and how one might use it to help the editor/LSP find Go libraries generated by Bazel from protobuf?

I've set the environment variable to the location of gopackagesdriver.sh, but in vim, gopls gives a message no packages returned: packages.Load error

[alexeagle]

The explanation is at https://github.com/bazelbuild/rules_go/wiki/Editor-and-tool-integration

[errb]

What's the status of this issue? GOPACKAGESDRIVER sort of works, but appears to have issues like #3014 which makes it less than ideal for daily use.

Wondering if that's the state where IDE support is currently at right now, or if I'm missing something.

[linzhp]

Uber is planning to invest into GOPACKAGESDRIVER in the coming months. You should see PRs from me, @JamyDev @blico fixing the major issues and potentially improve the performance.

[gonzojive]

I forked the rules_go version and made a few changes. Most notably, I split the package driver into a server and client that communicate using gRPC. The idea was to allow the driver to persist things in memory between runs. It should also be easier to debug because attaching to the server is easier than attaching to the binary launched by the go-pls. The client launches the server automatically, and the server dies after an idle period. The code is still pretty messy. I spent a decent amount of time getting integration testing working beyond the support available from rules_go today. A test workspace gets brought up inside a sandbox with no network connection, which isn't supported by current bazel go integration tests. It is not really in a user friendly state now. But if Uber is about to invest in improved tooling, it might be worth a look. https://github.com/gonzojive/bazelgopackagesdriver is the repo. I may still have local changes that need pushing.

…

On Mon, May 9, 2022, 7:47 AM Zhongpeng Lin ***@***.***> wrote: Uber is planning to invest into GOPACKAGESDRIVER in the coming months. You should see PRs from me, @JamyDev <https://github.com/JamyDev> @blico <https://github.com/blico> fixing the major issues and potentially improve the performance. — Reply to this email directly, view it on GitHub <#512 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAAUO557WCAC2R52RTZL6HTVJEQQJANCNFSM4DOFWFCQ> . You are receiving this because you are subscribed to this thread.Message ID: ***@***.***>

[alexeagle]

There's no mention of JetBrains GoLand editor on this thread or on the wiki https://github.com/bazelbuild/rules_go/wiki/Editor-and-tool-integration

Assuming that this ticket still represents the current state:
https://youtrack.jetbrains.com/issue/GO-8753
it seems like they assume the IntelliJ Bazel plugin is responsible for resolving and indexing generated files, instead of GOPACKAGESDRIVER, which they believe to be broken.

It seems like that's a mistake?

[linzhp]

We made IntelliJ Bazel plugin work for Go internally in Uber, but PRs to upstream our changes have been stalled for months (bazelbuild/intellij#2973).

[OscarVanL]

I'm having some trouble with the language server's compiler IDE suggestions since switching to using the GOPACKAGESDRIVER. Any suggestions would be greatly appreciated.

I have a project that's structured as follows:

github.com/company/repo/
    src/
        project/
            internal/
                a/
            my-service/
                internal/
                    service.go

Before making the change, I had service.go import github.com/company/repo/src/project/internal/a with no issues. Since making the switch I get the IDE error: could not import github.com/company/repo/src/project/internal/a (invalid use of internal package github.com/company/repo/src/project/internal/a)compiler(BrokenImport)

However, this is clearly not a real issue as the code compiles and works successfully.

Any ideas why this change might have introduced this strange error?

[loeffel-io]

Painful but works: bazelbuild/intellij#194 (comment) (need to cp your *.pb files manually)

[alexeagle]

@loeffel-io shouldn't need to be manual, see https://github.com/aspect-build/bazel-examples/tree/main/write_source_files

[sluongng]

Kythe.io, Google's code-intel indexer, indexes most Google's languages into a universal syntax tree using Bazel's aspect (exp action listener, but that is being deprecated in favor of aspect).

This universal syntax tree is then stored into LevelDB(in the open source version) and served via HTTP for codesearch frontend.
There is an LSP implementation that let you proxy the LSP JSON-RPC requests and translate them into Kythe serving API and query LevelDB. More over, kythe provides a VFS to let user read external dependencies sources.

What does this mean?

• Kythe is opensourced, https://github.com/kythe/kythe, but relatively unknown and requires some experimentation to get it working.
• Similar extraction architecture could be applied to different IDE code intel index formatting: LSIF, Jetbrains' shared-index, SourceGraph SCIP, Facebook's Glean etc... In fact, FlareBuild has https://github.com/flarebuild/rules_idea which does very similar thing, but for Java only.

This would be a super interesting area to dig into for those who are interested 🤗

[loeffel-io]

@linzhp is there an easy way to use your plugin version locally?

[linzhp]

@JamyDev Do you think it's possible to open source part of our fork to your personal GitHub account, as a staging area for upstreaming them in the future, and for other people to try out before Google accepts our PRs?

[loeffel-io]

@linzhp For now i could build your plugin myself and publish the zip file - but it looks like your branch is pretty outdated

[chong1144]

Hi @linzhp, thanks for improving this amazing tool. Any plan to issue a PR on https://github.com/linzhp/rules_go/? Looking forward to use this tool inside my repository!

[linzhp]

If we are going to send a PR, it will most likely go to https://github.com/bazelbuild/intellij

[loeffel-io]

Improved workaround for now for a proto external repository (no intellij Go Modules toggle required):

• create a go.mod file in your external repository (can normally run without it)
• copy your generated .pb files manually (or Expose generated Go files to editors and IDEs #512 (comment))
• use the go.mod replace to link to your local external repository: replace github.com/xxx/proto => ../proto

[jayconrod]

Closing this as done. Actually it's been done for a couple years I think.

The wiki page Editor setup has instructions on how you can get this to work on editors that use gopls. Editor and tool integration explains how it works. And, shameless plug, I gave a talk about it.

Thanks to everyone who worked on this, special shout-out to @steeve @linzhp and @fmeum.

If you find bugs or have feature requests (including those unaddressed in comments above), please open new issues, narrowly scoped so they're easier to track.