Expose generated Go files to editors and IDEs #512
Comments
This issue is in response to Bazel build, protobuf and code completion. TODO: update answer when this issue is resolved. |
What the status of the task? |
So, I have proposal how to close this issue. Suppose you generated code: Let's add symlink Use in your files As result you will get workable code-completition AND workable build. |
@excavador We're still developing a more complete solution to this (the workspace abstraction I mentioned above). Ideally, tools won't need to be aware of the project layout in use. I don't have any ETA for that yet, but we are making progress. As you showed, it's possible to work around this with symlinks. You can also copy or symlink generated files into your source directory. Bazel will prefer a generated file if a source file of the same name is present. I'd recommend against checking generated files into VCS though. |
Thank you for notify! Will wait. You can consider me as closed beta-tester :) |
Any updates? |
@raliste it works for me in IDEA Ultimate + Bazel plugin with following limitation: autocomplete works only if all files inside golang package are auto-generated. If mix together auto-generated and manually written file, autogenerated file would not be inspected |
The workspace abstraction mentioned above is being actively worked on. We've gone through a lot of internal prototypes, and we're in the process of reviewing, testing, and merging one that we're pretty happy with. It's a huge body of code though, and it will take time. Following that, a large number of Go tools will need to be updated to use the workspace abstraction so they can function in "go build", vgo, Bazel, and other kinds of workspaces. I know it seems like progress is slow, but it's an interface that many, many tools will interact with, and we want to be sure we get it right. It's a huge amount of work, but it is our highest priority task. |
@jayconrod Are there any public discussions about that? Design docs perhaps? |
Not yet, but soon. We're still working on simplifying and narrowing the interface. |
Can you briefly explain how this workspaces feature is going to work? |
The first public piece of this is |
@jayconrod Any updates on the proposed workspace abstraction layer? |
@akshayjshah We've made some progress. The If you set the So ideally, once that change is in, you'd be able to build and install the driver into your |
Thanks for the update! Would you be willing to push your driver to a branch so I can take a look? I'm not in a huge rush for you to officially release it, but I'd love to use your rough draft to better understand how these pieces should fit together. |
@akshayjshah Sure, it's at https://github.com/jayconrod/rules_go/tree/dev-gopackagesdriver if you'd like to take a look. As I said, it's very rough right now. |
That looks awesome, thanks! Is there any update on this? |
@pierreis Sorry, no updates since my last comment. I'm still planning to work on this this quarter. There are a lot of things going on, but this is one of the higher priority projects. |
Generated .pb.go files are now available in the Leaving this issue open though since it's not documented yet. |
@jayconrod is package driver still planned as a feature or are we going to rely on the aspect-based approach mentioned above? |
A driver is still planned, but I don't have any update to share yet. Implementation-wise, the data will be produced by the builder and accessed through an output group (but the driver will hide those details). |
Just in case: I am wrote to this ticket, but after that I found IDEA plugin which solved my problems completely: https://plugins.jetbrains.com/plugin/8609-bazel |
I've been digging in to this a little lately. The aspect approach does seem to be too slow, and it's causing bazel to run out of memory when I try to use it an a large go project. Adding the json generation to GoCompilePkg seems like a much better idea. gopls seems to be the main client of go packages, and it always uses the same configuration. Maybe we should optimize for that first? It doesn't require types, so it seems like the go builder could just output the json files by default. I'm wondering if the driver even needs to run bazel. I get annoyed when I have to wait for the editor's bazel commands to finish so I can use bazel from the command line, and I'm pretty sure I don't have enough memory to have two copies of bazel running. |
I wrote a hacky library that can solve this https://github.com/nikunjy/golink for protobufs and can be extended wherever |
Hey @jmhodges, would you be willing to contribute your work on this so far? @tomlu is interested in continuing this, and your work would be a great starting point. If you open a PR from the |
Is there any work going on to make this a reality? |
Sorry, no, not at the moment. |
I guess Go modules and their |
I'm hoping to have time to work on this next month. |
@ribrdb This is not a small project. Please read this comment. |
There is another, much simpler way: include generated code in the source tree. Then everything Just Works. I'm aware that this is contrary to the Bazel philosophy, but I consider the idea that some code should exist only at build time to be a misguided antipattern. As the aphorism says, code - even generated code - is primarily for humans to read, and only secondarily for computers to execute. And as the various other kinds of tooling breakage shows, there's more to do with source code than simply compile it into a binary. It belongs in the source tree. This does mean changes to rules_go. In particular it means that where rules_go presently generates code, it should validate generated code instead. Builds should fail if the source-tree copy of any generated code is out of date with respect to its upstream source (such as a .proto file). Ideally the same logic that validates generated code at |
@bobg Copying generated code back to the source tree is a fine workaround. For folks that need to interoperate with other build systems like rules_go (and Bazel rules in general) cannot do that though. Output files are always declared in a separate tree. Actions may run on a different machine. Even when they're run locally, they're usually sandboxed to prevent that that kind of thing. Generated files could be copied by a tool executed with |
I'm using |
Ooh, that looks super handy, thanks @duarten. What are the prospects for adding something like that to rules_go? |
Another possibility would be to try and get |
@jayconrod I totally get the premise of Bazel to restrict outputs into a separate tree. But often times, especially with this use case for generated content, I'm struggling to understand what would be the technical concerns for having this restriction. Why actions couldn't declare outputs in the source tree, optionally and maybe only available for people that know what they are doing? I mean, what could go wrong in this case, considering that the action itself is "pure" and reproducible? I'm thinking of this as maybe an additional feature of Bazel, which I'm pretty sure would never be accepted :) |
Polluting the source tree with generated build outputs would go against the design of Bazel, since it is intended to be both reproducible and hermetic. |
@jmillikin-stripe Yeah, I totally get it. That's why I said that I don't see this being accepted by Bazel at all :) But for the purpose of a thought experiment, where reproducibility and hermeticity would break if actions are allowed to output into the source tree? Considering that same inputs always produce same outputs. |
I suspect allowing Bazel to corrupt the source tree would cause problems for targets defined with a If someone wants generated files in the source tree, why use Bazel at all? Wouldn't |
@jmillikin-stripe Agree with your point about globs. That's why I said "for people that know what they are doing" :) Basically, you'd have to take more care when you output in the source tree. As for why use Bazel at all, I think it's because everything else about Bazel is very compelling, and getting the same workflow with Make Is cumbersome/impossible. Would be great if Bazel could allow more flexibility (like Ninja or Make) for users that know what they are doing, or maybe requiring that these actions are only ran locally and with no sandbox. Or even requiring that these actions are only ran manually, not included into |
This is drifting pretty far off-topic. Bazel rules cannot write files into the source tree. rules_go cannot change that. An external tool can copy files into the source tree (or verify they're up to date), but rules_go is not the place for that. |
How about a fuse filesystem overlay which maps generated code into the source tree? This would be compatible with all editor/IDEs, not specific not any particular language, and would maintain the clean separation between source and output. Perhaps https://github.com/linuxerwang/gobazel would be a good starting point? |
Subscribing so I can update https://github.com/plezentek/rules_wire with this solution when it is implemented |
@duarten re
There is https://github.com/bazelbuild/bazel-skylib/blob/master/rules/diff_test.bzl but it doesn't
I don't think there is anything non reproducible or non-hermetic with checking in generated files like generated_file_test That said, most of us here would find it a nuisance to have to accept files like .pb.go. No one on the thread has suggested to use
couldn't you? Seems like the only problem with this proposal is that it's a mess to maintain all the replace directives, but that could probably be tooled around. Maybe I'm missing something. |
@alexeagle I talked about the |
I happened across an example in Bazel itself of using a test target to verify that a generated .go file is up-to-date, as another example of how this is a valid practice. (and maybe the underlying rule might be useful to someone reading this issue) |
When a library is created with
go_proto_library
or some other mechanism that generates Go code (.pb.go files in this case), the generated sources files are not exposed to editors and IDEs. Consequently, code completion and refactoring tools don't work, lowering developer productivity.The Go team is developing a workspace abstraction mechanism that will allow editors and Bazel (and other build systems) to communicate with each other without the need for direct integration. This is the long-term solution to this problem.
This issue will track progress on the workspace abstraction and will track editor awareness of code generated with Bazel.
The text was updated successfully, but these errors were encountered: