x/tools/gopls: The Need For Gopls Extensibility (Proposal) #49018
Gopls is an incredible tool. Go programmers would truly not be as productive without it.
Gopls fully relies on
Therefore, the biggest benefit of gopls is arguably its caching of parsed Go files and their type system results that came from
Once you have a bird's eye view of an Editor's workspace, the amount of features you can implement to make Go programmers productive are quite limitless. Gopls has a lot of features from code navigation, to code generation, and more.
However, it is practically impossible to fit all the features a programmer wants into gopls. The more features gopls adds, the more unmaintainable it becomes, the slower it gets, and the larger its codebase and resulting binary grow.
Therefore, there is a strong argument for gopls to support the minimal feature for it to be useful out of the box.
But that leaves huge swaths of incredible features that people can want from gopls that are still not worthy of putting into the Gopls codebase.
Below, I've listed a number of such use cases that I've personally run into. I imagine there are a lot more.
There are already many popular libraries in the community that leverage
However, all of those packages suffer from the same problem that gopls has already solved: go/packages is too slow.
Projects that use multiple code generation libraries no longer enjoy the same speed typical Go programmers enjoy due to Go's fast compiler.
Anecdotally speaking, one company I interviewed with told me that their build system takes at least 10 seconds to generate all of their code and compile it.
Therefore, there is a strong need for such libraries to leverage the gopls cache without having to implement their features directly in the gopls codebase.
I think the need for Gopls to be extensible is vital to the Go community.
Most tools out there write their code analysis tools in terms of go/packages but I'd love for many of these tools to take advantage of the already cached modules in their gopls session.
There are several ways to accomplish this task (I listed some ideas below) but I am indifferent with the implementation details. Because as a Go programmer, I have run into numerous scenarios where I wish gopls was extensible in any sort of way.
Many projects has repeatable patterns for how they decorate and return errors in their functions. For example, Athens uses the Upspin Error Handling Pattern, CLIs just use
Gopls already has a great auto-complete snippet suggestion that just returns the error such as:
But an extension could be smarter and go beond just returning the error, such as:
Or in the case of http handlers, users can implement their own way of handling an error inside of a HandlerFunc. See this comment where it explicitly says:
This is a perfect use case for extension so that gopls doesn't implement a single correct way, but projects can have their own snippet completions.
Besides error snippet completions, many companies and teams can have templates as starter points for their servers.
For example, Uber has fx, while the New York Times has Gizmo and they follow specific patterns. A company can publish an extension that could create snippets for starting their own framework as well as other LSP features.
Other Use Cases
There are several ways that can make gopls extensible:
I completely understand that this task is nowhere near trivial.
The first thing that strikes me is the fact that the entirety of gopls is under an
Another thing that requires a lot of thinking is how gopls recognizes extensions and communicates with them to share its memory objects (Snapshots, Views, etc). I suggested some ideas above and I hope there are more ideas out there. But it is also not a trivial task to land at the perfect solution. I hope this issue can be a place to discuss those ideas and their tradeoffs.
The text was updated successfully, but these errors were encountered:
Thank you for filing this thoughtful and well-researched proposal! We'd love to support something like this, but this is definitely a large project that we'd have to look into prioritizing. We'll follow up here after some team discussions, and perhaps we can discuss this at the next golang-tools call.
Does it really make sense for a code generation tool to have a dependency on a running gopls instance? It seems like making these tools self contained would be a good thing.
For a long time devs at my company would compile Java programs using intellijs compiler, which hooked into ids state. But when the IDE got into a bad state, it would break your builds.
seems like not a great fit for codegen. Direct ide features make sense to me though
For sure. I'd be happy to accept any solution that would make code generation fast. The reality is,
I have thought about this a lot, both in the context of extensibility and in the context of simplifying gopls's implementation and improving gopls's performance.
The reason gopls has always had memory problems is that it naively loads the entire workspace (files+syntax+types), with the assumption that we need the workspace to be in-memory to satisfy queries like workspace symbols or references. But in fact, those workspace-wide queries can be offloaded to more efficient data structures that produce approximate results, and then we can refine our results with full package information as necessary -- I've already done this for workspace symbols, and the entire symbol data structure fits in ~30mb for Kubernetes, compared with a workspace memory footprint of ~3GB.
Therefore it's possible that we could drop a huge amount of syntax and type information, computing it on-demand as needed (or possibly loading types from export data), without any loss in gopls's functionality. The major barrier to doing this is the fact that our current package APIs inside of gopls assume that package information is already in memory. If we were to try to drop things from memory, we'd need a better API for package information. Such an API would need to be declarative, allowing the caller to specify exactly what information is needed, and for how long. Such an API would be very similar to go/packages, albeit with additional configuration for cache persistence.
So it has been in the back of my mind that an ideal final form of gopls might have an "in-memory go/packages", augmented with metadata for cache persistence. This could enable extensibility, maintainability, and performance. But the devil is in the details, and with the push for generics I haven't had time to dig into whether this would actually be possible.