Gocode needs a rewrite

[nsf]

That's just a fact. Given circumstances (Go 1.6 gets new binary format for packages), maybe it's a good time to start it now.

For those who are interested, the main problem with gocode is the way it was written. I was young and stupid (well, not that young, but stupid for sure) and I did quite a lot of mistakes in the core of the architecture. The worst idea was to use Go's AST structures from "go/ast" to describe the semantic information. Yes, Go is such a simple language that you can actually describe most of the semantics via use of AST structures alone. And so that's what gocode does. I believe in a lot of places it can be simplified if you add custom semantically augmented nodes into the equation. In particular anonymous types should become much simpler. Believe it or not, right now gocode rewrites all AST nodes which represent anonymous types with identifier nodes containing an invalid Go name such as $a_1231 and I add artificial types to the global scope with those names. Like, things are really weird in there.

And of course there is a list of long-standing feature requests. The most frequent one I hear is "parse the damn source code instead of packages". Let me remind you that gocode was written in the age of Makefiles, there was no GOPATH at that moment and the only thing we knew is that the compiled packages are located in GOROOT. But things have changed and everything moved on and I was lazy to implement a package parser for source code. Btw, did I tell you that the result of package parsing is also described in a form of "go/ast" nodes? :D The fact that it all works is a freaking miracle.

Anyways, I'm strongly considering this idea. Sadly, I don't have as much time as I had when I was writing initial version of the gocode. But gocode itself isn't that big. I mean most of the semantic analysis fits into 1-2K lines of code. Also as I said I was young and stupid and wrote quite a bit of shitty code. Recent experiments of mine with functional programming languages (F# in particular) straightened my brain quite a bit. So, I believe I can do it better.

It's unclear if I am willing to start this journey again or not. I do use Go and gocode actively myself these days. So, who knows. But I wanted to put this note here.

And at the end of this note let me try to give an idea of what kind of features need to be addressed:

1. Gocode needs better internal structures for storing semantic info. Whether it's permanent or temporary or whatever. Having proper structures should make it easier to cache things and to extend things. For example if I want to augment this semantical data with abstract "origin location" (for jump to feature) or documentation or something else. New data should allow that.
2. Allow using source files as imported packages. Users need that feature, they want gocode to work magically, that's the only way.
3. Better caching. We can even watch the file system for changes and try to update the cache on file changes. Caching is a tricky problem. It's very well possible that it's only worth caching files by themselves or maybe their AST trees or whatever parsed data representation is there. And redo full semantic analysis on each request. Seriously, cache invalidation is very hard. Gocode on each request does partial semantic analysis, it only looks into things it is interested in. "Hey, you use this variable X, it's defined here and its type is inferred from this method call. Method call is on this receiver Y and it's defined there.", Etc. We can store those individual items with lazy links between them and when the time comes just walk them. That's how gocode tries to do it today. But at the moment of its creation I didn't know what I was doing.
4. Much more featured cursor context evaluation. By cursor context I mean the things that are around the cursor. Do we write an argument to a function? Do we write a conditional expression of an if statement? Do we write a type in a type assertion expression? Do we write a type? Do we convert something to an int/string (there is a limited amount of types which can be converted). Do we write struct fields initialization? Do we write an import statement? Do we write Nth argument of a printf function with known (constant) format string? There is a lot of info that can be inferred about the surroundings of the cursor and it can be used as a hint for completion suggestions However, this area is super tricky. Almost always you work with incomplete unparsable code and you have to guess.
5. Along with caching source data, gocode should cache most recent results and provide support to query additional data about them, such as documentation, definition location. Sadly I don't think gocode will provide a list of usage locations and things alike which require full semantic analysis of all the code. Tools like variable renamers or other kind of refactoring should rely on code being completely correct. Gocode's area of interest is incomplete code, let's do just that one thing.

It's not the end of the list. That's just a few things to think about.

[dominikh]

There's also (experimental, unfinished) completion support for the oracle in https://go-review.googlesource.com/#/c/10318/ – maybe you want to draw inspiration from it. (Just putting it out there, not advocating it)

[nsf]

I know there are semantic tools out there and it's a good idea to look into them. But at the same time I know exactly how things should work and it's just a matter of implementing it. Thanks for the link.

[vansimke]

I'm trying to figure out if you're asking for help, venting, or signaling the start of "gocode 2.0". If you're looking to drum up some interest, you might want to think about sharing your thoughts over here: https://forum.golangbridge.org/. The community is much bigger than when you started this and I think there are a lot of people that would love to lend a hand.

Gocode seems to be one of those things that people just expect to be there and don't really think about the effort that it takes to continue to build out and improve it, especially if the code base is starting to age. I, for one, really appreciate the work that you've been doing and hope that you are able to keep moving things forward. To that end, I'm really a web-dev, not a hard-core parser/lexer guy. I can, however, try to get the word out if you're looking for more backs to carry the load. Just let me know and I'll try to get the word out.

[nsf]

Nah, I was just trying to put some thoughts to see what gocode 2.0 would be like. Somebody posted it on reddit, that's more publicity than I expected. The issue is targeted mostly at people who contributed to gocode in past, maybe they have some thoughts, maybe not. It's not that I really need some input, all the features that would be nice to have in gocode 2.0 are well known. It's just somebody needs to do a rewrite. Maybe you can call it a full scale refactoring. And it will be me, because, who else.

So, I don't know. It's just a note after all, nothing more.

[bruno-medeiros]

There's also (experimental, unfinished) completion support for the oracle in https://go-review.googlesource.com/#/c/10318/ – maybe you want to draw inspiration from it. (Just putting it out there, not advocating it)

I was wondering about this as well - the relationship between gocode and oracle - and was going to mention it too. The projects seem very similar in scope, or at least have lots of commonalities, and I was wondering if it would be more beneficial to try to have more integration, or code re-use, between the two.

In Goclipse we have this funny situation where gocode is used for Code Completion, but oracle is used for Find Definition. In Rust or D, there is only one tool for both semantics operations (respectively, https://github.com/phildawes/racer and https://github.com/Hackerpilot/DCD )

[nsf]

Well, the biggest problem for that to happen is people's opinions. I don't want gocode to be anything but autocompletion/helper tool. I'm okay with adding "jump to definition" eventually. The reason I'm not doing it, because gocode doesn't parse imported packages source code and hence no cross-package jumping is possible at the moment. But when and if gocode does so, "jump to definition" is easy to implement and comes for free. I can even add "jump to type definition". E.g. if you look at variable and its type is infered from some function and this type is a struct defined somewhere in the code (or it's one of the other explicit type declarations, like a map or slice or whatever).

As for oracle, as far as I understand it aims higher. Source code analysis tool and maybe refactoring tool. In the beginning of gocode I actually tried to do everything myself. There were even some swf screencasts of me showing renaming functionality gocode provided. E.g.: http://nosmileface.ru/images/gocode-renaming-demo.swf (the server is hosted on the github, you can also find the file here: https://github.com/nsf/nsf.github.com/tree/master/images). Btw, this screencast may bring nostalgic emotions if you're an old gopher like me.

But the problem with advanced semantic analysis is that it has to be very precise and correct. And it's something I wanted to avoid with gocode due to maintenance cost. With autocompletion I can say: maybe it works, sometimes it fails. With refactoring tools you can't say that, it will ruin people's code. It has to be 100% correct. So, at some point I simply removed all that functionality from gocode and it became an autocompletion tool alone.

As for autocompletion itself, often it requires hacks. For example gocode tries to find out the function you're editing and it strips it out and parses the rest of the go code separately. Well, this feature was done long time ago. Perhaps Go parser can actually recover with AllErrors flag, or maybe it can't, maybe it will destroy too many valuable type/var/func definitions. The point is I can do anything what's necessary with gocode to improve autocompletion results. Living inside of another program may result in having restrictions on what I can do or otherwise it ends up being two programs inside of a single one.

Yeah, I'm not sure about gocode and oracle. Maybe we could negotiate on making a shared in-memory cache of ASTs or something like that. That'd be an interesting experiment. Or maybe a shared library which defines semantic info structures (I think there is one already though). But at the same time gocode doesn't need full AST for imported packages for example. So, meh. I think things will be as they are. Two separate projects. I've never used oracle, but I like features it offers.

[bruno-medeiros]

@nsf To be clear, when I was suggesting more integration, I wasn't thinking about some sort of inter-process communication/integration - that would probably be too complex for little gain.

What I had in mind was more like source code sharing, reuse, submitting patches, etc. For example, like you mentioned, gocode handles incorrect/incomplete Go source code much better than oracle (which usually doesn't handle it at all). If gocode's parsing strategy was submitted to oracle, it would make it handle operations like Open Definition even with incomplete code (and possibly other operations as well).

I dunno how easy that would in practice though. Does the Go tools team (https://github.com/golang/tools/) even take pull requests for oracle?

[nsf]

Yes, I understand what you mean.

[nsf]

Does the Go tools team (https://github.com/golang/tools/) even take pull requests for oracle?

I'm sure they do, as long as you sign CLA and go through their code review process. That's something I definitely don't want to do for any of my hobby projects. So, yeah, oracle is oracle, gocode is gocode.

[bruno-medeiros]

I'm sure they do, as long as you sign CLA and go through their code review process. That's something I definitely don't want to do for any of my hobby projects. So, yeah, oracle is oracle, gocode is gocode.

What about forking oracle then, would that be worthwhile?

[gobijan]

Thx nsf for bringing gocode.
I am really happy with it and use it on a daily basis.

I came to the issues list to see if it's possible to have method description within the autocomplete results. When you do a 2.0 that parses source code this feature probably becomes a low hanging fruit and so I'm looking forward to it.

Maybe you create a donation link for us happy users to give something back to you for your great work on gocode? :)

Keep up the good work! Big thank you from Germany.

[nsf]

I don't accept donations, sorry. Money is not a big deal for me, time is. And you can't really work full time on your hobby projects on donations alone. Unless it's a very popular project, gocode isn't that popular. In other words, I don't mind being paid for working on gocode, but only if it allows me to do it full time, otherwise it's not a very useful thing.

Yes, one of the features that I would like to have is exactly that. Being able to fetch docs after request has been made.

I see it this way:

1. You get autocompletion results from gocode, each line comes with a unique ID.
2. Gocode caches results of a last autocompletion query.
3. And there is additional API for getting additional info about each line, such as documentation and perhaps something else (definition location maybe? I don't know).

[gobijan]

I understand. This is very honorable. Just thought of donations as a way to buy you a drink ;)
This three step approach sounds good.
I am totally looking forward to see something like this in my go code editor of choice (atm sublimetext3) for go: