go/ast: doc insufficient for inserting comments

[willfaught]

I don't see a way to insert documentation comments for file declarations such that they appear just above their decls after printing with go/printer.Fprint or go/format.Node (or both, as explained below). Is it impossible? If so, it should be noted in the doc; if not, it should be noted how to do it, since a straightforward mutation of the comment fields and/or creation/mutation/saving of a CommentMap for File.Comments doesn't work. Also note the related problems pointed out in the code below.

My scenario is writing a tool that inserts // TODO doc comments for undocumented exported decls.

var fs = token.NewFileSet()
var path = ... // path to .go file with exported decls with no doc
var af, _ = parser.ParseFile(fs, path, nil, parser.DeclarationErrors|parser.ParseComments)
var cm = ast.NewCommentMap(fs, af, af.Comments)

if cm == nil {
	cm = ast.CommentMap{} // problem: doc doesn't say cm can be nil! bug?
}

for _, d := range af.Decls {
	switch d := d.(type) {
	case *ast.FuncDecl:
		d.Doc = ... // insert doc

	case *ast.GenDecl:
		for _, s := range d.Specs {
			switch s := s.(type) {
			case *ast.TypeSpec:
				s.Doc = ... // insert doc

			case *ast.ValueSpec:
				s.Doc = ... // insert doc
			}
		}
	}
}

af.Comments = cm.Comments()

var b bytes.Buffer

format.Node(&b, fs, af) // problem: doesn't format new func doc comments! bug?

// func doc comments appear above their funcs correctly but replace blank line that was there before

bs, _ := format.Source(b.Bytes()) // format again for func doc comments

// var/const decl doc comments are put at the top of the file in one bunch

[griesemer]

@willfaught It's possible but it's complicated. In fact it's a major reason for an eventual rewrite of the AST (a more modern and simplified version is already in use in cmd/compile/internal/syntax).

Not sure this helps a lot for the moment but there are two mechanisms in place:

1. "Doc string" comments are attached to nodes directly via the respective Node's Doc fields.
2. All comments (incl. Doc strings) are collected in a sorted (in source order) list of comments found in the File node. When source code is printed, comments are interspersed with the rest of the code based on position information.

Thus, if you are inserting comments, you need to insert them in the right place in the general comment list in the ast.File.Comments list, and you need to give them the correct position relative to the nodes where they are inserted. Unfortunately that's a bit of a trial and error process.

(go/ast is one of the very earliest Go packages in existence, and the design decision made for handling comments - while it works for gofmt - is unfortunate when it comes to manipulating comments. Mea culpa.)

[griesemer]

Related: #14629

[pwaller]

My use-case is automatically generating an AST from scratch, so there are no existing positions to work with.

I found this useful StackOverflow comment which demonstrates putting comments in an existing ast (which looks rather complicated!), but no straightforward way to satisfy my need. I guess I would have to either figure out how to synthesise positions for all statements in the *token.FileSet, or serialise the AST without comments, parse it again, and then subsequently take the approach in the above linked StackOverflow post.

[griesemer]

@pwaller A token.Pos is basically an offset from the beginning of the file (+ some base). A comment comes before some other token, if its pos (its offset) is less than the pos of that token. In the simplest form, you could just give tokens increasing positions (increased by say 10 each time), and provide positions between the tokens for whatever comment you may have. That should comments in the right places between tokens (but maybe not on the right lines yet). For the right lines, you also need to register lines with token.File.AddLine (or in one fell swoop with SetLines). This should get you off the ground.

Again, there's no easy way at the moment to make this work.

[gopherbot]

CL https://golang.org/cl/46370 mentions this issue.