You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
proposal: go/parser: export parseStmtList to parse Go statement lists without a file wrapper #78565
I'm sympathetic to this proposal, or something like it. Having to add a package p; func _() {...} wrapper around the statements means that all the offsets in the FileSet are incorrect w.r.t. the string of statements.
I'll note that it's also possible to parse a "statement, expression, or declaration" based on the first token, and that this is useful for all kinds of REPL and debugger tools. The only ambiguity is that func may begin a FuncDecl or a FuncLit. (Resolving in favor of FuncDecl has been preferable in my experience, but it would probably be best to permit the user to control it.)
Having to add a package p; func _() {...} wrapper around the statements means that all the offsets in the FileSet are incorrect w.r.t. the string of statements.
I know it is not ideal but with AddExistingFiles I bet you can make the experience at least decent. You can create a fake fileset for parsing and after that extract the interesting nodes and create a token.File with your desired positions that you can add to the main/different fset.
... I bet you can make the experience at least decent. You can create a fake fileset for parsing and after that extract the interesting nodes and create a token.File with your desired positions that you can add to the main/different fset.
I don't see how the second sentence relates to the first. ;-)
I wrote a variant of this for a similar purpose many years ago, which I called parseDetectingNode, because it would attempt to parse almost any node type that seemed useful: a whole file, a single declaration, a block, a value expression, a statement list, a type expression, or a value spec. See the code here: https://github.com/mvdan/gogrep/blob/master/parse.go
I suppose that doing this in go/parser could be quite inefficient, so perhaps it's not the right approach. But I also think it would be nice if we could cover most or all of the cases above with public API.
I suppose that doing this in go/parser could be quite inefficient, so perhaps it's not the right approach. But I also think it would be nice if we could cover most or all of the cases above with public API.
It's not the efficiency that I worry about; such snippets are usually small and provided interactively (i.e. slowly). My concern is ambiguity: ideally the caller should have the ability to completely control which grammar productions--expr, stmt(s), decl(s), file--are active.
Yep agreed. I'm hinting at the fact that exporting parseStmtList is probably not enough; we should ideally also have an API for parsing a declaration, for example.
One way to do this would be to provide a single Parse function, similar to ParseFile, that returns an ast.Node; its Mode bits would enable or disable various productions. (We would need some additional Mode bits; and most of the existing bits don't make sense for non-File productions.)
Another would be to shoehorn this into the existing ParseFile function, again adding some new Mode bits. The result would always be a File, but it might have synthetic nodes enclosing the nodes of interest, e.g. package _; DECLS or package _; func _() { STMTS } or package _; var _ = EXPR, and the caller would need to peel them off.
also you don't get access to the comments in that expr. I think this proposal should also consider that case too for stms/decls/exprs.
And that makes me also wonder what about printing of such parsed Stms/Decls/Exprs? Say you want to parse an Stmt modify it + print. That would imply some kind of wrapper, that is also recognized by go/printer:
Activity
mateusz834 commentedon Apr 7, 2026
What problems are you facing with the current workaround?
mateusz834 commentedon Apr 7, 2026
Quick grep shows that even
goplsdoes this that way:https://github.com/golang/tools/blob/b36d1d12a1a724eb9be6609c9789aec3d99e6030/gopls/internal/golang/extract.go#L1620-L1640
adonovan commentedon Apr 7, 2026
I'm sympathetic to this proposal, or something like it. Having to add a
package p; func _() {...}wrapper around the statements means that all the offsets in the FileSet are incorrect w.r.t. the string of statements.I'll note that it's also possible to parse a "statement, expression, or declaration" based on the first token, and that this is useful for all kinds of REPL and debugger tools. The only ambiguity is that
funcmay begin a FuncDecl or a FuncLit. (Resolving in favor of FuncDecl has been preferable in my experience, but it would probably be best to permit the user to control it.)gnituy18 commentedon Apr 8, 2026
No problem with this workaround.
Great! I thought I was the only one having this problem. I guess there's a need for parseStmtList to be exported?
mateusz834 commentedon Apr 8, 2026
I know it is not ideal but with AddExistingFiles I bet you can make the experience at least decent. You can create a fake fileset for parsing and after that extract the interesting nodes and create a token.File with your desired positions that you can add to the main/different fset.
Hmm it might even work without AddExistingFiles.
adonovan commentedon Apr 8, 2026
I don't see how the second sentence relates to the first. ;-)
mvdan commentedon Apr 8, 2026
I wrote a variant of this for a similar purpose many years ago, which I called
parseDetectingNode, because it would attempt to parse almost any node type that seemed useful: a whole file, a single declaration, a block, a value expression, a statement list, a type expression, or a value spec. See the code here: https://github.com/mvdan/gogrep/blob/master/parse.goI suppose that doing this in
go/parsercould be quite inefficient, so perhaps it's not the right approach. But I also think it would be nice if we could cover most or all of the cases above with public API.adonovan commentedon Apr 8, 2026
It's not the efficiency that I worry about; such snippets are usually small and provided interactively (i.e. slowly). My concern is ambiguity: ideally the caller should have the ability to completely control which grammar productions--expr, stmt(s), decl(s), file--are active.
mvdan commentedon Apr 8, 2026
Yep agreed. I'm hinting at the fact that exporting parseStmtList is probably not enough; we should ideally also have an API for parsing a declaration, for example.
mateusz834 commentedon Apr 8, 2026
#78565 (comment)
There might be subtle bugs here, but I think you will get the idea, and fset can't be used concurrently
adonovan commentedon Apr 8, 2026
One way to do this would be to provide a single
Parsefunction, similar to ParseFile, that returns an ast.Node; its Mode bits would enable or disable various productions. (We would need some additional Mode bits; and most of the existing bits don't make sense for non-File productions.)Another would be to shoehorn this into the existing ParseFile function, again adding some new Mode bits. The result would always be a File, but it might have synthetic nodes enclosing the nodes of interest, e.g.
package _; DECLSorpackage _; func _() { STMTS }orpackage _; var _ = EXPR, and the caller would need to peel them off.I think I like option 2 better.
mateusz834 commentedon Apr 8, 2026
Note we already have a ParseExpr. https://pkg.go.dev/go/parser#ParseExpr
mvdan commentedon Apr 8, 2026
@mateusz834 yes, we are aware of ParseExpr, but it doesn't cover the other cases.
mateusz834 commentedon Apr 8, 2026
also you don't get access to the comments in that expr. I think this proposal should also consider that case too for stms/decls/exprs.
And that makes me also wonder what about printing of such parsed Stms/Decls/Exprs? Say you want to parse an
Stmtmodify it + print. That would imply some kind of wrapper, that is also recognized by go/printer:or with the 2nd approach a way to configure
go/printerto skip the ast up to the stmt/decl/expr.unless we can solve #20744 before this.