Skip to content

proposal: go/parser: export parseStmtList to parse Go statement lists without a file wrapper #78565

Open

Description

@gnituy18

Proposal Details

There is no public API to parse a list of statements (the content of a function body). Internally the package has parseStmtList, but it is unexported.

motivation

tmplx is a Go HTML template compiler. In tmplx, you write Go statements directly inside HTML attributes to trigger dynamic behavior on a web page.

At compile time, tmplx needs to parse and validate those inline statements. Since there is no public API for this, it falls back to the wrapper hack:

  fileAst, err := parser.ParseFile(token.NewFileSet(), comp.FilePath,
      fmt.Sprintf("package p\nfunc f() {\n%s\n}", attr.Val), 0)
  if err != nil {
      // ...
  }
  decl := fileAst.Decls[0].(*ast.FuncDecl)

With a public ParseStmtList this becomes:

  stmts, err := parser.ParseStmtList(fset, comp.FilePath, attr.Val, 0)

Activity

  1. added this to the Proposal milestone on Apr 7, 2026
  2. added
    LibraryProposalIssues describing a requested change to the Go standard library or x/ libraries, but not to a tool
    on Apr 7, 2026
  3. mateusz834 commented on Apr 7, 2026

    @mateusz834
    Member

    What problems are you facing with the current workaround?

  4. mateusz834 commented on Apr 7, 2026

    @mateusz834
    Member
  5. adonovan commented on Apr 7, 2026

    @adonovan
    Member

    I'm sympathetic to this proposal, or something like it. Having to add a package p; func _() {...} wrapper around the statements means that all the offsets in the FileSet are incorrect w.r.t. the string of statements.

    I'll note that it's also possible to parse a "statement, expression, or declaration" based on the first token, and that this is useful for all kinds of REPL and debugger tools. The only ambiguity is that func may begin a FuncDecl or a FuncLit. (Resolving in favor of FuncDecl has been preferable in my experience, but it would probably be best to permit the user to control it.)

  6. gnituy18 commented on Apr 8, 2026

    @gnituy18
    Author

    What problems are you facing with the current workaround?

    No problem with this workaround.

    Quick grep shows that even gopls does this that way:

    https://github.com/golang/tools/blob/b36d1d12a1a724eb9be6609c9789aec3d99e6030/gopls/internal/golang/extract.go#L1620-L1640

    Great! I thought I was the only one having this problem. I guess there's a need for parseStmtList to be exported?

  7. mateusz834 commented on Apr 8, 2026

    @mateusz834
    Member

    Having to add a package p; func _() {...} wrapper around the statements means that all the offsets in the FileSet are incorrect w.r.t. the string of statements.

    I know it is not ideal but with AddExistingFiles I bet you can make the experience at least decent. You can create a fake fileset for parsing and after that extract the interesting nodes and create a token.File with your desired positions that you can add to the main/different fset.

    Hmm it might even work without AddExistingFiles.

  8. adonovan commented on Apr 8, 2026

    @adonovan
    Member

    ... I bet you can make the experience at least decent. You can create a fake fileset for parsing and after that extract the interesting nodes and create a token.File with your desired positions that you can add to the main/different fset.

    I don't see how the second sentence relates to the first. ;-)

  9. mvdan commented on Apr 8, 2026

    @mvdan
    Member

    I wrote a variant of this for a similar purpose many years ago, which I called parseDetectingNode, because it would attempt to parse almost any node type that seemed useful: a whole file, a single declaration, a block, a value expression, a statement list, a type expression, or a value spec. See the code here: https://github.com/mvdan/gogrep/blob/master/parse.go

    I suppose that doing this in go/parser could be quite inefficient, so perhaps it's not the right approach. But I also think it would be nice if we could cover most or all of the cases above with public API.

  10. adonovan commented on Apr 8, 2026

    @adonovan
    Member

    I suppose that doing this in go/parser could be quite inefficient, so perhaps it's not the right approach. But I also think it would be nice if we could cover most or all of the cases above with public API.

    It's not the efficiency that I worry about; such snippets are usually small and provided interactively (i.e. slowly). My concern is ambiguity: ideally the caller should have the ability to completely control which grammar productions--expr, stmt(s), decl(s), file--are active.

  11. mvdan commented on Apr 8, 2026

    @mvdan
    Member

    Yep agreed. I'm hinting at the fact that exporting parseStmtList is probably not enough; we should ideally also have an API for parsing a declaration, for example.

  12. mateusz834 commented on Apr 8, 2026

    @mateusz834
    Member

    #78565 (comment)

    func main() {
    	fset := token.NewFileSet()
    	stmts, err := parseStmts(fset, []byte("a := 3"))
    	if err != nil {
    		panic(err)
    	}
    	fmt.Printf("fset.Position(stmts[0].Pos()): %v\n", fset.Position(stmts[0].Pos()))
    	fmt.Printf("fset.Position(stmts[len(stmts)-1].End()): %v\n", fset.Position(stmts[len(stmts)-1].End()))
    }
    
    func parseStmts(fset *token.FileSet, src []byte) ([]ast.Stmt, error) {
    	text := "package main\nfunc _() { " + string(src) + " }"
    
    	fset2 := token.NewFileSet()
    	fset2.AddFile("", -1, fset.Base())
    
    	file, err := parser.ParseFile(fset2, "", text, parser.ParseComments|parser.SkipObjectResolution)
    	if err != nil {
    		return nil, err
    	}
    
    	stmts := file.Decls[0].(*ast.FuncDecl).Body.List
    
    	start := stmts[0].Pos()
    	end := stmts[len(stmts)-1].End()
    
    	fset.AddFile("", int(start), int(end-start))
    
    	return stmts, nil
    }

    There might be subtle bugs here, but I think you will get the idea, and fset can't be used concurrently

  13. adonovan commented on Apr 8, 2026

    @adonovan
    Member

    One way to do this would be to provide a single Parse function, similar to ParseFile, that returns an ast.Node; its Mode bits would enable or disable various productions. (We would need some additional Mode bits; and most of the existing bits don't make sense for non-File productions.)

    Another would be to shoehorn this into the existing ParseFile function, again adding some new Mode bits. The result would always be a File, but it might have synthetic nodes enclosing the nodes of interest, e.g. package _; DECLS or package _; func _() { STMTS } or package _; var _ = EXPR, and the caller would need to peel them off.

    I think I like option 2 better.

  14. mateusz834 commented on Apr 8, 2026

    @mateusz834
    Member

    package _; var _ = EXPR

    Note we already have a ParseExpr. https://pkg.go.dev/go/parser#ParseExpr

  15. mvdan commented on Apr 8, 2026

    @mvdan
    Member

    @mateusz834 yes, we are aware of ParseExpr, but it doesn't cover the other cases.

  16. mateusz834 commented on Apr 8, 2026

    @mateusz834
    Member

    but it doesn't cover the other cases.

    also you don't get access to the comments in that expr. I think this proposal should also consider that case too for stms/decls/exprs.

    And that makes me also wonder what about printing of such parsed Stms/Decls/Exprs? Say you want to parse an Stmt modify it + print. That would imply some kind of wrapper, that is also recognized by go/printer:

    type Stmts struct {
        Stmts []ast.Stmt
        Comments []*ast.CommentGroup
    }

    or with the 2nd approach a way to configure go/printer to skip the ast up to the stmt/decl/expr.

    unless we can solve #20744 before this.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    LibraryProposalIssues describing a requested change to the Go standard library or x/ libraries, but not to a toolProposal

    Type

    No type

    Projects

    No projects

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions