Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

go/parser: add a SkipObjectResolution mode flag #46485

Closed
findleyr opened this issue Jun 1, 2021 · 12 comments
Closed

go/parser: add a SkipObjectResolution mode flag #46485

findleyr opened this issue Jun 1, 2021 · 12 comments

Comments

@findleyr
Copy link
Contributor

findleyr commented Jun 1, 2021

In #45104, go/parser was refactored to make parsing and object resolution separate passes within ParseFile. As a side effect of this change it is trivial to add a mode that skips object resolution entirely. This issue proposes adding such a mode flag: SkipObjectResolution. (note that this mode was initially added as part of #45104, but that is going to be backed-out pending the discussion here).

Background

Object resolution is a feature of go/parser that attempts to resolve identifiers to their declaration, setting ast.Ident.Obj, for use in tools or basic analyzers that need such information. Notably this information is incomplete and in some cases inaccurate: object resolution does not properly handle selectors or fields in composite literals, and often lacks information about declarations in other files or packages. It is also redundant with the information produced by go/types. For this reason object resolution has limited utility, particularly for tools that either deal only with syntax, or which use the output of go/types for identifier information (go/types itself does not need object resolution in order to function).

Pros and Cons

In my testing, skipping object resolution saves 18-20% of the parsing time, which made gofmt around 10% faster (when not rewriting or simplifying, which make use of object resolution). It also avoids unnecessary allocation. It's a small but meaningful optimization for tools that know they won't need ast.Ident.Obj. See #45104 for some further discussion.

For completeness, it's worth considering the downside of adding such a mode. For one, it adds a new mode that must be supported. However, I think this is a negligible cost: it's unlikely that in the future we revert the refactoring of #45104, but even so, skipping object resolution can still be achieved by adding guards in the functions that record or resolve identifiers. It's also worth noting that this mode is not entirely orthogonal to existing modes: ReportDeclarationErrors only works if object resolution is not skipped.

I think the strongest argument against adding this new mode is that it's simply not necessary. However, my current expectation is that it would add a small benefit for some users at a relatively smaller cost.

CC @griesemer @mvdan

@gopherbot
Copy link

Change https://golang.org/cl/323909 mentions this issue: go/parser: un-export the SkipObjectResolution mode pending discussion

@griesemer
Copy link
Contributor

griesemer commented Jun 1, 2021

The object resolution mode is not useful for tools that run a resulting AST through the type checker or which only care about an unadorned syntax tree, e.g. for formatting. This feature is present for historical reasons (the go/parser is one of the first larger go/package that was ever written and still in use).

The fact that not doing object resolution can save up to 20% parse time is significant. Arguably, not doing object resolution should be the default but that would be a non-backward compatible change of behavior. Note also that this mode was implemented and operational well before the Go1.17 freeze.

In short, the suggested new SkipObjectResolution mode provides an easy way to increase performance for many tools simply by setting the flag.

I don't see any good reason for not doing this.

@mvdan
Copy link
Member

mvdan commented Jun 1, 2021

I fully agree with what Robert said. I don't see a reason why this change must be in 1.17, but the code and API have been there for a while, so it's not like we missed the freeze deadline. So I also don't see a reason to push it back to 1.18.

We did not follow the proposal process properly, which perhaps meant some potential objections weren't raised. We can see here if anyone brings any in the next week or two, though I personally doubt it :)

@ianlancetaylor ianlancetaylor added this to Incoming in Proposals (old) Jun 1, 2021
@ianlancetaylor ianlancetaylor changed the title go/parser: add a SkipObjectResolution mode flag proposal: go/parser: add a SkipObjectResolution mode flag Jun 1, 2021
@ianlancetaylor ianlancetaylor modified the milestones: Backlog, Proposal Jun 1, 2021
@dominikh
Copy link
Member

dominikh commented Jun 2, 2021

This also benefits go list -test, which fully parses test files to extract test functions.

For go list -test -json std, we go from 0.61s to 0.53s.

This speedup would automatically trickle down to users of go/packages, without them having to change their code in any way.

@rsc
Copy link
Contributor

rsc commented Jun 2, 2021

This proposal has been added to the active column of the proposals project
and will now be reviewed at the weekly proposal review meetings.
— rsc for the proposal review group

@rsc
Copy link
Contributor

rsc commented Jun 9, 2021

Based on the discussion above, this proposal seems like a likely accept.
— rsc for the proposal review group

@rsc rsc moved this from Active to Likely Accept in Proposals (old) Jun 9, 2021
@findleyr findleyr mentioned this issue Jun 10, 2021
83 tasks
@rsc
Copy link
Contributor

rsc commented Jun 16, 2021

No change in consensus, so accepted. 🎉
This issue now tracks the work of implementing the proposal.
— rsc for the proposal review group

@rsc rsc moved this from Likely Accept to Accepted in Proposals (old) Jun 16, 2021
@rsc rsc changed the title proposal: go/parser: add a SkipObjectResolution mode flag go/parser: add a SkipObjectResolution mode flag Jun 16, 2021
@rsc rsc modified the milestones: Proposal, Backlog Jun 16, 2021
@findleyr findleyr self-assigned this Jun 17, 2021
@findleyr
Copy link
Contributor Author

Closing as the work was done in #45104.

#46298 tracks documenting this new mode for 1.17

@gopherbot
Copy link

Change https://go.dev/cl/401454 mentions this issue: cmd/gofmt: only resolve go/ast objects when needed

@gopherbot
Copy link

Change https://go.dev/cl/401474 mentions this issue: go/format: skip go/ast's object resolution

@gopherbot
Copy link

Change https://go.dev/cl/401455 mentions this issue: all: use go/parser.SkipObjectResolution in more places

gopherbot pushed a commit that referenced this issue Apr 21, 2022
go/parser will by default resolve objects as per the go/ast.Object type,
which is then used by gofmt's rewrite and simplify flags.
However, none of that is needed if neither of the flags is set,
so we can avoid the work entirely for a nice speed-up.

	benchcmd -n 8 GofmtSrcCmd gofmt -l ~/tip/src/cmd

	name         old time/op         new time/op         delta
	GofmtSrcCmd          957ms ± 7%          908ms ± 7%  -5.12%  (p=0.028 n=8+8)

	name         old user-time/op    new user-time/op    delta
	GofmtSrcCmd          11.2s ± 1%          10.4s ± 1%  -7.23%  (p=0.001 n=7+7)

	name         old sys-time/op     new sys-time/op     delta
	GofmtSrcCmd          325ms ±29%          286ms ±22%    ~     (p=0.065 n=8+8)

	name         old peak-RSS-bytes  new peak-RSS-bytes  delta
	GofmtSrcCmd          295MB ±17%          276MB ±15%    ~     (p=0.328 n=8+8)

See #46485.

Change-Id: Iad1ae294953710c233f7837d7eb02e23d11c6185
Reviewed-on: https://go-review.googlesource.com/c/go/+/401454
Run-TryBot: Daniel Martí <mvdan@mvdan.cc>
TryBot-Result: Gopher Robot <gobot@golang.org>
Reviewed-by: Bryan Mills <bcmills@google.com>
Reviewed-by: Ian Lance Taylor <iant@google.com>
gopherbot pushed a commit that referenced this issue Apr 21, 2022
Just like https://golang.org/cl/401454 removed the work from gofmt for a
nice ~5% speed-up in the default case, we can also use the option in the
equivalent go/format for programs which use it rather than gofmt,
as go/format makes no use of objects either.

No benchmark numbers as we already measured the ~5% speed-up with gofmt
in the other CL linked above.

See #46485.

Change-Id: Icbf98e6d46a616081314e2faa13f1dfade3bbaef
Reviewed-on: https://go-review.googlesource.com/c/go/+/401474
Run-TryBot: Daniel Martí <mvdan@mvdan.cc>
TryBot-Result: Gopher Robot <gobot@golang.org>
Reviewed-by: Bryan Mills <bcmills@google.com>
Reviewed-by: Ian Lance Taylor <iant@google.com>
@gopherbot
Copy link

Change https://go.dev/cl/401875 mentions this issue: cmd/gofmt: use SkipObjectResolution with -s as well

gopherbot pushed a commit that referenced this issue May 19, 2022
The "simplify" feature used go/ast's object tracking in only one place -
to replace s[a:len(s)] with s[a:].
Using go/ast.Object did allow us to not simplify code like:

	len := func(s []int) int { ... }
	s = s[a:len(s)]

The existing code already noted the limitation with that approach,
such as "len" being redeclared in a different file in the same package.
Since go/ast's object tracking is file-based and very basic,
it wouldn't work with edge cases like those.

The reasoning is that redeclaring len and abusing it that way is
extremely unlikely, and hasn't been a problem in about a decade now.
I reason that the same applies to len being redeclared in the same file,
so we should be able to safely remove the use of go/ast.Object here.

Per https://go.dev/cl/401454, this makes "gofmt -s" about 5% faster.
If we ever wanted to truly get rid of false positive simplifications,
I imagine we'd want to reimplement the feature under go/analysis,
which is able to fully typecheck packages and suggest edits.
That seems unnecessary at this point, but we can always course correct
in the presumably unlikely scenario that users start reporting bugs.

See #46485.
For #52463.

Change-Id: I77fc97adceafde8f0fe6887ace83ae325bfa7416
Reviewed-on: https://go-review.googlesource.com/c/go/+/401875
TryBot-Result: Gopher Robot <gobot@golang.org>
Run-TryBot: Daniel Martí <mvdan@mvdan.cc>
Reviewed-by: Robert Griesemer <gri@google.com>
Reviewed-by: Michael Knyszek <mknyszek@google.com>
gopherbot pushed a commit that referenced this issue Aug 23, 2022
None of cgo, "go test", nor srcimporter make use of go/ast's object
resolution via go/ast.Object. As such, we can skip that work during
parse time, which should save some CPU time.

We don't have any benchmark numbers, as none of the three packages have
any usable benchmarks, but we measured gofmt to be about 5% faster
thanks to this tweak in https://go.dev/cl/401454.
These three packages are quite different to gofmt, but one can expect
similar speed-ups in the 1-5% range.

Two notable exceptions, which do make use of go/ast.Object, are cmd/fix
and cmd/doc - we do not modify those here.

See #46485.

Change-Id: Ie3e65600d4790641c4e4d6f1c379be477fa02cee
Reviewed-on: https://go-review.googlesource.com/c/go/+/401455
Run-TryBot: Bryan Mills <bcmills@google.com>
Reviewed-by: Bryan Mills <bcmills@google.com>
Reviewed-by: David Chase <drchase@google.com>
TryBot-Result: Gopher Robot <gobot@golang.org>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
No open projects
Development

No branches or pull requests

7 participants