Skip to content

proposal: x/tools/go/analysis/passes/shadow: deprecate it #75342

Open

Description

@adonovan

Background: The shadow analyzer aims to report mistakes related to shadowing of local variables, which is a not infrequent source of mistakes in Go. However, the shadow analyzer has at no point in its history gotten remotely close to the signal-to-noise ratio we expect of our static analysis tools, and that's why it was never enabled by default in the old cmd/vet, nor even included in the vet suite once vet was modularized using the Go analysis framework. Yet we still receive occasional CLs trying to improve the algorithm, and they take time to review.

If anyone wishes to develop their own shadow analyzer, either based on our code or from whole cloth, they are free to do so and we wish them luck; and if they succeed at eliminating false positives (for example, demonstrating a <5% false positive rate on the goalng.org/x repos), we could discuss replacing the current implementation and perhaps even adding it to vet. Until then:

Proposal: We propose to formally deprecate the shadow analyzer on the grounds that it is a failed experiment, and to summarily reject CLs to incrementally improve its algorithm.

@findleyr

Activity

  1. moved this to Incoming in Proposalson Sep 9, 2025
  2. added
    ToolsThis label describes issues relating to any tools in the x/tools repository.
    on Sep 9, 2025
  3. added this to the Unreleased milestone on Sep 9, 2025
  4. changed the title [-]x/tools/go/analysis/passes/shadow: deprecate it[/-] [+]Proposal: x/tools/go/analysis/passes/shadow: deprecate it[/+] on Sep 9, 2025
  5. removed
    ToolsThis label describes issues relating to any tools in the x/tools repository.
    on Sep 9, 2025
  6. findleyr commented on Sep 9, 2025

    @findleyr
    Contributor

    I guess this qualifies as a Proposal, since we typically treat deprecations as API changes (albeit reversible ones).

    Fully agree. We have somewhat different standards for gopls, where hint level diagnostics are less prominent, and even there it is disabled by default.

  7. added
    ToolProposalIssues describing a requested change to a Go tool or command-line program.
    on Sep 9, 2025
  8. mateusz834 commented on Sep 10, 2025

    @mateusz834
    Member

    Does that also mean removal from gopls? I have this analyzer enabled all the time in gopls. I don’t really use it in the sense of renaming variables based on its results, but rather as a hint, so I can spot such cases more easily while debugging. I’m fine with all the false positives.
    The description of the analyzer mentions that "check for possible unintended shadowing of variables", which i have been not aware of. I thought that it reports all shadowed variables.
    Perhaps an inlay hint (similar to the recently added ignoredError) would have been more helpful in such cases.

  9. changed the title [-]Proposal: x/tools/go/analysis/passes/shadow: deprecate it[/-] [+]proposal: x/tools/go/analysis/passes/shadow: deprecate it[/+] on Sep 10, 2025
  10. adonovan commented on Sep 10, 2025

    @adonovan
    MemberAuthor

    Does that also mean removal from gopls?

    That was my intent, yes; I should have been more explicit.

    I have this analyzer enabled all the time in gopls. I don’t really use it in the sense of renaming variables based on its results, but rather as a hint, so I can spot such cases more easily while debugging. I’m fine with all the false positives.

    Huh, interesting. And it's not even configured at level protocol.SeverityHint as it should have been, so presumably it pollutes the VS Code problems pane with non-problems?

  11. mateusz834 commented on Sep 10, 2025

    @mateusz834
    Member

    so presumably it pollutes the VS Code problems pane with non-problems?

    No idea how it is displayed in VS Code, in neovim i see it being displayed as warnings.

    main.go|6 col 3-4 warning| declaration of "a" shadows declaration at line 4
    
  12. mateusz834 commented on Sep 10, 2025

    @mateusz834
    Member

    I have this analyzer enabled all the time in gopls. I don’t really use it in the sense of renaming variables based on its results, but rather as a hint, so I can spot such cases more easily while debugging. I’m fine with all the false positives.

    Idea to make that use-case better: #75368

  13. findleyr commented on Sep 10, 2025

    @findleyr
    Contributor

    We should at the very least make shadow hint-level in gopls. IMO that provides the best UX in VS Code (it is, after all, a hint).

  14. gopherbot commented on Sep 10, 2025

    @gopherbot
    Contributor

    Change https://go.dev/cl/702537 mentions this issue: gopls/internal/settings: shadow.severity = hint

  15. seankhliao commented on Sep 10, 2025

    @seankhliao
    Member

    I also have it enabled all the time, it's more distracting when browsing other people's code. For my own code, I'm only annoyed by the warnings for shadowing err.

    also using neovim, with https://git.sr.ht/~whynothugo/lsp_lines.nvim so the warnings take up an event line (motivation to not shadow variables in my own code).

  16. jakebailey commented on Sep 10, 2025

    @jakebailey
    Contributor

    If anyone wishes to develop their own shadow analyzer, either based on our code or from whole cloth, they are free to do so and we wish them luck; and if they succeed at eliminating false positives (for example, demonstrating a <5% false positive rate on the goalng.org/x repos), we could discuss replacing the current implementation and perhaps even adding it to vet.

    I do have the one I wrote for typescript-go (explainer here) that I was hoping to upstream or similar, but I have not run it on a larger corpus. In our codebase, it found a lot of bugs and very few false positives.

    I don't know if it meets the bar, though; how do I know if it's meeting the 5% rate?

  17. adonovan commented on Sep 10, 2025

    @adonovan
    MemberAuthor

    I don't know if it meets the bar, though; how do I know if it's meeting the 5% rate?

    Run it on a random large repo and inspect a random 20 of the diagnostics. If only about one is false, you're doing well.

  18. jakebailey commented on Sep 10, 2025

    @jakebailey
    Contributor

    Thanks, I get what you mean now. Mine is probably still on the noisy side. Maybe a couple of potential bugs, but the rest are just code that could be unclear (that I would probably have used new names for).

  19. bkreitch commented on Sep 26, 2025

    @bkreitch

    I have an updated version based on the usage logic discussed in the other PR. Shadowing is now reported when an inner variable is assigned but not used in the inner scope, while the outer variable is used afterwards.

    I've run the analyzer on a number of repositories. So far, unless I'm missing something, there are only 2 reports, and these appear to be actually problematic cases:

    https://github.com/golang/tools:
    internal/gcimporter/gcimporter.go:83:6: declaration of "err" shadows declaration at line 47

    https://github.com/prometheus/prometheus:
    cmd/promtool/rules.go:118:18: declaration of "err" shadows declaration at line 101

    And it passes simple synthetic tests.
    It still needs some optimizations and additions, but would the general functionality be acceptable?

  20. adonovan commented on Sep 26, 2025

    @adonovan
    MemberAuthor

    https://github.com/golang/tools:
    internal/gcimporter/gcimporter.go:83:6: declaration of "err" shadows declaration at line 47

    FWIW, this one is a false positive:

    func Import(fset *token.FileSet, packages map[string]*types.Package, path, srcDir string, lookup func(path string) (io.ReadCloser, error)) (pkg *types.Package, err error) {
    ...	if lookup != nil {
    ...	} else {
    		var filename string
    		filename, id, err = FindPkg(path, srcDir)
    		if filename == "" {
    ...			return nil, err
    		}
    
    ...		f, err := os.Open(filename)
    		if err != nil {
    			return nil, err
    		}
    		defer func() {
    			if err != nil {
    				// add file name to error
    				err = fmt.Errorf("%s: %v", filename, err)
    			}
    		}()
    		rc = f
    	}
    ...

    I'll admit that the code is a little tricky, but I don't see any bugs here.

    https://github.com/prometheus/prometheus:
    cmd/promtool/rules.go:118:18: declaration of "err" shadows declaration at line 101

    This one also doesn't seem problematic to me:

    func (importer *ruleImporter) importRule(ctx context.Context, ruleExpr, ruleName string, ruleLabels labels.Labels, start, end time.Time,
    	maxBlockDuration int64, grp *rules.Group,
    ) (err error) {
    ...
    	for startOfBlock := blockDuration * (startInMs / blockDuration); startOfBlock <= endInMs; startOfBlock += blockDuration {
    ...		val, warnings, err := importer.apiClient.QueryRange(ctx,...)
    		if err != nil {
    			return fmt.Errorf("query range: %w", err)
    		}

    It's true that there are two err variables, the inner shadowing the outer, but the return statement is essentially a copy from the inner to the outer. This pattern is pretty common; it's not a bug.

  21. bkreitch commented on Sep 26, 2025

    @bkreitch

    https://github.com/golang/tools:
    internal/gcimporter/gcimporter.go:83:6: declaration of "err" shadows declaration at line 47

    I'll admit that the code is a little tricky, but I don't see any bugs here.

    I think when defer func() will run it will take shadowing err (which is nil - otherwise it would return at 85) instead of top level err from line 98 or 105. So successfully open filename won't be prepended to the error.

    https://github.com/prometheus/prometheus:
    cmd/promtool/rules.go:118:18: declaration of "err" shadows declaration at line 101

    It's true that there are two err variables, the inner shadowing the outer, but the return statement is essentially a copy from the inner to the outer. This pattern is pretty common; it's not a bug.

    Here I think there is also at line 139 case similar to gcimporter.go where defer will use/update shadowing err.

    		w, err := tsdb.NewBlockWriter(promslog.NewNopLogger(), importer.config.outputDir, 2*blockDuration)
    		if err != nil {
    			return fmt.Errorf("new block writer: %w", err)
    		}
    		var closed bool
    		defer func() {
    			if !closed {
    				err = tsdb_errors.NewMulti(err, w.Close()).Err()
    			}
    		}()
  22. adonovan commented on Sep 26, 2025

    @adonovan
    MemberAuthor

    I think when defer func() will run it will take shadowing err (which is nil - otherwise it would return at 85) instead of top level err from line 98 or 105. So successfully open filename won't be prepended to the error.

    Ah, but this is the magic of defer and named result variables: when the return statement is executed, it augments err (fmt.Errorf(...)) and updates the named result variable. Then the deferred function runs, and it reads the named result variable again, augments the error, and updates the variable again. It is tricky, but I don't think it's incorrect.

  23. bkreitch commented on Sep 26, 2025

    @bkreitch

    I seem to get different results when it's shadowed and when it's not:

    package main
    
    import (
    	"fmt"
    )
    
    func main() {
    	fmt.Println(ShadowedDefer())
    	fmt.Println(NonShadowedDefer())
    }
    
    func ShadowedDefer() (err error) {
    	{
    		err := fmt.Errorf("shadowing error")
    		defer func() {
    			err = fmt.Errorf("wrapped: %v", err)
    		}()
    	}
    	err = fmt.Errorf("final error")
    	return
    }
    
    func NonShadowedDefer() (err error) {
    	{
    		err = fmt.Errorf("non-shadowing error")
    		defer func() {
    			err = fmt.Errorf("wrapped: %v", err)
    		}()
    	}
    	err = fmt.Errorf("final error")
    	return
    }

    Gives:

    final error
    wrapped: final error
    
  24. adonovan commented on Sep 26, 2025

    @adonovan
    MemberAuthor

    Oh, you're absolutely right: in both cases the defer is operating on the inner (shadowing) var, not on the outer name result variable, so, as you say, it will never have any observable effect.

    My apologies for not reading more closely. I think your analyzer has found two very subtle true positives. Well done, this is real progress. I'll run it over the corpus and see what else it turns up.

  25. adonovan commented on Sep 30, 2025

    @adonovan
    MemberAuthor

    I ran it over the corpus and found 4100 matches, of which a random 50 are shown here. Of those, I sampled that first 10, and found that 8 were clear errors; the remaining two were not necessarily correctness issues but appeared in code that was clearly confused. I'll study another batch in more detail, and post the complete results in due course, though unfortunately our pipeline is in the middle of some upgrades at the moment. Promising results.

    I think it might improve precision to ignore shadowing of package-level declarations.

    https://go-mod-viewer.appspot.com/github.com/uvalib/orcid-access-ws@v0.0.0-20250612130209-7d062dbabf9d/orcidaccessws/dao/db-datastore.go#L115: "declaration of ""err"" shadows declaration at line 87"
    https://go-mod-viewer.appspot.com/github.com/nwidger/bat@v0.0.2/httplib/httplib.go#L467: "declaration of ""err"" shadows declaration at line 458"
    https://go-mod-viewer.appspot.com/github.com/msinev/pmc@v0.0.0-20220512222710-eb40bf080315/pkg/uploader/xform.go#L192: "declaration of ""err"" shadows declaration at line 181"
    https://go-mod-viewer.appspot.com/github.com/cwntr/go-defi@v0.0.0-20210629134751-07f9ec2f7e66/client/client.go#L990: "declaration of ""err"" shadows declaration at line 980"
    https://go-mod-viewer.appspot.com/github.com/maikovskiys/l1tasks@v0.0.0-20230927052451-6436d7687dc9/learn/dev12/main.go#L11: "declaration of ""n"" shadows declaration at line 9"
    https://go-mod-viewer.appspot.com/github.com/brioux/go-keycloak@v0.0.0-20240929191119-b54a3a01d90b/keycloak.go#L230: "declaration of ""err"" shadows declaration at line 213"
    https://go-mod-viewer.appspot.com/github.com/andybalholm/giopdf@v0.0.0-20220317170119-aad9a095ad48/font.go#L324: "declaration of ""err"" shadows declaration at line 300"
    https://go-mod-viewer.appspot.com/github.com/nizigama/simple-note@v0.0.0-20210501235539-2cc762e06fe8/services/database/bolt.go#L291: "declaration of ""err"" shadows declaration at line 15"
    https://go-mod-viewer.appspot.com/github.com/informationsea/shellflow@v0.1.3/viewlog.go#L230: "declaration of ""err"" shadows declaration at line 227"
    https://go-mod-viewer.appspot.com/github.com/pr2apisdk/testgo/sdk@v0.0.0-20240626131719-36a9d76518b2/sdk.go#L398: "declaration of ""err"" shadows declaration at line 359"
    https://go-mod-viewer.appspot.com/github.com/10ego/gthp@v0.0.0-20241025155251-e1514fa71fbb/internal/templ/templates/index_templ.go#L33: "declaration of ""ctx"" shadows declaration at line 13"
    https://go-mod-viewer.appspot.com/github.com/!aspie!soft/goutil/v6@v6.0.1/compress.go#L27: "declaration of ""q"" shadows declaration at line 25"
    https://go-mod-viewer.appspot.com/github.com/!youth!camp!proj/douyin@v0.0.0-20220619060819-b7c3e9420930/service/feed.go#L23: "declaration of ""err"" shadows declaration at line 20"
    https://go-mod-viewer.appspot.com/github.com/ngsalvo/datastar-rick-and-morty@v0.0.0-20250318143428-01c52f047cfc/web/pages/home_templ.go#L35: "declaration of ""ctx"" shadows declaration at line 15"
    https://go-mod-viewer.appspot.com/github.com/ars0915/glossika-exercise@v0.0.0-20240904180321-a5617a00d892/pkg/db/db.go#L31: "declaration of ""err"" shadows declaration at line 14"
    https://go-mod-viewer.appspot.com/github.com/zartbot/goflow@v0.0.0-20221001080520-456f7dfad288/service/optiontemplatemap/option_template_viptela_tloc.go#L174: "declaration of ""err"" shadows declaration at line 129"
    https://go-mod-viewer.appspot.com/github.com/germla/auth@v0.0.0-20230505173936-bc9a4159cf17/providers/password/confirm.go#L82: "declaration of ""err"" shadows declaration at line 74"
    https://go-mod-viewer.appspot.com/github.com/systematiccaos/gorm@v1.22.6/schema/relationship.go#L417: "declaration of ""primaryFields"" shadows declaration at line 358"
    https://go-mod-viewer.appspot.com/github.com/jasoet/gopki@v1.15.0/signing/signer.go#L347: "declaration of ""err"" shadows declaration at line 343"
    https://go-mod-viewer.appspot.com/github.com/alexander-fernandez/go-group@v0.0.0-20220317235734-2d15c54e6f5f/UtmStack-Setup-Linux.go#L775: "declaration of ""err"" shadows declaration at line 381"
    https://go-mod-viewer.appspot.com/bitbucket.org/lygo/lygo_ext_dbbolt@v0.1.3/lygo_db_bolt_collection.go#L307: "declaration of ""err"" shadows declaration at line 299"
    https://go-mod-viewer.appspot.com/github.com/brioux/go-keycloak@v0.0.0-20240929191119-b54a3a01d90b/keycloak.go#L220: "declaration of ""err"" shadows declaration at line 213"
    https://go-mod-viewer.appspot.com/github.com/tanqiangyes/govalidator@v0.0.5/converter.go#L46: "declaration of ""err"" shadows declaration at line 28"
    https://go-mod-viewer.appspot.com/github.com/miguelangel-nubla/oauth2commander@v0.3.7/oauth2server/command.go#L49: "declaration of ""err"" shadows declaration at line 47"
    https://go-mod-viewer.appspot.com/github.com/cwntr/go-defi@v0.0.0-20210629134751-07f9ec2f7e66/client/client.go#L648: "declaration of ""err"" shadows declaration at line 634"
    https://go-mod-viewer.appspot.com/github.com/korsakjakub/snk@v0.0.0-20230625212658-1d6c119c67ee/main.go#L27: "declaration of ""game"" shadows declaration at line 18 in game.go"
    https://go-mod-viewer.appspot.com/github.com/topi314/pogo-icons@v0.0.0-20250403082451-64aa0bf0322a/pogoicons/commands.go#L213: "declaration of ""err"" shadows declaration at line 168"
    https://go-mod-viewer.appspot.com/github.com/litongjava/hfile@v0.0.0-20250829082833-a538817521ac/main.go#L98: "declaration of ""repoDir"" shadows declaration at line 78"
    https://go-mod-viewer.appspot.com/github.com/cwntr/go-defi@v0.0.0-20210629134751-07f9ec2f7e66/client/client.go#L1003: "declaration of ""err"" shadows declaration at line 980"
    https://go-mod-viewer.appspot.com/github.com/qinghon/debinterface@v0.0.6/Interfaces.go#L155: "declaration of ""err"" shadows declaration at line 113"
    https://go-mod-viewer.appspot.com/github.com/cwntr/go-defi@v0.0.0-20210629134751-07f9ec2f7e66/client/client.go#L781: "declaration of ""err"" shadows declaration at line 764"
    https://go-mod-viewer.appspot.com/github.com/agrigoryan/aoc_2023_go@v0.0.0-20231216221323-4ace361ec685/day5/d5p1.go#L37: "declaration of ""mapping"" shadows declaration at line 19"
    https://go-mod-viewer.appspot.com/github.com/timoth-y/kicksware-api/order-service@v0.0.0-20201002192818-87b546a7ae5a/api/rest/handler.go#L57: "declaration of ""err"" shadows declaration at line 53"
    https://go-mod-viewer.appspot.com/github.com/cwntr/go-defi@v0.0.0-20210629134751-07f9ec2f7e66/client/client.go#L774: "declaration of ""err"" shadows declaration at line 764"
    https://go-mod-viewer.appspot.com/github.com/predbdotovh/pre-api@v0.0.0-20211230223408-096f265fde6a/handlers.go#L85: "declaration of ""err"" shadows declaration at line 72"
    https://go-mod-viewer.appspot.com/github.com/dallinallred/goftp@v0.0.0-20240626003405-61936bc079f1/server/ftpServer.go#L70: "declaration of ""err"" shadows declaration at line 47"
    https://go-mod-viewer.appspot.com/github.com/meowstercatel/discordgo@v0.0.0-20250615204139-29b43d676735/state.go#L1072: "declaration of ""err"" shadows declaration at line 954"
    https://go-mod-viewer.appspot.com/github.com/!chronostasys/calc@v0.0.1/compiler/ast/function.go#L786: "declaration of ""i"" shadows declaration at line 869 in ast.go"
    https://go-mod-viewer.appspot.com/github.com/!jack-!z!l/go_rookie@v1.0.40/rpc/tcp.go#L332: "declaration of ""err"" shadows declaration at line 319"
    https://go-mod-viewer.appspot.com/github.com/hashicorp/golang-math-big@v0.0.0-20180316142257-561262b71329/big/prime.go#L289: "declaration of ""t2"" shadows declaration at line 252"
    https://go-mod-viewer.appspot.com/github.com/msinev/pmc@v0.0.0-20220512222710-eb40bf080315/pkg/uploader/result.go#L188: "declaration of ""err"" shadows declaration at line 177"
    https://go-mod-viewer.appspot.com/github.com/omnigres/cli@v0.1.4/cmd/run.go#L37: "declaration of ""err"" shadows declaration at line 26"
    https://go-mod-viewer.appspot.com/github.com/aerth/userinfo@v0.0.0-20160917202326-6c837af3a47b/userinfo.go#L64: "declaration of ""err"" shadows declaration at line 29"
    https://go-mod-viewer.appspot.com/github.com/krzko/gosh@v0.0.0-20241128233202-5abd4ef06633/internal/shell/builtins/ls.go#L124: "declaration of ""err"" shadows declaration at line 112"
    https://go-mod-viewer.appspot.com/github.com/gorocks/apikit@v0.0.0-20170808011444-396ab031afbd/apikit.go#L119: "declaration of ""err"" shadows declaration at line 108"
    https://go-mod-viewer.appspot.com/github.com/googollee/clic@v0.1.5/source/clone_test.go#L60: "declaration of ""a3"" shadows declaration at line 10 in source_fields_test.go"
    https://go-mod-viewer.appspot.com/github.com/tacerus/ldifdiff@v0.0.0-20181030102753-4dccbe38183b/input_test.go#L24: "declaration of ""entries"" shadows declaration at line 26 in types.go"
    https://go-mod-viewer.appspot.com/github.com/nesv/gothub@v0.0.0-20140406181320-f0c824044d5f/gothub.go#L258: "declaration of ""err"" shadows declaration at line 247"
    https://go-mod-viewer.appspot.com/github.com/googollee/clic@v0.1.5/source/clone_test.go#L60: "declaration of ""a2"" shadows declaration at line 10 in source_fields_test.go"
    https://go-mod-viewer.appspot.com/github.com/junaozun/protoc-gen-go-clone@v0.0.0-20231229025521-89199567f720/generator/ext.go#L39: "declaration of ""err"" shadows declaration at line 35"

  26. bkreitch commented on Oct 1, 2025

    @bkreitch

    Great, thanks for the update. I think checking that the shadowed identifier is a variable could help illuminate issues like shadowing of "type mapping".

    Also, handling naked returns as outer variable usage could be added for cases like this:

    func NamedShadowedReturn() (err error) {
    	{
    		err := fmt.Errorf("inner error") // shadowing
    		_ = err
    		err = fmt.Errorf("new inner error")
    	}
    	return // naked return of outer err
    }
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    ProposalToolProposalIssues describing a requested change to a Go tool or command-line program.

    Type

    No type

    Projects

    • Status
      Incoming

    Milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions