Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

x/tools/gopls: format on save breaks imports when reordered #35388

Open
zikaeroh opened this issue Nov 6, 2019 · 21 comments
Assignees
Labels
Milestone

Comments

@zikaeroh
Copy link

@zikaeroh zikaeroh commented Nov 6, 2019

What did you do?

In https://github.com/hortbot/hortbot/blob/master/internal/cli/flags/ircflags/ircargs.go, swap around some import lines and save.

My Go-related configs:

    "go.lintOnSave": "workspace",
    "go.lintTool": "golangci-lint",
    "go.formatTool": "goimports",
    "go.useLanguageServer": true,
    "go.alternateTools": {
        "go-langserver": "gopls"
    },
    "go.buildOnSave": "off",
    "go.vetOnSave": "off",
    "[go]": {
        "editor.snippetSuggestions": "none",
        "editor.formatOnSave": true,
        "editor.codeActionsOnSave": {
            "source.organizeImports": true
        },
        "editor.codeActionsOnSaveTimeout": 3000
    },
    "gopls": {
        "completeUnimported": true,
        "watchFileChanges": true,
        "deepCompletion": true,
        "go-diff": true,
        "local": "github.com/hortbot/hortbot"
    },

Though I'd normally configure local in the project's settings, since it doesn't make much sense globally.

What did you expect to see?

The imports are reordered back to the way they were (or re-sorted by the local feature).

What did you see instead?

The imports get formatted very wrong. Imports get deleted, duplicated, and sometimes turn out syntatically correct with misplaced quotes. I tried to do a screencap.

https://streamable.com/ffxjz

I also verified this without go-diff enabled to see if it was the diff algorithm, but there was no difference.

Also, the local option is applied at an unknown time. I want things to behave exactly as they do with VS Code + goimports, i.e. "hit save, imports are reordered and fixed", but can't seem to get that to work.

Build info

golang.org/x/tools/gopls master
    golang.org/x/tools/gopls@v0.1.8-0.20191105231337-689d0f08e67a h1:4tnHOn2ZDC1EOp6PyPlUfrxeuw1IzXmJ6Hh359rRlSI=
    github.com/BurntSushi/toml@v0.3.1 h1:WXkYYl6Yr3qBf1K79EBnL4mak0OimBfB0XUf9Vl28OQ=
    github.com/sergi/go-diff@v1.0.0 h1:Kpca3qRNrduNnOQeazBd0ysaKrUJiIuISHxogkT9RPQ=
    golang.org/x/sync@v0.0.0-20190423024810-112230192c58 h1:8gQV6CLnAEikrhgkHFbMAEhagSSnXWGV915qUMm9mrU=
    golang.org/x/tools@v0.0.0-20191105231337-689d0f08e67a h1:RzzIfXstYPS78k0QViPGpDcTlV+QuYrbxVmsxDHdxTs=
    golang.org/x/xerrors@v0.0.0-20190717185122-a985d3407aa7 h1:9zdDQZ7Thm29KFXgAX/+yaf3eVbP7djjWp/dXAppNCc=
    honnef.co/go/tools@v0.0.1-2019.2.3 h1:3JgtbtFHMiCmsznwGVTUWbgGov+pVqnlf1dEJTNAXeM=

Go info

go version go1.13.4 linux/amd64

GO111MODULE=""
GOARCH="amd64"
GOBIN=""
GOCACHE="/home/jake/.cache/go-build"
GOENV="/home/jake/.config/go/env"
GOEXE=""
GOFLAGS=""
GOHOSTARCH="amd64"
GOHOSTOS="linux"
GONOPROXY=""
GONOSUMDB=""
GOOS="linux"
GOPATH="/home/jake/go"
GOPRIVATE=""
GOPROXY="https://proxy.golang.org,direct"
GOROOT="/usr/lib/go"
GOSUMDB="sum.golang.org"
GOTMPDIR=""
GOTOOLDIR="/usr/lib/go/pkg/tool/linux_amd64"
GCCGO="gccgo"
AR="ar"
CC="gcc"
CXX="g++"
CGO_ENABLED="1"
GOMOD="/home/jake/zikaeroh/hortbot/hortbot/go.mod"
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -m64 -pthread -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build676081404=/tmp/go-build -gno-record-gcc-switches"
@gopherbot gopherbot added this to the Unreleased milestone Nov 6, 2019
@gopherbot gopherbot added the Tools label Nov 6, 2019
@gopherbot

This comment has been minimized.

Copy link

@gopherbot gopherbot commented Nov 6, 2019

Thank you for filing a gopls issue! Please take a look at the Troubleshooting guide, and make sure that you have provided all of the relevant information here.

@stamblerre

This comment has been minimized.

Copy link
Contributor

@stamblerre stamblerre commented Nov 6, 2019

Can you reproduce this by running the gopls imports command? Can you share your gopls logs from when this happens (https://github.com/golang/tools/blob/master/gopls/doc/troubleshooting.md#capturing-logs)?

@zikaeroh

This comment has been minimized.

Copy link
Author

@zikaeroh zikaeroh commented Nov 6, 2019

gopls imports makes no change to my file after I reorder my imports (-d flag shows no output).

Logs here: https://gist.github.com/zikaeroh/7eb4a0a7d7d21d929185315bd8cdcb45

Here is a screenshot of the bad quoting (since I couldn't get it in the screen recording):

image

@stamblerre

This comment has been minimized.

Copy link
Contributor

@stamblerre stamblerre commented Nov 6, 2019

Thanks! I'll take a look at the logs. Can you also share the output of running gopls imports with gopls -rpc.trace -v imports?

@stamblerre

This comment has been minimized.

Copy link
Contributor

@stamblerre stamblerre commented Nov 6, 2019

Interestingly enough, the only time textDocument/codeAction returns a value in your log is this:

[Trace - 20:38:56.202 PM] Received response 'textDocument/codeAction - (10)' in 1445ms.
Result: [{"title":"Organize Imports","kind":"source.organizeImports","edit":{"changes":{"file:///home/jake/zikaeroh/hortbot/hortbot/internal/cli/flags/ircflags/ircargs.go":[{"range":{"start":{"line":10,"character":3},"end":{"line":11,"character":3}},"newText":""},{"range":{"start":{"line":12,"character":53},"end":{"line":12,"character":53}},"newText":"\"\n\t\"github.com/hortbot/hortbot/internal/pkg/ctxlog"}]}}}]

All it's doing here is deleting a line and adding the "github.com/hortbot/hortbot/internal/pkg/ctxlog" important, which doesn't sound like it should cause the results you're seeing. Is it possible that both gopls and some other save hook are applying edits to your file?

@zikaeroh

This comment has been minimized.

Copy link
Author

@zikaeroh zikaeroh commented Nov 6, 2019

Output:

2019/11/05 20:51:19 Info:2019/11/05 20:51:19 Build info
----------
golang.org/x/tools/gopls master
    golang.org/x/tools/gopls@v0.1.8-0.20191105231337-689d0f08e67a h1:4tnHOn2ZDC1EOp6PyPlUfrxeuw1IzXmJ6Hh359rRlSI=
    github.com/BurntSushi/toml@v0.3.1 h1:WXkYYl6Yr3qBf1K79EBnL4mak0OimBfB0XUf9Vl28OQ=
    github.com/sergi/go-diff@v1.0.0 h1:Kpca3qRNrduNnOQeazBd0ysaKrUJiIuISHxogkT9RPQ=
    golang.org/x/sync@v0.0.0-20190423024810-112230192c58 h1:8gQV6CLnAEikrhgkHFbMAEhagSSnXWGV915qUMm9mrU=
    golang.org/x/tools@v0.0.0-20191105231337-689d0f08e67a h1:RzzIfXstYPS78k0QViPGpDcTlV+QuYrbxVmsxDHdxTs=
    golang.org/x/xerrors@v0.0.0-20190717185122-a985d3407aa7 h1:9zdDQZ7Thm29KFXgAX/+yaf3eVbP7djjWp/dXAppNCc=
    honnef.co/go/tools@v0.0.1-2019.2.3 h1:3JgtbtFHMiCmsznwGVTUWbgGov+pVqnlf1dEJTNAXeM=

Go info
-------
go version go1.13.4 linux/amd64

GO111MODULE=""
GOARCH="amd64"
GOBIN=""
GOCACHE="/home/jake/.cache/go-build"
GOENV="/home/jake/.config/go/env"
GOEXE=""
GOFLAGS=""
GOHOSTARCH="amd64"
GOHOSTOS="linux"
GONOPROXY=""
GONOSUMDB=""
GOOS="linux"
GOPATH="/home/jake/go"
GOPRIVATE=""
GOPROXY="https://proxy.golang.org,direct"
GOROOT="/usr/lib/go"
GOSUMDB="sum.golang.org"
GOTMPDIR=""
GOTOOLDIR="/usr/lib/go/pkg/tool/linux_amd64"
GCCGO="gccgo"
AR="ar"
CC="gcc"
CXX="g++"
CGO_ENABLED="1"
GOMOD="/home/jake/zikaeroh/hortbot/hortbot/go.mod"
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -m64 -pthread -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build496587565=/tmp/go-build -gno-record-gcc-switches"
2019/11/05 20:51:20 Info:2019/11/05 20:51:20 go/packages.Load
	packages = 1
2019/11/05 20:51:20 Info:2019/11/05 20:51:20 go/packages.Load
	package = github.com/hortbot/hortbot/internal/cli/flags/ircflags
	files = [/home/jake/zikaeroh/hortbot/hortbot/internal/cli/flags/ircflags/ircargs.go]
// Package ircflags processes IRC-related flags.
package ircflags

import (
	"context"
	"database/sql"
	"time"

	"github.com/hortbot/hortbot/internal/birc"
	"github.com/hortbot/hortbot/internal/db/modelsx"
	"github.com/hortbot/hortbot/internal/db/redis"
	"go.uber.org/zap"
	"github.com/hortbot/hortbot/internal/pkg/apis/twitch"
	"github.com/hortbot/hortbot/internal/pkg/ctxlog"
	"github.com/hortbot/hortbot/internal/pkg/twitchx"
)

type IRC struct {
	Nick         string        `long:"irc-nick" env:"HB_IRC_NICK" description:"IRC nick" required:"true"`
	Pass         string        `long:"irc-pass" env:"HB_IRC_PASS" description:"IRC pass" required:"true"`
	PingInterval time.Duration `long:"irc-ping-interval" env:"HB_IRC_PING_INTERVAL" description:"How often to ping the IRC server"`
	PingDeadline time.Duration `long:"irc-ping-deadline" env:"HB_IRC_PING_DEADLINE" description:"How long to wait for a PONG before disconnecting"`
	Token        bool          `long:"irc-token" env:"HB_IRC_TOKEN" description:"Use a token from the database if available"`

	RateLimitSlow   int           `long:"irc-rate-limit-slow" env:"HB_IRC_RATE_LIMIT_RATE" description:"Message allowed per rate limit period (slow)"`
	RateLimitFast   int           `long:"irc-rate-limit-fast" env:"HB_IRC_RATE_LIMIT_RATE" description:"Message allowed per rate limit period (fast)"`
	RateLimitPeriod time.Duration `long:"irc-rate-limit-period" env:"HB_IRC_RATE_LIMIT_PERIOD" description:"Rate limit period"`
}

var DefaultIRC = IRC{
	PingInterval:    5 * time.Minute,
	PingDeadline:    5 * time.Second,
	RateLimitSlow:   15,
	RateLimitFast:   80,
	RateLimitPeriod: 30 * time.Second,
}

func (args *IRC) Pool(ctx context.Context, db *sql.DB, tw twitch.API) *birc.Pool {
	logger := ctxlog.FromContext(ctx)

	channels, err := modelsx.ListActiveChannels(ctx, db, args.Nick)
	if err != nil {
		logger.Fatal("error listing initial channels", zap.Error(err))
	}

	nick := args.Nick
	pass := args.Pass

	if args.Token {
		token, err := twitchx.FindBotToken(ctx, db, tw, nick)
		if err != nil {
			logger.Fatal("error querying for bot token", zap.Error(err))
		}
		if token != nil {
			logger.Debug("using token from database")
			pass = "oauth:" + token.AccessToken
		}
	}

	pc := birc.PoolConfig{
		Config: birc.Config{
			UserConfig: birc.UserConfig{
				Nick: nick,
				Pass: pass,
			},
			InitialChannels: channels,
			Caps:            []string{birc.TwitchCapCommands, birc.TwitchCapTags},
			PingInterval:    args.PingInterval,
			PingDeadline:    args.PingDeadline,
		},
	}

	return birc.NewPool(pc)
}

func (args *IRC) SendMessageAllowed(ctx context.Context, rdb *redis.DB, origin, target string) (bool, error) {
	return rdb.SendMessageAllowed(ctx, origin, target, args.RateLimitSlow, args.RateLimitFast, args.RateLimitPeriod)
}

I am not aware of any other save hook unless the VS Code extension is still trying to run an external tool even though I have not disabled gopls's formatting. For fun, I replaced my goimports binary with a script that just echo'd a string to try and get it to break or replace the buffer, but the behavior I gave in my original post persists.

@stamblerre

This comment has been minimized.

Copy link
Contributor

@stamblerre stamblerre commented Nov 6, 2019

Thanks for trying that out. I guess we can rule that out then. To confirm with the gopls imports command - this is the original file? Can you see what happens when you remove some imports? If you run it a couple times, does it give you consistent results?

@zikaeroh

This comment has been minimized.

Copy link
Author

@zikaeroh zikaeroh commented Nov 6, 2019

The above tests were done simply with the reorderings. If I only remove an import, I do see it trying to add it back. With gopls imports -d:

$ gopls imports -d ircargs.go                      
--- /home/jake/zikaeroh/hortbot/hortbot/internal/cli/flags/ircflags/ircargs.go.orig
+++ /home/jake/zikaeroh/hortbot/hortbot/internal/cli/flags/ircflags/ircargs.go
@@ -10,7 +10,8 @@
 	"github.com/hortbot/hortbot/internal/db/modelsx"
 	"github.com/hortbot/hortbot/internal/db/redis"
 	"github.com/hortbot/hortbot/internal/pkg/apis/twitch"
-	"github.com/hortbot/hortbot/internal/pkg/twitchx"
+	"github.com/hortbot/hortbot/internal/pkg/ctxlog"
+	"github.com/hortbot/hortbot/internal/pkg/twitchx"
 	"go.uber.org/zap"
 )
@stamblerre

This comment has been minimized.

Copy link
Contributor

@stamblerre stamblerre commented Nov 6, 2019

Can you try using the imports from gopls without the local setting? I would be curious to see if you experience the same behavior without it. Also, if you are able to share a log from when the imports behave strangely that would be helpful, even if it's very long - it's just hard to investigate since your previous log didn't seem to have any incorrect behavior in it.

@zikaeroh

This comment has been minimized.

Copy link
Author

@zikaeroh zikaeroh commented Nov 6, 2019

Hopefully this is what you're looking for: https://gist.github.com/zikaeroh/3300ac58971886fd3f217f9338da2c56

local is unset. The behavior is still there. I made a number of changes, ordering, removed imports, swapped lines, undid, etc. Experienced bad import replacements, quotes getting misplaced, as well as the imports even appearing outside of the import block.

@zikaeroh

This comment has been minimized.

Copy link
Author

@zikaeroh zikaeroh commented Nov 6, 2019

Just to try and eliminate the Go extension as the source of the problem, I configured it to run the formatting tool with a bad flag (-THISISBAD). The bad changes still happen, which tell me at least that the Go extension isn't running an external tool (since it'd otherwise give me an error, or at least do nothing). I have nothing else installed that could do any formatting.

@stamblerre

This comment has been minimized.

Copy link
Contributor

@stamblerre stamblerre commented Nov 6, 2019

Yes, that log was helpful - thank you! One thing I noticed about it is that there were a number of textDocument/codeAction responses that seem to be duplicated. If you look at the responses here, you will see the 3 code action responses that are all the same. I wonder if these diffs are all being applied simultaneously and then colliding. I would expect that this could be fixed via #35243.

@zikaeroh

This comment has been minimized.

Copy link
Author

@zikaeroh zikaeroh commented Nov 6, 2019

Is there any plan to not try and use both format and code actions separately in order to manage imports, instead just doing everything as goimports does as a single action?

With how formatting and import management have been combined via goimports/goreturns in editors for a good while, I would really like that behavior to continue, rather than trying to do both individually (and then have to deal with figuring out which edits match which version, and bugs like this).

@stamblerre

This comment has been minimized.

Copy link
Contributor

@stamblerre stamblerre commented Nov 6, 2019

No, the current plan is actually to separate goimports from gofmt as much as possible (see #30843). This way, the source.organizeImports code action will actually do just that, and gofmt will only format a file. Similarly, we would be happy to add the functionality of goreturns, but only as a quickfix, not a format action.

In general, it is much more correct for us to match edits to a specific version of a file - that's really the issue here. If your editor is slow or something gets out of sync with the client and the server - you will immediately get strange behavior if edits are not associated with a specific version of the file.

@zikaeroh

This comment has been minimized.

Copy link
Author

@zikaeroh zikaeroh commented Nov 6, 2019

Absolutely, the versions matter and that should be fixed. My only thought was that it seemed like splitting things is complicating a scenario that a lot of people have been using for a while, even if doing a bunch of stuff on format is different than most every other LSP enabled language (mainly because doing something like "find me the right new imports to make this file sane" is such a cheap operation in Go, like so many other things).

@ianthehat

This comment has been minimized.

Copy link

@ianthehat ianthehat commented Nov 7, 2019

Historically they were combined only because a save hook was the only way to trigger them, and the fact that they are combined is the source of considerable problems.
For instance, goreturns needed to include the functionality of goimports which needed to include the functionality of gofmt, just because people wanted all three, but then people also wanted gofmt -s which was not part of goimports so they end up running multiple save hooks anyway, all of which are redoing the same behaviors for the parts that the authors decided all their users needed, and now there is gofumpt...

By carefully teasing the functionality apart we can allow users to pick and choose, and authors of a new piece of functionality will no longer need to include a random collection of other save hooks they think are important.

Also much of this functionality does not (and should not) need you to save the file to trigger.

@stamblerre stamblerre self-assigned this Nov 7, 2019
@zikaeroh

This comment has been minimized.

Copy link
Author

@zikaeroh zikaeroh commented Nov 7, 2019

That's fair and reasonable, and I understand and agree with it from that point of view. I was going to stick this on #30843, but I can put my thoughts here too.

I'm just thinking towards the day where gopls is "released" and something like VS Code switches from it being an experiment to the main experience. The default behavior for VS Code users at the moment is "run goreturns, or goimports if in module mode". Formatting on save does everything those tools did. If you enable gopls, the format on save behavior now does a different thing, and you find out from the gopls docs that you have to enable language-scoped settings like codeActionsOnSave to get the old behavior back. That might be jarring. New users who start with gopls may not even know that this feature exists (as in the past where people "discovered" goimports until it was just the default in many editors).

At the end of the day, if I can hit save and things are formatted and reorganized as they would be if I ran goimports (with the bugs fixed 😉), then I'll be happy. More fixes and options showing up I can work with would be nice, too. I just don't want to end up regressing something I've been happy with for many years. 😄

@ianthehat

This comment has been minimized.

Copy link

@ianthehat ianthehat commented Nov 7, 2019

Yes, the intention is that the default configuration will be what we believe to be the best normal user experience, which will include those kinds of things. The configuration options will still be there for power users, but the defaults will be sane.

@zikaeroh

This comment has been minimized.

Copy link
Author

@zikaeroh zikaeroh commented Nov 7, 2019

At the risk of extending this longer (when my bug isn't related, per se), how do you plan on setting the sane defaults when it's the client's job to apply things like fixes?

(At least, it's my impression that options like editor.codeActionsOnSave are things you ask the editor to do, and thus not something the server controls.)

@ianthehat

This comment has been minimized.

Copy link

@ianthehat ianthehat commented Nov 7, 2019

Each client needs to decide what the sensible defaults are, we have opinions, and to some extent we hope that clients will respect them so that the unconfigured experience will be reasonably common across all clients, but some things will be different because the model the clients work in is different, and that is okay.
We will probably have the most influence over the VSCode behavior because we directly contribute to that plugin, and that will probably be our exemplar for what we think the user experience should be.

@zikaeroh

This comment has been minimized.

Copy link
Author

@zikaeroh zikaeroh commented Nov 13, 2019

Since #35243 was fixed, I went and retried formatting/import sorting again as above, but the same behavior persists.

https://gist.github.com/zikaeroh/618e23c7292123e62a65dbcb71d2c107

I can see in the log that the code actions now include version numbers, but maybe there's something else that's needed (or #35243 alone wasn't the entire story).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
4 participants
You can’t perform that action at this time.