x/tools/gopls: new code disappears or is modified after formatting

[stamblerre]

A number of VSCode issues have been raised regarding problems with formatting (see microsoft/vscode-go#2824, microsoft/vscode-go#2787). There are a few possibilities about what the source of this issue might be, but it is likely that either, (1) formatting is being run on old code, or (2) something is going wrong with the diff algorithm.

If you have this issue, please share the following information so that we can diagnose it:

1. Your VSCode settings (Ctrl + Shift + P -> Preferences: Open Settings (JSON))
2. Your gopls version (gopls version)
3. The output of gopls -rpc.trace -v format path/to/file.go
4. Your gopls logs from when this issue occurs (see https://github.com/golang/tools/blob/master/gopls/doc/troubleshooting.md#capturing-logs for information on how to capture logs).

Furthermore, we have recently substituted the gopls diff algorithm for a better alternative. It's possible that this was the culprit. If you are able to install gopls at master:

$ GO111MODULE=on go get golang.org/x/tools/gopls@master golang.org/x/tools@master

then you can try out this new diff algorithm. Try it by adding the following setting:

"gopls": {
    "go-diff": true
}

If any users test out go-diff, please do report if this changes anything.

Finally, it will be helpful to determine if the issue is formatting or imports. Please try disabling the following settings one at a time and please report which alters the behavior.

"editor.formatOnSave": true,
"editor.codeActionsOnSave": {
    "source.organizeImports": true,
}

[jpvillaseca]

The issue ocurred with the gopls@(devel), and the normal v0.1.6 version. Also with the v0.1.7 version suggested by vscode-go.

It's worth noting that the type of "code rewrite" is different with the go-diff: true and without it.
Without it, it discards the current changes and repeats the last couple of lines of the code.
With it, it discards the first columns of the current file. For example:
Before saving:

	// insert in the database
	client.generateID()
	client.hub.register <- client

And after saving:

/ insert in the database
	client.generateID()
lient.hub.register <- client

What I have to do to recover from this is to undo the 2 edits made by the formatter, and then execute "reload window".

gopls version:

golang.org/x/tools/gopls v0.1.6
    golang.org/x/tools/gopls@(devel)`

Vscode settings.json:

{
	"workbench.startupEditor": "newUntitledFile",
	"extensions.ignoreRecommendations": true,
	"tslint.jsEnable": true,
	"editor.formatOnPaste": true,
	"editor.formatOnSave": true,
	"editor.codeActionsOnSave": {
		"source.fixAll.tslint": true
	},
	"typescript.updateImportsOnFileMove.enabled": "always",
	"editor.detectIndentation": false,
	"editor.insertSpaces": false,
	"zenMode.hideTabs": false,
	"editor.renderWhitespace": "boundary",
	"workbench.iconTheme": "vscode-icons",
	"bracketPairColorizer.showHorizontalScopeLine": false,
	"bracketPairColorizer.showBracketsInRuler": true,
	"vsicons.dontShowNewVersionMessage": true,
	"explorer.confirmDragAndDrop": false,
	"files.watcherExclude": {
		"dist/**": true
	},
	"files.exclude": {
		"**/dist/**": true
	},
	"search.exclude": {
		"**/dist": true,
		"**/docs": true
	},
	"workbench.colorTheme": "Monokai",
	"terminal.integrated.shell.windows": "C:\\Windows\\System32\\cmd.exe",
	"git.path": "C:/Users/noflo/AppData/Local/Atlassian/SourceTree/git_local/bin/git.exe",
	"editor.mouseWheelScrollSensitivity": 3,
	"go.useLanguageServer": true,
	"gopls": {
		"go-diff": true
	},
}

Output of gopls window:

[Info  - 9:21:14 AM] 2019/10/22 09:21:14 c:\Users\noflo\go\src\jarvis-server\app\session does not have prefix C:\Users\noflo\go\src
[Info  - 9:21:14 AM] 2019/10/22 09:21:14 go/packages.Load
	packages = 1
[Info  - 9:21:14 AM] 2019/10/22 09:21:14 go/packages.Load
	package = builtin
	files = [c:\go\src\builtin\builtin.go]
[Error - 9:21:15 AM] Request textDocument/definition failed.
  Message: no identifier found
  Code: 0 
[Error - 9:22:42 AM] Request textDocument/definition failed.
  Message: no identifier found
  Code: 0 
[Error - 9:23:43 AM] Request textDocument/definition failed.
  Message: no identifier found
  Code: 0 
[Error - 9:25:08 AM] Request textDocument/definition failed.
  Message: no identifier found
  Code: 0 

The output of gopls trace is on this gist:
https://gist.github.com/jpvillaseca/8a63cb8ca3f5e5343a5b28c91b4f423d

EDIT:
I tried setting this, but the formatting error appeared after a while.

"editor.codeActionsOnSave": {
    "source.organizeImports": true,
}

[stamblerre]

@jpvillaseca: Thank you for the detailed report! What is the output of gopls -rpc.trace -v format path/to/file.go?

[stamblerre]

Oh I'm sorry I see you've attached it. I will look into this more thoroughly ASAP.

[stamblerre]

To get more detailed gopls logs, you can add:

 "go.languageServerFlags": [
    "serve",
    "-rpc.trace",
],

to your VSCode settings. Based on the trace you attached, it seems like the file you were trying to format already had syntax errors and wasn't formatted. Can you try formatting a file that doesn't have any syntax errors?

[jpvillaseca]

Thank you, @stamblerre .
Oh, the problem was that the file shouldn't have had syntax errors since they were introduced during saving, and I couldn't repair the file within vscode. To get those traces from gopls in that situation, when the issue resurfaces, I'll have to edit the file in another editor, fix the issues there and save the file, and run gopls trace. Because I can't do that from vscode. Stay tuned 😅

[stamblerre]

Ah I see - makes sense. Thank you for doing this - it will be very helpful to get this information!

[jpvillaseca]

Ok. The verbose trace from within vscode is:
https://gist.github.com/jpvillaseca/8cdc67772db6a40d8f1dfb8f190c49bf

And the output of the gopls trace command on the file without issues:
https://gist.github.com/jpvillaseca/b62aee8c5094d0f150631473e5154a1a

(This is without go-diff: true)

[stamblerre]

Do you mind sharing the same but with go-diff: true? At this point I think we will definitely be migrating to the go-diff algorithm in the next release.

[jpvillaseca]

Of course. Here are the traces with go-diff: true
It was easy to trigger the issue: I set the go-diff: true option, restarted vscode, saved an open file from the workspace (the same from the previous post), to trigger a format document and it crippled the following line of code:
Before saving, this is line 18:

var db *gorm.DB // gorm handles the connection pool using the recommended approach for each database engine

After saving. It ate the comments and some words. I had to do 4 undos to get back to the working version, but it is already crippled on-disk.

var db *gorm.DB  pool using the recommended approach for each database engine

Here is the trace from vscode:
https://gist.github.com/jpvillaseca/9187d9c3f9aebe92e87c9da837b3ca19

And this is the trace from gopls: (I reverted the file to the original version using notepad)
https://gist.github.com/jpvillaseca/45a285c72b15b5de5efb99e567c3cb62

I hope this helps =)

EDIT: Another example with the same file, but now it started replacing the first columns of some lines:
https://gist.github.com/jpvillaseca/b1ad7e13e3a73545244a0eb875ee0638
Such as:

var db *gorm.DB 

// Config stores database connection parameters
var Config DatabaseConfig

Was crippled to:

r db *gorm.DB 

// Config stores databaseonnection parameters
r Config DatabaseConfig