UI for running fuzz tests

[qiulaidongfeng]

gopls version

Build info ---------- golang.org/x/tools/gopls v0.14.2 golang.org/x/tools/gopls@v0.14.2 h1:sIw6vjZiuQ9S7s0auUUkHlWgsCkKZFWDHmrge8LYsnc= github.com/BurntSushi/toml@v1.2.1 h1:9F2/+DoOYIOksmaJFPw1tGFy1eDnIJXg+UHjuD8lTak= github.com/google/go-cmp@v0.5.9 h1:O2Tfq5qg4qc4AmwVlvv0oLiVAGB7enBSJ2x2DqQFi38= github.com/sergi/go-diff@v1.1.0 h1:we8PVUC3FE2uYfodKH/nBHMSetSfHDR6scGdBi+erh0= golang.org/x/exp/typeparams@v0.0.0-20221212164502-fae10dda9338 h1:2O2DON6y3XMJiQRAS1UWU+54aec2uopH3x7MAiqGW6Y= golang.org/x/mod@v0.14.0 h1:dGoOF9QVLYng8IHTm7BAyWqCqSheQ5pYWGhzW00YJr0= golang.org/x/sync@v0.4.0 h1:zxkM55ReGkDlKSM+Fu41A+zmbZuaPVbGMzvvdUPznYQ= golang.org/x/sys@v0.14.0 h1:Vz7Qs629MkJkGyHxUlRHizWJRG2j8fbQKjELVSNhy7Q= golang.org/x/telemetry@v0.0.0-20231114163143-69313e640400 h1:brbkEFfGwNGAEkykUOcryE/JiHUMMJouzE0fWWmz/QU= golang.org/x/text@v0.13.0 h1:ablQoSUd0tRdKxZewP80B+BaqeKJuVhuRxj/dkrun3k= golang.org/x/tools@v0.14.1-0.20231114185516-c9d3e7de13fd h1:Oku7E+OCrXHyst1dG1z10etCTxewCHXNFLRlyMPbh3w= golang.org/x/vuln@v1.0.1 h1:KUas02EjQK5LTuIx1OylBQdKKZ9jeugs+HiqO5HormU= honnef.co/go/tools@v0.4.5 h1:YGD4H+SuIOOqsyoLOpZDWcieM28W47/zRO7f+9V3nvo= mvdan.cc/gofumpt@v0.4.0 h1:JVf4NN1mIpHogBj7ABpgOyZc65/UUOkKQFkoURsz4MM= mvdan.cc/xurls/v2@v2.4.0 h1:tzxjVAj+wSBmDcF6zBB7/myTy3gX9xvi8Tyr28AuQgc= go: devel go1.22-46410b7 Tue Nov 28 05:13:40 2023 +0000

go env

set GO111MODULE=auto
set GOARCH=amd64
set GOBIN=
set GOCACHE=C:\Users\26454\AppData\Local\go-build
set GOENV=C:\Users\26454\AppData\Roaming\go\env
set GOEXE=.exe
set GOEXPERIMENT=
set GOFLAGS=
set GOHOSTARCH=amd64
set GOHOSTOS=windows
set GOINSECURE=
set GOMODCACHE=D:\file\gofile\pkg\mod
set GONOPROXY=
set GONOSUMDB=
set GOOS=windows
set GOPATH=D:\file\gofile
set GOPRIVATE=
set GOPROXY=https://goproxy.cn,direct
set GOROOT=C:\Users\26454\.go\current
set GOSUMDB=sum.golang.org
set GOTMPDIR=
set GOTOOLCHAIN=local
set GOTOOLDIR=C:\Users\26454\.go\current\pkg\tool\windows_amd64
set GOVCS=
set GOVERSION=devel go1.22-b2dbfbfc Fri Jan 12 00:56:20 2024 +0000
set GCCGO=gccgo
set GOAMD64=v3
set AR=ar
set CC=gcc
set CXX=g++
set CGO_ENABLED=1
set GOMOD=
set GOWORK=
set CGO_CFLAGS=-O2 -g
set CGO_CPPFLAGS=
set CGO_CXXFLAGS=-O2 -g
set CGO_FFLAGS=-O2 -g
set CGO_LDFLAGS=-O2 -g
set PKG_CONFIG=pkg-config
set GOGCCFLAGS=-m64 -mthreads -Wl,--no-gc-sections -fmessage-length=0 -ffile-prefix-map=D:\tmp\go-build2134721204=/tmp/go-build -gno-record-gcc-switches

What did you do?

In vscode , open .go file , it have FuzzTest.

What did you see happen?

I see have "run test" on FuzzTest , it can run go test -run ^FuzzTest$

What did you expect to see?

I see have "run fuzz" on FuzzTest , it can run go test -fuzz ^FuzzTest$
I hope such feature requests will enable gopls to better support fuzz test.

Editor and settings

"go.editorContextMenuCommands": {
    "toggleTestFile": false,
    "addTags": false,
    "testFile": true,
    "testPackage": true,
    "generateTestForFile": true,
    "generateTestForPackage": true,
    "playground": false,
    "debugTestAtCursor": false
},

Logs

No response

[mauri870]

cc @golang/tools-team

[hyangah]

Fuzz tests are supposed to keep running for some time until a crasher is found. How long do you want to run the tests, when, where, how do you you want results to be present to you?

[qiulaidongfeng]

I hope to run until it fail or vscode is closed.

…

---Original--- From: "Hyang-Ah Hana ***@***.***> Date: Fri, Jan 12, 2024 21:44 PM To: ***@***.***>; Cc: ***@***.******@***.***>; Subject: Re: [golang/go] x/tools/gopls: support run fuzz test (Issue golang/vscode-go#3152) Fuzz tests are supposed to keep running for some time until a crasher is found. How long do you want to run the tests, when, where, how do you you want results to be present to you? — Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you authored the thread.Message ID: ***@***.***>

[qiulaidongfeng]

Where the result is output.
Any possible way is acceptable within vscode.

[qiulaidongfeng]

Background information that may be useful.
I created this issue because the first time I clicked "run test" on FuzzTest did go test -run ^FuzzTest$ instead of go test -fuzz ^FuzzTest$
When I need to go test -fuzz ^FuzzTest$ (usually when I finish writing new code to test), I can use the "run package tests" provided by gopls every time go test, and fuzzy test can only start in cmd by itself. This led me to note that it would be easier to use fuzz test if gopls provided a feature to run a specific fuzz test.

[findleyr]

the first time I clicked "run test" on FuzzTest did go test -run ^FuzzTest$ instead of go test -fuzz ^FuzzTest$

Oops, that's just a bug! Thanks for reporting. @hyangah this would be a VS Code bug, right? Shall we just roll this into our grand unification of tests?

I personally think it's easier to run fuzz tests from the terminal -- better for browsing output, having terminal history, and cancelling with ctrl-c. But I tend to prefer the terminal more than others. I'm curious to hear if there is support for the idea of running from VS Code.

[hyangah]

It's not a bug but intentional behavior for "run test" codelens.
FuzzTest is also a normal test. When it runs with -run, it runs against the corpus. So the intended workflow is:

• go test -fuzz=FuzzTest and collect a crasher.
• fix bug.
• run go test -run=FuzzTest to verify the fix
• check in crasher dataset, so the FuzzTest can keep serving as a regression detector.

[qiulaidongfeng]

I'm curious to hear if there is support for the idea of running from VS Code.

There is a scenario where run fuzz is the last step before git commit, used to check if modifications introduce new bugs, which is more convenient to run in vscode.
I encountered this scenario not long ago, and all I need to know is whether run fuzz has not failed. At the same time, I also encountered two fuzz tests sharing the same prefix, and I don't think it's easier to run one fuzz on the terminal than pressing a "run fuzz" in VScode.

[hyangah]

At the same time, I also encountered two fuzz tests sharing the same prefix, and I don't think it's easier to run one fuzz on the terminal than pressing a "run fuzz" in VScode.

Just provide the full function name to -fuzz parameter, not prefix.

The test functionality in vscode go is implemented in the vscode side, not in gopls.
LSP doesn't have methods for test execution. Should we move this discussion to vscode?

If it is about vscode, an alternative approach for fuzzing is to utilize the vscode tasks.

[qiulaidongfeng]

At the same time, I also encountered two fuzz tests sharing the same prefix, and I don't think it's easier to run one fuzz on the terminal than pressing a "run fuzz" in VScode.

Just provide the full function name to -fuzz parameter, not prefix.

What I'm trying to say is.
If the fuzz test involves typing dozens of letters on the command line, it's easier to click "run fuzz".

The test functionality in vscode go is implemented in the vscode side, not in gopls.
LSP doesn't have methods for test execution. Should we move this discussion to vscode?

I researched this issue in vscode before opening it
"go.useLanguageServer": false
The "run test" will be gone
"go.useLanguageServer": true
"run test" will be available

[hyangah]

I researched this issue in vscode before opening it
"go.useLanguageServer": false
The "run test" will be gone
"go.useLanguageServer": true
"run test" will be available

Vscode go extension uses gopls to extract test, benchmark, fuzz function names, that are needed to implement the code lens provider. Thus, we disabled the test code lens if the language server is disabled. Other than that the implementation is done by the extension, not gopls.

The code is in https://github.com/golang/vscode-go/blob/c5607ecbf6305f208ae2786834c16240662dc53c/extension/src/goRunTestCodelens.ts,

vscode-go/extension/src/testUtils.ts

Line 51 in c5607ec

const fuzzFuncRegx = /^Fuzz$|^Fuzz\P{Ll}.*/u;

, etc.

Gopls can help detect fuzz test, but proxying to run fuzz testing seems to be beyond what can be easily offered by LSP spec. We currently have no plan for it.

I think, it's possible that we can consider vscode-go specific UX.

[findleyr]

Moving this to the VS Code issue tracker. I don't think this type of workflow is appropriate to run from the language server -- better to invoke the test client side, if desired.