Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

x/tools/internal/lsp/regtest: tracking various regtest improvements #39384

Open
findleyr opened this issue Jun 3, 2020 · 7 comments
Open

x/tools/internal/lsp/regtest: tracking various regtest improvements #39384

findleyr opened this issue Jun 3, 2020 · 7 comments
Assignees

Comments

@findleyr
Copy link
Contributor

@findleyr findleyr commented Jun 3, 2020

This is a follow-up to #36879. We've been using the gopls regtests for a few months now, and several usability bugs / missing features / overall themes have emerged. None of these seemed worthy of an issue, but I want to track them in aggregate. I've been trying to let the regtests soak in their current form, but plan to take another pass soon.

Here's a list of what I've noted along the way, either from my own usage or feedback from others'. Feel free to comment with more, and I'll edit this comment to keep it up to date.

  • Once @ianthehat's jsonrpc2 shutdown fixes are merged, we should t.Parallel the regtests once more.
  • Add more documentation about how to develop new regtests (tricks like using -regtest_timeout to avoid waiting for the default regtest timeout).
  • Having the run options after the test body in runner.Run is hard to spot. Use a different options pattern to allow passing them before the test body.
  • Fix the 'NoOutstandingWork' expectation, by making sure that 'progress finished' messages never fail to be sent due to a cancelled context.
  • Somehow fix regtest cleanup on Windows (which runs into problems with file locking from the go command). Perhaps awaiting 'NoOutstandingWork' could help here.
  • Move regtests into the gopls module, so that they can run staticcheck.
  • Once regtests are in the gopls module, replace the synthetic file watching with actual file watching, using fsnotify. The fact that file-watching is faked has been a source of bugs. The file watching should also respect the GlobPattern provided in the file watching capability registration.
  • Make the DiagnosticAtRegexp expectation lazy, so that it doesn't eagerly evaluate the regexp position (and therefore no longer needs the Env receiver).
  • Make DiagnosticAtRegexp match the full range of the regexp match, not just the starting position (this was a cause of some confusion in a CL).
  • Return a copy of State from Await: whatever caused expectations to be be met (the underlying requirement was instead achieved with less effort by adding a 'ReadDiagnostics' expectation that atomically reads from State).
  • Explicitly set GOPACKAGESDRIVER=off for all regtests.
  • Use a top-level TMPDIR for the entire regtest suite, rather than letting each regtest create it's own sandbox in /tmp. This reduces cruft in /tmp when tests are ctrl-c'ed before they can clean themselves up.
  • Add continuous integration to run the regtests against a true daemon, not an in-process daemon. This is not done currently because it will be problematic on Windows, and would increase the test duration.
  • Fix handling of multi-byte runes in source text.

CC @pjweinbgo @stamblerre @heschik

@gopherbot gopherbot added this to the Unreleased milestone Jun 3, 2020
@findleyr findleyr self-assigned this Jun 3, 2020
@stamblerre stamblerre modified the milestones: Unreleased, gopls/v0.5.0 Jun 17, 2020
@stamblerre stamblerre added the Testing label Jun 24, 2020
@gopherbot
Copy link

@gopherbot gopherbot commented Jun 25, 2020

Change https://golang.org/cl/240059 mentions this issue: internal/lsp/regtest: use a common directory regtests sandboxes

@gopherbot
Copy link

@gopherbot gopherbot commented Jun 25, 2020

Change https://golang.org/cl/240058 mentions this issue: internal/lsp/fake: explicitly set GOPACKAGESDRIVER=off in regtests

gopherbot pushed a commit to golang/tools that referenced this issue Jun 25, 2020
In the past, changes to GOPACKAGESDRIVER have led to some confusing
regtest failures. Explicitly set it off.

Updates golang/go#39384

Change-Id: I303a58380a5e46e6621c19b2edc40d43199bb343
Reviewed-on: https://go-review.googlesource.com/c/tools/+/240058
Run-TryBot: Robert Findley <rfindley@google.com>
Reviewed-by: Rebecca Stambler <rstambler@golang.org>
TryBot-Result: Gobot Gobot <gobot@golang.org>
@gopherbot
Copy link

@gopherbot gopherbot commented Jul 9, 2020

Change https://golang.org/cl/241739 mentions this issue: internal/lsp/regtest: refactor to use a different options pattern

gopherbot pushed a commit to golang/tools that referenced this issue Jul 9, 2020
For easier debugging (and less cruft if regtests are ctrl-c'ed), root
all regtest sandboxes in a common directory.

This also tries one last time to clean up the directory, and fails on an
error. This might be flaky on windows, but hasn't been so far...

Also give regtest sandboxes names derived from their test name.

Updates golang/go#39384
Updates golang/go#38490

Change-Id: Iae53c29e75f5eb2b8d938d205fbeb463ffc82eb2
Reviewed-on: https://go-review.googlesource.com/c/tools/+/240059
Run-TryBot: Robert Findley <rfindley@google.com>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Rebecca Stambler <rstambler@golang.org>
@gopherbot
Copy link

@gopherbot gopherbot commented Jul 16, 2020

Change https://golang.org/cl/243057 mentions this issue: internal/lsp/regtest: simpler way to invert options

gopherbot pushed a commit to golang/tools that referenced this issue Jul 16, 2020
This is an alternative to CL 241739, preserving the exiting variadic
options pattern and just adding a helper method to put the options up
front.

All things considered, I think this is better. CL 241739 was too
complicated, and being able to reuse "curried" runners isn't actually
important.

Updates golang/go#39384

Change-Id: I7ecd80d310cac879520b2d1feebcf5bd5e96e89b
Reviewed-on: https://go-review.googlesource.com/c/tools/+/243057
Run-TryBot: Robert Findley <rfindley@google.com>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Heschi Kreinick <heschi@google.com>
@gopherbot
Copy link

@gopherbot gopherbot commented Sep 2, 2020

Change https://golang.org/cl/252683 mentions this issue: integration/regtest: use gopls hooks and add a test for staticcheck

@gopherbot
Copy link

@gopherbot gopherbot commented Sep 2, 2020

Change https://golang.org/cl/252682 mentions this issue: gopls/integration/regtest: move regtests to the gopls module

gopherbot pushed a commit to golang/tools that referenced this issue Sep 3, 2020
Regtests are slow, and make `go test ./internal/lsp/...` slow. Also,
having them in the tools module means they can't use staticcheck,
go-diff, etc.

Move them to the gopls module. This means that they're annoying to work
with unless you open the gopls module, but hopefully that annoyance will
be gone soon when we support multi-module workspaces.

For golang/go#39384

Change-Id: Ib99c994ffdac56d4da13af981ad397a90a7523af
Reviewed-on: https://go-review.googlesource.com/c/tools/+/252682
Run-TryBot: Robert Findley <rfindley@google.com>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Rebecca Stambler <rstambler@golang.org>
gopherbot pushed a commit to golang/tools that referenced this issue Sep 3, 2020
To match the actual gopls binary, use hooks.Options when creating the
regtest server.

Add a test for staticcheck diagnostics to leverage this.

For golang/go#39384

Change-Id: I52837c2b12bb586a2530343bdfae5172b08df49c
Reviewed-on: https://go-review.googlesource.com/c/tools/+/252683
Run-TryBot: Robert Findley <rfindley@google.com>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Rebecca Stambler <rstambler@golang.org>
@gopherbot
Copy link

@gopherbot gopherbot commented Sep 16, 2020

Change https://golang.org/cl/255126 mentions this issue: gopls/internal/regtest: simplify expectation return values

gopherbot pushed a commit to golang/tools that referenced this issue Sep 16, 2020
Returning 'metBy' from expectations was not an ideal API, as extracting
the result value was cumbersome and error prone. It also forced quite a
bit of plumbing.

Using OnceMet, we can improve this by instead using a 'ReadDiagnostics'
expectation that reads diagnostics into a variable.

For golang/go#39384

Change-Id: Ia73e5b496089240df3200626e5696305cb507c9a
Reviewed-on: https://go-review.googlesource.com/c/tools/+/255126
Trust: Robert Findley <rfindley@google.com>
Run-TryBot: Robert Findley <rfindley@google.com>
gopls-CI: kokoro <noreply+kokoro@google.com>
TryBot-Result: Go Bot <gobot@golang.org>
Reviewed-by: Rebecca Stambler <rstambler@golang.org>
@stamblerre stamblerre added this to the gopls/unplanned milestone Oct 21, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

None yet
3 participants
You can’t perform that action at this time.