Add initial coverage support to neotest #3462

Slava0135 · 2024-05-24T08:51:56Z

~~Requires #3460~~

Problem

Neotest doesn't provide coverage collection support when running contract tests

Solution

Executed opcodes collected by using OnExecHook for each script.
Then, using DebugSeqPoints, coverage for source file is evaluated and saved in file after each test.

Current implementation has some quirks with it:

File is written during each test cleanup, which can be heavy / slow down test execution significantly
In order to save coverage, coverProfile flag is reset, so go tooling doesn't overwrite coverage file. This can be a problem when running both contract and non-contract tests in the same project.
Scripts from dependencies are added to coverage file too, which is overkill.
Only set mode is supported.

Slava0135 · 2024-05-25T19:26:46Z

Showcase (using https://github.com/nspcc-dev/neofs-contract)

coverage3.mp4

It seems like some coverage data is lost when running all package tests at once, need more testing...

Slava0135 · 2024-05-25T20:34:55Z

Forgot to add reporting if script was already compiled. It does fix problem in the test file on video, but for some reason in other files almost all coverage data is lost.

Slava0135 · 2024-05-25T20:47:14Z

Fixed the issue.

Also the question is, where do we need add the coverage instrumentation? Currently, its only in TestInvoke() method, but there are other methods where VM is run.

pkg/vm/vm.go

pkg/neotest/coverage.go

fyfyrchik · 2024-06-06T12:40:23Z

pkg/neotest/coverage.go

+
+func coverageHook() vm.OnExecHook {
+	return func(sh scriptHash, offset int, opcode opcode.Opcode) {
+		if cov, ok := rawCoverage[sh]; ok {


Unprotected access to globals prevents us from executing tests in parallel. What about having a mutex (or sync.Map)?

I think we just need to use normal Mutex, because there is no atomic "load and update" operation on sync.Map. If multiple goroutines read the map at the same time then append will be called from different goroutines too and only 1 value will be added likely.

Worth to update the implementation, it's a frequent path.

fyfyrchik · 2024-06-06T12:44:12Z

pkg/neotest/coverage.go

+		return true
+	}
+	const coverProfileFlag = "test.coverprofile"
+	flag.VisitAll(func(f *flag.Flag) {


The function name looks like it doesn't have any side-effects. What about doing this flag.VisitAll in init(). This way it will run once and set enabled variable.

Flags are defined and parsed when main is run - after init blocks, so you can't use init() here.
Maybe we should just change name to something more clear.

fyfyrchik · 2024-06-06T12:44:58Z

pkg/neotest/coverage.go

+	if enabled {
+		// this is needed so go cover tool doesn't overwrite
+		// the file with our coverage when all tests are done
+		flag.Set(coverProfileFlag, "")


This is quite a hack and it could violate some go test internal invariants, in theory.
Could you describe other solutions you tried?

In theory yes, but in practice as of Go 1.20 coverProfile is used before all tests are run (see image) and after all tests were run (to write coverage)

The other solution is writing go test wrapper that rewrites coverage report file after it finishes, but this would require more effort from user.

pkg/neotest/coverage.go

fyfyrchik · 2024-06-06T12:48:21Z

pkg/neotest/basic.go

@@ -401,6 +401,10 @@ func TestInvoke(bc *core.Blockchain, tx *transaction.Transaction) (*vm.VM, error
 	ttx := *tx
 	ic, _ := bc.GetTestVM(trigger.Application, &ttx, b)

+	if isCoverageEnabled() {
+		ic.VM.SetOnExecHook(coverageHook())


Speaking of parallel execution: the whole coverage* functions operate on global variables.
It seems we can create some CoverageContext here, provide its method to the VM and then process the results in defer. This way, our only conflict between different goroutines will be write to file.

Or we can add mutex that locks when the file is being written, or vm executes next instruction. The major concern is speed, but currently the bottleneck is actually processing coverage and writing to file (during each test cleanup).

To me the approach with context is slightly better because it's more verbose (or explicit).

This reverts commit 2b1ea89.

Slava0135 · 2024-07-03T10:53:11Z

rebased branch on master

Slava0135 · 2024-07-03T11:09:56Z

It seems to work, but I found out that when running this test, coverage profile for ListContainerSizes will be empty, even though it was clearly run inside the test.

Need to write unit tests for this feature somehow, to make sure it works as expected.

AnnaShaleva

Please, consider reformatting commit structure according to the https://github.com/nspcc-dev/.github/blob/master/git.md#logical-separation.

A really nice feature.

AnnaShaleva · 2024-07-03T15:47:30Z

pkg/neotest/basic.go

@@ -401,6 +401,10 @@ func TestInvoke(bc *core.Blockchain, tx *transaction.Transaction) (*vm.VM, error
 	ttx := *tx
 	ic, _ := bc.GetTestVM(trigger.Application, &ttx, b)

+	if isCoverageEnabled() {
+		ic.VM.SetOnExecHook(coverageHook())


To me the approach with context is slightly better because it's more verbose (or explicit).

AnnaShaleva · 2024-07-03T15:54:34Z

pkg/neotest/compile.go

 }

 // CompileFile compiles a contract from the file and returns its NEF, manifest and hash.
 func CompileFile(t testing.TB, sender util.Uint160, srcPath string, configPath string) *Contract {
 	if c, ok := contracts[srcPath]; ok {
+		collectCoverage(t, rawCoverage[c.Hash].debugInfo, c.Hash)


Have you considered moving collectCoverage to a separate exported function? If we extend neotest.Contract with DebugInfo field, then it will be accessible from neotest.CollectCoverage and users will be able to call it explicitly for those tests where it's needed (after the call to CompileFile, for example).

AnnaShaleva · 2024-07-03T15:59:03Z

pkg/neotest/coverage.go

+func reportCoverage() {
+	f, err := os.Create(coverProfile)
+	if err != nil {
+		panic(fmt.Sprintf("coverage: can't create file '%s' to write coverage report", coverProfile))
+	}


Errors may be handled more graceful if we introduce dependency on *testing.T.

Although by the moment error occurs the *testing.T instance is likely to be finished already.

AnnaShaleva · 2024-07-03T16:05:59Z

pkg/neotest/coverage.go

+func coverageHook() vm.OnExecHook {
+	return func(scriptHash util.Uint160, offset int, opcode opcode.Opcode) {
+		if cov, ok := rawCoverage[scriptHash]; ok {
+			cov.offsetsVisited = append(cov.offsetsVisited, offset)
+		}
+	}
+}


Could you explain why coverageHook is a function? Can't we have a global variable that contains an instance of vm.OnExecHook? It's read-only anyway, no one modifies it.

AnnaShaleva · 2024-07-03T16:19:52Z

pkg/neotest/compile.go

 }

 // CompileFile compiles a contract from the file and returns its NEF, manifest and hash.
 func CompileFile(t testing.TB, sender util.Uint160, srcPath string, configPath string) *Contract {
 	if c, ok := contracts[srcPath]; ok {
+		collectCoverage(t, rawCoverage[c.Hash].debugInfo, c.Hash)


We need a documentation for it, please, consider adding a section to https://github.com/nspcc-dev/neo-go/blob/master/pkg/neotest/doc.go.

AnnaShaleva · 2024-07-03T16:21:22Z

pkg/neotest/coverage.go

+	counts    uint
+}
+
+type documentName = string


Deserves a comment.

AnnaShaleva · 2024-07-03T16:23:54Z

Need to write unit tests for this feature somehow

That would be nice.

codecov · 2024-07-03T16:31:36Z

Codecov Report

Attention: Patch coverage is 0% with 90 lines in your changes missing coverage. Please review.

Project coverage is 73.92%. Comparing base (6f77195) to head (917d4a2).

Files	Patch %	Lines
pkg/neotest/coverage.go	0.00%	76 Missing ⚠️
pkg/neotest/compile.go	0.00%	12 Missing ⚠️
pkg/neotest/basic.go	0.00%	2 Missing ⚠️

Additional details and impacted files

@@             Coverage Diff             @@
##           master    #3462       +/-   ##
===========================================
- Coverage   86.11%   73.92%   -12.19%     
===========================================
  Files         331      332        +1     
  Lines       38564    38653       +89     
===========================================
- Hits        33209    28576     -4633     
- Misses       3824     8508     +4684     
- Partials     1531     1569       +38

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

Slava0135 · 2024-07-10T06:55:07Z

Before I fix anything, we need to agree on some details:

Collect coverage explicitly (context) or implicitly (when compiling), I picked second option because I though people wouldn't bother to enable coverage manually for every test. However, when writing tests for contracts, you have to do quite a lot of setup already.
Enable coverage using external tool (sh + bat / python?) or using go test cleanup + unsetting flag, I picked second option because it requires no effort from user, but this relies on some assumptions about Go test infrastructure that may be changed in future. For first option you would need to put custom PATH for go when creating project. VSCode will probably handle it fine, not sure about GoLand and vim/neovim etc.

AnnaShaleva · 2024-07-10T07:55:32Z

Collect coverage explicitly (context) or implicitly (when compiling)

As I said in review, to me the first option is more preferable because it's explicit, i.e. allows users of neotest to pick those tests that will affect coverage. I think that sometimes there might be a situation when you need to exclude some test from the coverage if this test is designated for some other purpose. For example, our TestCreateBasicChain should not affect the coverage of our test contracts. Also, with this approach an explicit context is passed to the user of neotest so that it's clear that coverage is being collected (or not) for this test. And finally, implicit coverage collection bothers compilation step, so this step is actually not a compilation step anymore. But I'd like to hear @fyfyrchik and @roman-khimov opinion on this topic.

Enable coverage using external tool (sh + bat / python?) or using go test cleanup + unsetting flag

To me the second approach also looks better because cleanups are native for Go, it's easy-to-go way, and I don't feel negative about the unsetting flag "hack".

but this relies on some assumptions about Go test infrastructure that may be changed in future

I think we'll be able to modify coverage module accordingly if something is changed. And for now this approach works fine.

Slava0135 requested review from AnnaShaleva and roman-khimov as code owners May 24, 2024 08:51

Slava0135 marked this pull request as draft May 24, 2024 08:52

fyfyrchik suggested changes Jun 6, 2024

View reviewed changes

Slava0135 mentioned this pull request Jun 14, 2024

Implement OnExecHook VM API #3460

Merged

Slava0135 added 8 commits July 3, 2024 13:40

add coverage support to neotest

7e5e298

report coverage if script was already compiled

1d769ce

Revert "report coverage if script was already compiled"

49737f1

This reverts commit 2b1ea89.

(fixed) report coverage if script was already compiled

ff676d3

fix enabled

2176091

remove scriptHash type

324b150

remove TODO

2049e28

panic if can't create report file

77173f5

Slava0135 force-pushed the add-coverage-support branch from d3472d3 to 77173f5 Compare July 3, 2024 10:51

Slava0135 marked this pull request as ready for review July 3, 2024 10:54

panic if coverprofile flag not found

917d4a2

AnnaShaleva reviewed Jul 3, 2024

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add initial coverage support to neotest #3462

Add initial coverage support to neotest #3462

Slava0135 commented May 24, 2024 •

edited

Loading

Slava0135 commented May 25, 2024 •

edited

Loading

Slava0135 commented May 25, 2024

Slava0135 commented May 25, 2024

fyfyrchik Jun 6, 2024

Slava0135 Jun 7, 2024

AnnaShaleva Jul 3, 2024

fyfyrchik Jun 6, 2024

Slava0135 Jun 7, 2024

fyfyrchik Jun 6, 2024

Slava0135 Jun 7, 2024

fyfyrchik Jun 6, 2024

Slava0135 Jun 7, 2024 •

edited

Loading

AnnaShaleva Jul 3, 2024

Slava0135 commented Jul 3, 2024

Slava0135 commented Jul 3, 2024 •

edited

Loading

AnnaShaleva left a comment

AnnaShaleva Jul 3, 2024

AnnaShaleva Jul 3, 2024

AnnaShaleva Jul 3, 2024

AnnaShaleva Jul 3, 2024

AnnaShaleva Jul 3, 2024

AnnaShaleva Jul 3, 2024

AnnaShaleva Jul 3, 2024

AnnaShaleva commented Jul 3, 2024

codecov bot commented Jul 3, 2024

Slava0135 commented Jul 10, 2024 •

edited

Loading

AnnaShaleva commented Jul 10, 2024 •

edited

Loading

Add initial coverage support to neotest #3462

Are you sure you want to change the base?

Add initial coverage support to neotest #3462

Conversation

Slava0135 commented May 24, 2024 • edited Loading

Problem

Solution

Slava0135 commented May 25, 2024 • edited Loading

Slava0135 commented May 25, 2024

Slava0135 commented May 25, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Slava0135 Jun 7, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Slava0135 commented Jul 3, 2024

Slava0135 commented Jul 3, 2024 • edited Loading

AnnaShaleva left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

AnnaShaleva commented Jul 3, 2024

codecov bot commented Jul 3, 2024

Codecov Report

Slava0135 commented Jul 10, 2024 • edited Loading

AnnaShaleva commented Jul 10, 2024 • edited Loading

Slava0135 commented May 24, 2024 •

edited

Loading

Slava0135 commented May 25, 2024 •

edited

Loading

Slava0135 Jun 7, 2024 •

edited

Loading

Slava0135 commented Jul 3, 2024 •

edited

Loading

Slava0135 commented Jul 10, 2024 •

edited

Loading

AnnaShaleva commented Jul 10, 2024 •

edited

Loading