testscript: IgnoreMissedCoverage is unused #161
Comments
|
Looks like I'm to blame here; |
|
In the previous mechanism, each command ran as a sub-process and was expected to produce exactly one coverage profile. This is why the boolean option mattered: if the sub-process failed to produce the coverage file, by default we would error, unless the option was enabled. The new mechanism allows command sub-processes to produce multiple coverage files, for the sake of collecting and merging all of them. This allows us to measure the code coverage of Go tools or programs which execute themselves as children, akin to However, swapping the "must produce 1 file" for "may produce any number of files", I failed to add back the requirement that at least one file should be produced - since each command spawns at least one Go sub-process. When that is fixed, then It's worth noting that, right now, |
Our code was a fairly hacky version of what Go 1.20 does for us, since we had to externally reach into the testing internals to do the right thing before and after each program execution. With Go 1.20, all we actually need to do is ensure that the GOCOVERDIR environment variable is forwarded properly. With that, test binaries will know how to produce multiple coverage profiles, and "go test" will know how to collect and merge them. We could keep our hacky workaround for the sake of "deep" coverage information for Go 1.19, but that doesn't seem worthwhile. The old mechanism caused test flakes like rogpeppe#130, is incompatible with Go 1.20, and is overall worse than what Go 1.20 can do. If a user wants code coverage with the latest testscript version, they can use Go 1.20. If they are stuck on Go 1.19 and need code coverage, I imagine they can also stick to a slightly older testscript version. On a main package, the old testscript with Go 1.19 reports: PASS coverage: 8.0% of statements total coverage: 90.1% of statements ok mvdan.cc/sh/v3/cmd/shfmt 0.063s The new testscript with Go 1.20 reports: PASS mvdan.cc/sh/v3/cmd/shfmt coverage: 90.1% of statements ok mvdan.cc/sh/v3/cmd/shfmt 0.047s Fixes rogpeppe#130, as the RemoveAll call is no longer present. Fixes rogpeppe#161, as the API is now deprecated. Fixes rogpeppe#199, as "go test -coverprofile" now works on Go 1.20.
Our code was a fairly hacky version of what Go 1.20 does for us, since we had to externally reach into the testing internals to do the right thing before and after each program execution. With Go 1.20, all we actually need to do is ensure that the GOCOVERDIR environment variable is forwarded properly. With that, test binaries will know how to produce multiple coverage profiles, and "go test" will know how to collect and merge them. We could keep our hacky workaround for the sake of "deep" coverage information for Go 1.19, but that doesn't seem worthwhile. The old mechanism caused test flakes like rogpeppe#130, is incompatible with Go 1.20, and is overall worse than what Go 1.20 can do. If a user wants code coverage with the latest testscript version, they can use Go 1.20. If they are stuck on Go 1.19 and need code coverage, I imagine they can also stick to a slightly older testscript version. On a main package, the old testscript with Go 1.19 reports: PASS coverage: 8.0% of statements total coverage: 90.1% of statements ok mvdan.cc/sh/v3/cmd/shfmt 0.063s The new testscript with Go 1.20 reports: PASS mvdan.cc/sh/v3/cmd/shfmt coverage: 90.1% of statements ok mvdan.cc/sh/v3/cmd/shfmt 0.047s Fixes rogpeppe#130, as the RemoveAll call is no longer present. Fixes rogpeppe#161, as the API is now deprecated. Fixes rogpeppe#199, as "go test -coverprofile" now works on Go 1.20.
Our code was a fairly hacky version of what Go 1.20 does for us, since we had to externally reach into the testing internals to do the right thing before and after each program execution. With Go 1.20, all we actually need to do is ensure that the GOCOVERDIR environment variable is forwarded properly. With that, test binaries will know how to produce multiple coverage profiles, and "go test" will know how to collect and merge them. We could keep our hacky workaround for the sake of "deep" coverage information for Go 1.19, but that doesn't seem worthwhile. The old mechanism caused test flakes like rogpeppe#130, is incompatible with Go 1.20, and is overall worse than what Go 1.20 can do. If a user wants code coverage with the latest testscript version, they can use Go 1.20. If they are stuck on Go 1.19 and need code coverage, I imagine they can also stick to a slightly older testscript version. On a main package, the old testscript with Go 1.19 reports: PASS coverage: 8.0% of statements total coverage: 90.1% of statements ok mvdan.cc/sh/v3/cmd/shfmt 0.063s The new testscript with Go 1.20 reports: PASS mvdan.cc/sh/v3/cmd/shfmt coverage: 90.1% of statements ok mvdan.cc/sh/v3/cmd/shfmt 0.047s Fixes rogpeppe#130, as the RemoveAll call is no longer present. Fixes rogpeppe#161, as the API is now deprecated. Fixes rogpeppe#199, as "go test -coverprofile" now works on Go 1.20.
It doesn't look like
Params.IgnoreMissedCoverageandIgnoreMissedCoverage()are wired up to anything, and indeed code paths that end withos.Exitare missing from the coverage report without an error being raised.The text was updated successfully, but these errors were encountered: