testscript: remove our code coverage mechanism thanks to Go 1.20

[mvdan]

(see commit message)

Fixes #130, as the RemoveAll call is no longer present.
Fixes #161, as the API is now deprecated.
Fixes #199, as "go test -coverprofile" now works on Go 1.20.

[mvdan]

FYI @FiloSottile @prymitive @bitfield

[mvdan]

I also think this means that our func() int signature for top-level commands is no longer necessary; we could let commands call os.Exit directly, or test func main directly, as we no longer need to run code after each command to write a coverage profile.

[rogpeppe]

Brilliant stuff, thanks! So glad that these hacks can go.

One thing: I wonder if we should remove the flag.Parse invocation inside exe.go.
We do add the testWork flag, but go test will itself call flag.Parse, and we don't need the value of the flag until the tests have actually started running (unlike before where we needed the coverage-related flags earlier).

[mvdan]

You're absolutely right - I had spotted the flag.Parse and thought I removed it, but clearly not!

[bitfield]

Does this mean the next release of testscript will not work with earlier Go versions than 1.20?

I also think this means that our func() int signature for top-level commands is no longer necessary; we could let commands call os.Exit directly, or test func main directly, as we no longer need to run code after each command to write a coverage profile.

How would we specify that in the commands map?

func TestMain(m *testing.M) {
    os.Exit(testscript.RunMain(m, map[string]func() {
        "hello": main.main, // 🤷
    }))
}

[mvdan]

Does this mean the next release of testscript will not work with earlier Go versions than 1.20?

It still works with 1.18 and 1.19 (CI is green, after all), but the "deep" code coverage will only work correctly on 1.20.

How would we specify that in the commands map?

I guess you're right - using func main directly only works with internal test packages, where you can reference main directly, not with external test packages like package main_test.

[prymitive]

Thanks, with this commit testscript now works in my tests and no coverage was lost.

I do see occasional errors like this:

error: coverage meta-data emit failed: writing /var/folders/81/9y735kxj4mq2z84d7c0p5_sw0000gn/T/go-build3855161740/b001/gocoverdir/covmeta.5537ddddae94ce05e27c70f2972f4eae: rename from /var/folders/81/9y735kxj4mq2z84d7c0p5_sw0000gn/T/go-build3855161740/b001/gocoverdir/tmp.covmeta.5537ddddae94ce05e27c70f2972f4eae1676025253284676000 failed: rename /var/folders/81/9y735kxj4mq2z84d7c0p5_sw0000gn/T/go-build3855161740/b001/gocoverdir/tmp.covmeta.5537ddddae94ce05e27c70f2972f4eae1676025253284676000 /var/folders/81/9y735kxj4mq2z84d7c0p5_sw0000gn/T/go-build3855161740/b001/gocoverdir/covmeta.5537ddddae94ce05e27c70f2972f4eae: no such file or directory

But it's hard to tell if this is a bad interaction between Go & testscript, or an issue with Go 1.20 itself.

[mvdan]

If you keep seeing that, please file a separate issue. I haven't seen that one myself, yet.

[bitfield]

the "deep" code coverage will only work correctly on 1.20.

By this do you mean that the results of go test -coverprofile... will no longer show the coverage for statements executed by the binary?

That would be a shame, since this is one of the things that usually gets people excited about testscript. Could we keep support for it for the traditional two minor versions? (i.e. until Go 1.22 is released).

[mvdan]

I describe the reasoning behind the decision in the commit message, have you seen that?

[bitfield]

Thanks for pointing me to the commit message—and, as a side note, if that message had been included in the PR description, I wouldn't have needed to ask the questions!

If I'm interpreting it correctly, then your answers to my two questions are, effectively:

Do you mean that the results of go test -coverprofile... will no longer show the coverage for statements executed by the binary? [using the latest testscript, but a Go version less than 1.20]

Yes.

Could we keep support for it for the traditional two minor versions? (i.e. until Go 1.22 is released).

Yes, but we're not going to.

Understood, thank you—the reason I ask is that I anticipate many people (readers of my tutorials and book, for example) will write programs using testscript, but compiled with Go 1.18 and 1.19, and as a result it will not behave as I have described.

The advice I should give them, I understand, is to explicitly import an older version of go-internal (for example, 1.9.0), assuming they are not, for various good reasons, able to immediately upgrade to Go 1.20.

[mvdan]

Thanks for pointing me to the commit message—and, as a side note, if that message had been included in the PR description, I wouldn't have needed to ask the questions!

I do include "see the commit message" in the PR body, but it's relatively easy to miss. GitHub's PR UI is just bad with commit messages. I used to also copy commit messages into PR bodies when creating them from the command line, but that was worse, because any updates to commit messages weren't reflected in the PR body. Plus it doesn't really work past one commit.

The advice I should give them, I understand, is to explicitly import an older version of go-internal (for example, 1.9.0), assuming they are not, for various good reasons, able to immediately upgrade to Go 1.20.

Personally, I think it's fine to simply recommend the latest stable version of testscript, just like you would recommend the latest stable version of Go. You could make a note that the latest testscript will not have full code coverage on Go versions before 1.19, and suggest a slightly older testscript version for those stuck on 1.19, but I don't think that should matter to the majority of users. Go 1.20 has been out for ten days now, and we haven't done a testcript release yet.

[ldemailly]

As a user I will second that the main reason I adopted testscript is to get coverage on CLIs

Consider making the go.mod say 1.20 min for 1.10.1 and retracting 1.10.0 because right now it does stuff like this:

fortio/dnsping#35 (comment)

ps: pretty sure I'm not the only one who doesn't jump and adopt the next version of go right away, there are always a ton of issue with linters, tooling etc.. as well as bug so typically 1.20.3 would be the first version I'd switch away from 1.19.7 - in other words, it's being "stuck" with 1.19, it's being safe and not wanting to be on bleeding edge

ps2: not to say this MR isn't great, red diffs and using built in support is fantastic, just that the incompatibility with go 1.19 should be highlighted and I believe avoided/clarified for dependabot style updates using 1.20 in go.mod

[mvdan]

I don't really agree with specifying Go 1.20 as our minimum version. testscript works OK with Go 1.19, it's just that full test coverage (which was always an extra feature, and was a bit wonky as it caused flakes like #130) now only works with 1.20 or later.

I understand that some users may still be on Go 1.19 as their main version, but they can also remain on the previous release of go-internal. Note that 1.20 came out nearly two months ago at this point; we are not rushing anything in my opinion. If someone is eagerly updating dependency versions, I would hope that they are eagerly updating their Go version as well; otherwise that's not consistent :)

As I described in my previous comments, I wouldn't minded keeping support with 1.19 for another release or two if it were easy, but the old code was quite complex and buggy, so I leaned against it.

[ldemailly]

Getting bug fixes and using dependabot to get dependency isn't inconsistent with staying 1 behind bleeding edge go version

Ok, maybe a compromise is editing https://github.com/rogpeppe/go-internal/releases/tag/v1.10.0 to highlight/spell out you will lose coverage if using go < 1.20 ?

[mvdan]

I'm curious, why do you call Go stable "bleeding edge"? Go does have pre-releases and development versions (master); 1.20 is not bleeding edge by any means.

[ldemailly]

.0 is never "stable" a lot of the tooling doesn't work despite prereleases etc and best effort issues are always discovered during wider adoption.

https://go.dev/doc/devel/release#go1.20.minor