testing: normalize line endings in example output comparisons

[benhoyt]

If you have code which outputs \r\n (CR LF) as a newline delimiter on Windows, testable examples using this code will fail. I noticed this on my GoAWK project, which translates \n to \r\n on Windows to be a good Windows line-ending citizen on that platform. I'm currently using a !windows build constraint to skip this on Windows so these testable examples don't fail (source).

In addition, it's not obvious from the failure message what's going on, because even though it prints the got/want output, it looks correct (CR LF doesn't look any different from just LF when printed):

--- FAIL: Example (0.00s)
got:
foo
bar
want:
foo
bar
FAIL

Test output and source. My stripped-down example to show what's happening is as follows (though of course in GoAWK it's a bit more complex and hidden):

func Example() {
	if runtime.GOOS == "windows" {
		io.WriteString(os.Stdout, "foo\r\nbar\r\n")
	} else {
		io.WriteString(os.Stdout, "foo\nbar\n")
	}
	// Output:
	// foo
	// bar
}

It seems to me that examples should normalize line endings, and that these tests should succeed on Windows. When you're writing examples you don't want to be bothered by which kind of line ending the code is outputting. (If you care at that level, write a non-example test to ensure that behaviour.)

I'm on Go version 1.17, but this affects all versions going way back.

For reference, here are a few other "testable example whitespace" issues. Most of them are about spaces at the end of lines, so don't directly address this issue:

• cmd/gofmt: gofmt breaks examples with multline comments as output #43548
• testing: examples with trailing whitespace in multiple lines of output can be confusing #26460
• testing: example output whitespace trimming can be confusing #23542
• testing: multiline example output comparison doesn't strip per-line whitespace #6416

In #6416, Rob Pike says:

I would prefer to see the output of the examples use exact matching because that's
easiest to specify and allows one to test exact output. For the rare examples for which
that is problematic, sanitize/regularlize the output before matching. That is, write
code.

This could be taken as applying to this issue as well. I don't love this -- it is "easiest" to specify, but it seems to me not what you want, and definitely not visually obvious (which is what example tests are about). In any case, if that's the way it is, so be it. Maybe we could at least make the failure message more obvious in that case?

And if there are better ways to approach this problem in GoAWK or wherever else, I'm all ears!

[bcmills]

#41980 could help somewhat here, in that the diff could at least (potentially) show the difference in line endings.

[bcmills]

It seems to me that examples should normalize line endings, and that these tests should succeed on Windows. When you're writing examples you don't want to be bothered by which kind of line ending the code is outputting.

I tend to agree, but I think this is a subtle enough change to send through the proposal process. Do you want to go that route, or does #41980 suffice?

[benhoyt]

Thanks @bcmills. #41980 would definitely have helped me debug the issue to begin with (way back when). But I think it'd be even better if these tests could work on Windows, i.e., that the line endings were normalized. Would be great if this can be discussed as a proposal -- even if rejected.

[benhoyt]

I just re-ran this under Go 1.22 and the issue is still there (GitHub Actions failure).

It seems very reasonable to me to have code that outputs \r\n line endings on Windows, and also very reasonable that the testable examples take that into account, that is, normalize line endings (at least on Windows) before comparison.

Can this issue be added to the proposal review?

[rsc]

This seems completely reasonable to me. Thanks for the good use case description.

[rsc]

This proposal has been added to the active column of the proposals project
and will now be reviewed at the weekly proposal review meetings.
— rsc for the proposal review group

[rsc]

Based on the discussion above, this proposal seems like a likely accept.
— rsc for the proposal review group

The proposal is to convert example output \r\n into \n before comparing against the golden output in the source files, but only when running on Windows. This is necessary since source files have the same conversion applied to them, and some programs aim to print \r\n on Windows. On Unix, using \r\n is a mistake so we won’t do the conversion there.

(If we change our mind about Unix we can always change that later, but it seems right not to convert.)

[rsc]

No change in consensus, so accepted. 🎉
This issue now tracks the work of implementing the proposal.
— rsc for the proposal review group

The proposal is to convert example output \r\n into \n before comparing against the golden output in the source files, but only when running on Windows. This is necessary since source files have the same conversion applied to them, and some programs aim to print \r\n on Windows. On Unix, using \r\n is a mistake so we won’t do the conversion there.

(If we change our mind about Unix we can always change that later, but it seems right not to convert.)

[gopherbot]

Change https://go.dev/cl/627035 mentions this issue: testing: replace CRLF by LF on windows before comparing to the expected output

[benhoyt]

Thanks @iwdgo for the fix!