ui test fails on Mac OSX

[cuonglm]

$ cd ui
$ go1.13.4 test
--- FAIL: TestForm (0.00s)
    --- FAIL: TestForm/Banner (0.00s)
        Error Trace:    form_test.go:42
        Error:      	Not equal:
                    	expected: "Hi!\n\n"
                    	actual  : "\x1b[34mHi!\x1b[0m\n\n"
        Test:       	TestForm/Banner
    --- FAIL: TestForm/Defaults (0.00s)
        Error Trace:    form_test.go:83
        Error:      	Not equal:
                    	expected: "  label a [default a]:   label b [default b]: "
                    	actual  : "  label a \x1b[2;36m[default a]\x1b[0m:   label b \x1b[2;36m[default b]\x1b[0m: "
        Test:       	TestForm/Defaults
    --- FAIL: TestForm/Errors (0.00s)
        Error Trace:    form_test.go:96
        Error:      	Not equal:
                    	expected: "  label: - invalid input, min length is 6\n  label: - invalid input, max length is 10\n  label: "
                    	actual  : "  label: \x1b[31m- invalid input, min length is 6\x1b[0m\n  label: \x1b[31m- invalid input, max length is 10\x1b[0m\n  label: "
        Test:       	TestForm/Errors
--- FAIL: TestSummary (0.00s)
    --- FAIL: TestSummary/SummarizeMetrics (0.00s)
        --- FAIL: TestSummary/SummarizeMetrics/[avg_min_med_max_p(90)_p(95)_p(99.9)] (0.00s)
        Error Trace:    summary_test.go:75
        Error:      	Not equal:
                    	expected: "     █ child\n\n       ✗ check1\n        ↳  33% — ✓ 5 / ✗ 10\n\n   ✓ checks......: 100.00% ✓ 3   ✗ 0  \n   ✗ http_reqs...: 3       3/s\n     my_trend....: avg=15ms min=10ms med=15ms max=20ms p(90)=19ms p(95)=19.5ms p(99.9)=19.99ms\n     vus.........: 1       min=1 max=1\n"
                    	actual  : "     █ child\n\n\x1b[31m       ✗ check1\n\x1b[0m\x1b[31m        ↳  33% — ✓ 5 / ✗ 10\n\x1b[0m\n   \x1b[32m✓\x1b[0m checks\x1b[2m......:\x1b[0m \x1b[36m100.00%\x1b[0m \x1b[36;2m\x1b[36;2m✓ 3\x1b[0m   \x1b[36;2m✗ 0\x1b[0m  \x1b[0m\n   \x1b[31m✗\x1b[0m http_reqs\x1b[2m...:\x1b[0m \x1b[36m3\x1b[0m       \x1b[36;2m3/s\x1b[0m\n   \x1b[m \x1b[0m my_trend\x1b[2m....:\x1b[0m avg=\x1b[36m15ms\x1b[0m min=\x1b[36m10ms\x1b[0m med=\x1b[36m15ms\x1b[0m max=\x1b[36m20ms\x1b[0m p(90)=\x1b[36m19ms\x1b[0m p(95)=\x1b[36m19.5ms\x1b[0m p(99.9)=\x1b[36m19.99ms\x1b[0m\n   \x1b[m \x1b[0m vus\x1b[2m.........:\x1b[0m \x1b[36m1\x1b[0m       \x1b[36;2m\x1b[36;2mmin=1\x1b[0m \x1b[36;2mmax=1\x1b[0m\x1b[0m\n"
        Test:       	TestSummary/SummarizeMetrics/[avg_min_med_max_p(90)_p(95)_p(99.9)]
        --- FAIL: TestSummary/SummarizeMetrics/[count] (0.00s)
        Error Trace:    summary_test.go:75
        Error:      	Not equal:
                    	expected: "     █ child\n\n       ✗ check1\n        ↳  33% — ✓ 5 / ✗ 10\n\n   ✓ checks......: 100.00% ✓ 3   ✗ 0  \n   ✗ http_reqs...: 3       3/s\n     my_trend....: count=3\n     vus.........: 1       min=1 max=1\n"
                    	actual  : "     █ child\n\n\x1b[31m       ✗ check1\n\x1b[0m\x1b[31m        ↳  33% — ✓ 5 / ✗ 10\n\x1b[0m\n   \x1b[32m✓\x1b[0m checks\x1b[2m......:\x1b[0m \x1b[36m100.00%\x1b[0m \x1b[36;2m\x1b[36;2m✓ 3\x1b[0m   \x1b[36;2m✗ 0\x1b[0m  \x1b[0m\n   \x1b[31m✗\x1b[0m http_reqs\x1b[2m...:\x1b[0m \x1b[36m3\x1b[0m       \x1b[36;2m3/s\x1b[0m\n   \x1b[m \x1b[0m my_trend\x1b[2m....:\x1b[0m count=\x1b[36m3\x1b[0m\n   \x1b[m \x1b[0m vus\x1b[2m.........:\x1b[0m \x1b[36m1\x1b[0m       \x1b[36;2m\x1b[36;2mmin=1\x1b[0m \x1b[36;2mmax=1\x1b[0m\x1b[0m\n"
        Test:       	TestSummary/SummarizeMetrics/[count]
        --- FAIL: TestSummary/SummarizeMetrics/[avg_count] (0.00s)
        Error Trace:    summary_test.go:75
        Error:      	Not equal:
                    	expected: "     █ child\n\n       ✗ check1\n        ↳  33% — ✓ 5 / ✗ 10\n\n   ✓ checks......: 100.00% ✓ 3   ✗ 0  \n   ✗ http_reqs...: 3       3/s\n     my_trend....: avg=15ms count=3\n     vus.........: 1       min=1 max=1\n"
                    	actual  : "     █ child\n\n\x1b[31m       ✗ check1\n\x1b[0m\x1b[31m        ↳  33% — ✓ 5 / ✗ 10\n\x1b[0m\n   \x1b[32m✓\x1b[0m checks\x1b[2m......:\x1b[0m \x1b[36m100.00%\x1b[0m \x1b[36;2m\x1b[36;2m✓ 3\x1b[0m   \x1b[36;2m✗ 0\x1b[0m  \x1b[0m\n   \x1b[31m✗\x1b[0m http_reqs\x1b[2m...:\x1b[0m \x1b[36m3\x1b[0m       \x1b[36;2m3/s\x1b[0m\n   \x1b[m \x1b[0m my_trend\x1b[2m....:\x1b[0m avg=\x1b[36m15ms\x1b[0m count=\x1b[36m3\x1b[0m\n   \x1b[m \x1b[0m vus\x1b[2m.........:\x1b[0m \x1b[36m1\x1b[0m       \x1b[36;2m\x1b[36;2mmin=1\x1b[0m \x1b[36;2mmax=1\x1b[0m\x1b[0m\n"
        Test:       	TestSummary/SummarizeMetrics/[avg_count]
FAIL
exit status 1
FAIL	github.com/loadimpact/k6/ui	0.397s

[na--]

Hmm looking at the escape sequences, this is probably related to the way we detect if we're running on a TTY and if we should output with colors... It seems that for some reason the test has differences, and looking at the code, I'm honestly not sure how the Linux tests worked 😄

Anyway, we need to fix the test for now, since we'd like to have cross-platform automated tests with Azure Pipelines/GitHub actions (#959). But actually fixing the underlying issues might be a somewhat big task. I stumbled on some of the issues with the way we handle colored output in #1007 and even added an inline comment: https://github.com/loadimpact/k6/blob/3421078d34e9eb2dce507c2687eeb3981d1afc87/cmd/root.go#L79-L94

So yeah, not sure why anyone though it would be a good idea to output colored text with escape codes and then subsequently strip those codes... But if refactoring the code isn't much more complicated, we might end up doing that and not just patching the test...

[joanlopez]

Hi,

I have executed both tests on MacOS (Sonoma) and Go v1.21.4, multiple times to detect potential flakiness, with:

• go test ./ui/... -run TestForm -count=10
• go test ./ui/... -run TestSummary -count=10

and none of them fail. So, perhaps could we consider this as fixed already?

Summoning another contributor with MacOS (cc/ @oleiade) to double-check it.

Thanks!

[oleiade]

I can confirm the same results on a similar setup 👍🏻 🎉