Option to disable log intercepting and/or interactive mode

[kleinfreund]

While working on the colorjs.io package, I frequently run into the problem that logs are inaccessible/hidden and it seemingly relates to interactive mode and/or the log interception.

For example, I'm running npx htest ./test/parse.js (I've installed htest 0.0.17 in colorjs.io locally) and see "Color parse Tests ✅ 129/129 PASS 8 messages (4.9 ms)", but expanding the first tree node only shows one test group (color() in my scenario) even though the parse test suite has many more test groups. I can only assume that there's an exception somewhere that's being intercepted and that somehow breaks the interactive mode, but I can only guess at that because there's virtually no further output.

Disabling the color() test group in an attempt to get to some output worsens the problem. I now see "Color parse Tests ✅ 80/80 PASS (4.4 ms)" but now I can't even expand the first tree node.

I think it would be helpful to be able to turn off interactive mode and just log things as they come (perhaps while still collecting output of the test runner until the test run is complete). This way, one would be able to see logged exceptions even if something in the interactive mode breaks.

Notes & references

• Relates to Janky cli test output #7

[DmitrySharabin]

Hey @kleinfreund,

Is there a chance you could check whether you face the same issue with hTest v0.0.18 (I published it yesterday)? Before, we had a bug where intercepted messages could be mixed up between tests.

In the latest version, we can see groups of tests and individual passed tests on the result screen if only the group has skipped tests or if tests have console messages associated with them. If I run tests locally, I can see the following picture (that seems relevant):

[kleinfreund]

Hey :) Ah, I hadn't seen that. I've tried it now and so far the results seem to be just like I had reported them initially. I verified that node_modules/htest.dev/package.json reports pkg.version as 0.0.18.

[kleinfreund]

One notable observation that I can provide for now is that I can sometimes "fix" the report not containing all tests/test groups by adding logs somewhere. In my current branch, if I add console.log("") just after let { value, meta } = parseArgument(rawArg); in parse.js, suddenly the report is complete again. Removing it, and the report is incomplete. 🤔

[DmitrySharabin]

Is there a chance you could share screenshots of the complete and incomplete reports? I want to make sure I understand the issue correctly.

[kleinfreund]

I've pushed https://github.com/kleinfreund/color.js/tree/fix/parser-not-being-able-to-distinguish-between-legacy-and-modern-formats to my fork of colorjs.io now. Running npx htest ./test on it will work fine; however, removing the console.log in src/parse.js:280 (labeled by a TODO: comment) will break them as described originally.

Here the broken output with some sections expanded.

Without the log

👻 npx htest ./test

Use ↑ and ↓ arrow keys to navigate groups of tests, → and ← to expand and collapse them, respectively.
Use Ctrl+↑ and Ctrl+↓ to go to the first or last child group of the current group.
To expand or collapse the current group and all its subgroups, use Ctrl+→ and Ctrl+←.
Press Ctrl+Shift+→ and Ctrl+Shift+← to expand or collapse all groups, regardless of the current group.
Use any other key to quit interactive mode.

▽ (All tests) ✅ 642/712 PASS 70/712 skipped 18 messages (40 ms)
 ├── ▷ Color conversion Tests ✅ 195/195 PASS 10 messages (10 ms)
 ├── ▷ DeltaE tests ✅ 88/158 PASS 70/158 skipped (4.1 ms)
 └─┬ ▽ Color parse Tests ✅ 129/129 PASS 8 messages (2.4 ms)
   └─┬ ▽ color() ✅ 49/49 PASS 8 messages (1 ms)
     ├─┬ ▼  PASS  "color(display-p3-linear 0 1 .5)" 1 message (0.07 ms)
     │ └─┬ (Messages)
     │   └── (warn) Linear P3 is a non-standard space and not currently supported in the CSS spec. Use prefixed color(--display-p3-linear) instead of color(display-p3-linear).
     ├── ▷  PASS  "color(rec2020-linear 0 1 .5)" 1 message (0.05 ms)
     ├── ▷  PASS  "color(--rec2100-hlg 0 100% 50%)" 1 message (0.05 ms)
     ├── ▷  PASS  "color(--rec2100-pq 0 100% 50%)" 1 message (0.05 ms)
     ├── ▷  PASS  "color(a98-rgb-linear 0 1 .5)" 1 message (0.03 ms)
     ├── ▷  PASS  "color(prophoto-rgb-linear 0 1 .5)" 1 message (0.05 ms)
     ├── ▷  PASS  "color(acescc 0 1 .5)" 1 message (0.02 ms)
     └─┬ ▽  PASS  "color(acescg 0 1 .5)" 1 message (0.03 ms)
       └─┬ (Messages)
         └── (warn) ACEScg is a non-standard space and not currently supported in the CSS spec. Use prefixed color(--acescg) instead of color(acescg).

[DmitrySharabin]

Why do you think the result screen is broken? Do I get it right that you expect to see this:

instead of this:

If so, this is not a bug; it's a feature. 🙃 By default, hTest doesn't show in the final output groups where all tests passed. This is why you don't see the groups like: “Chromatic adaptation tests,” “Construcrot tests,” “Contrast tests,” and so on.

Only groups with failed, skipped, or tests with messages are shown:

And even in that case, passed tests (without messages) are hidden in such groups.

Probably, you need a way to show all groups of tests with all tests in them, regardless? Right? Like this?

[kleinfreund]

Aha, I see. Yes, I hadn't considered at all that that could be intentional.

When it comes to debugging, I'm a stubborn console.log user and I'll often compare log output from passing with that from failing tests. It looks like depending on where I add tests in an htest test suite, I can "provoke" the output for passing tests to appear in the report.

For this usage scenario, I think disabling the interactive mode together with showing results for passing tests would be a fairly direct approach to tackle this.

[DmitrySharabin]

For this usage scenario, I think disabling the interactive mode together with showing results for passing tests would be a fairly direct approach to tackle this.

With our last changes, this is possible. You can use the --ci and --verbose CLI options for that: npx htest ./test/parse.js --ci --verbose. Those changes are not yet released, but if you could install hTest from the GitHub repo, try them, and let us know what you think, it would be helpful.

[kleinfreund]

Yeah, this works great! 👍 Thank you very much.

One more thing I noticed, but it's not related to this issue:

The logs often end up being [object Object] when logging things like console.log({value}) so I end up having to do console.log(JSON.stringify({value}))

[DmitrySharabin]

Yeah, this works great! 👍 Thank you very much.

Yay! 🥳

The logs often end up being [object Object] when logging things like console.log({value}) so I end up having to do console.log(JSON.stringify({value}))

Oh, that's an excellent observation. We should serialize the intercepted messages before logging them. Thanks for the heads up!

[LeaVerou]

Those changes are not yet released,

Let's make a new release?

Oh, that's an excellent observation. We should serialize the intercepted messages before logging them. Thanks for the heads up!

We should probably avoid serializing them ourselves if we can.