feat(unstable): deno test --coverage

[caspervonb]

This adds a coverage option to the test command which prints a summary report of lines covered to stdout.

Closes #106

[CLAassistant]

All committers have signed the CLA.

[buttercubz]

@caspervonb Would it be something similar to coverage in jest?

[caspervonb]

@caspervonb Would it be something similar to coverage in jest?

Not sure what jest does but this collects block based call coverage directly from the runtime.

Output formats to be determined but lcov is fairly obvious as it's a ubiquitous format.

[bartlomieju]

I think we'll need to implement something like inspector agent that can automatically track proper ids

[caspervonb]

Don't need anything particularly fancy for this case, just increment by one per message is fine.

Long term for the REPL etc yeah we'll definitively want a full blown inspector client.

[bartlomieju]

Right, but then you need to receive messages from inspector for each sent message and I believe inspector can send other messages as well which should be ignored.

[caspervonb]

Right, but then you need to receive messages from inspector for each sent message

The two TODO's tie together somewhat, e.g each socket send will loop awaiting the next response, breaking when a response with the sent id is received then error checking the result.

I believe inspector can send other messages as well which should be ignored.

It can.

Enabling a domain (e.g calling the Runtime.enable method) will enable event notifications for that domain. Events do not carry identifiers however so this doesn't interfere with the above.

[bartlomieju]

test coverage:
async/mux_async_iterator.ts 97.101%
async/deferred.ts 100.000%
io/writers.ts 100.000%
io/ioutil.ts 97.619%
io/util.ts 100.000%
io/streams.ts 100.000%
io/_iotest.ts 88.679%
fs/exists.ts 93.548%
encoding/toml.ts 98.995%
encoding/base64url.ts 100.000%
encoding/csv.ts 97.382%
encoding/binary.ts 83.083%

Very nice 👍

[caspervonb]

So this works, needs to be cleaned up a bit this approach will work.

However, it turns out tracefile sections (lcov format) can start with the test name.

A tracefile is made up of several human-readable lines of text, divided into sections. If available, a tracefile begins with the testname which is stored in the following format:

TN:<test name>

And we can't get at the test name since the test runner is implemented in JS; If we were to integrate the coverage collector directly in the test runner on the however we could support this.

I think implementing it in JS would be the cleaner implementation in the long run, for one the threading issue just goes away. Other things implemented in JS like the REPL are also in need of an inspector client for things like tab completion so it's somewhat inevitable that we'll end up with an internal inspector client in JS someday.

🤔

[bartlomieju]

So this works, needs to be cleaned up a bit this approach will work.

However, it turns out tracefile sections (lcov format) can start with the test name.

A tracefile is made up of several human-readable lines of text, divided into sections. If available, a tracefile begins with the testname which is stored in the following format:
TN:

And we can't get at the test name since the test runner is implemented in JS; If we were to integrate the coverage collector directly in the test runner on the however we could support this.

I think implementing it in JS would be the cleaner implementation in the long run, for one the threading issue just goes away. Other things implemented in JS like the REPL are also in need of an inspector client for things like tab completion so it's somewhat inevitable that we'll end up with an internal inspector client in JS someday.

🤔

@caspervonb thanks for detailed description. Inspector client in JS is a desirable feat, but I think we could land this PR only with terminal output for the first pass. Could the report be made a bit more detailed, eg. showing distinct percentages for branches and statements?

[caspervonb]

Could the report be made a bit more detailed, eg. showing distinct percentages for branches and statements?

Hmm, I'll clean this up first but it's doable, just have to walk the AST checking if node ranges are inside covered ranges to to calculate that.

[bartlomieju]

@caspervonb could you rebase the branch? I'll look into the problems you described above.

[caspervonb]

We shouldn't start the inspector web(socket)server when collecting coverage only.

Agreed; I was considering making the host argument of DenoInspector::new an Option and only register with the web server when it is Some as a quick and dirty hack pending a refactor (a lot, if not all of this can be generalised into a general purpose inspector client).

--coverage shouldn't require --inspect.

Agreed.

Nice that you managed to implement your own inspector::Channel. I see that it uses a mpsc to communicate, which suggests that multiple isolates in different threads share the same channel. Is this true?

Thought I had to thread it initially but turned out to be fine to just let it run on the same thread as the main isolate so this could probably just be a queue.

[caspervonb]

Thanks @bartlomieju for merging things into CoverageCollector which more or less what I had in mind.

I think this is good enough to land for 1.4, still things to be desired but I'll follow up on hardening the inspector session and separating inspector web socket server from the inspector along with lcov and json reporters etc for ~1.5 or ~1.6.

[bartlomieju]

LGTM, thank you @caspervonb this is a massive feature! Looking forward to future iterations and other output formats support.

[antoniogamiz]

Where can I see the docs for this feature?

[bartlomieju]

https://deno.land/manual@v1.4.4/testing#test-coverage

[antoniogamiz]

Thanks a lot!