fix test coverage

[MartinNowak]

• use single tests as workaround for Issue 16397
• fix single tests (broken sed command)

[dlang-bot]

Thanks for your pull request, @MartinNowak!

Bugzilla references

Auto-close	Bugzilla	Description
✗	16397	missing coverage from template instances when using separate compilation

[wilzbach]

Awesome, thanks a lot for looking into this, Martin!

https://codecov.io/gh/dlang/phobos/compare/0b6b743b5bb4f3bccbfd9769f9bae2a9029f98c0...87587ce66195177b02e12b4f247c6d60057bb9a6/changes

[MartinNowak]

Only running the single tests (against a libphobos2.a w/o -cov) means we're no longer gathering coverage that happens from executing unittests of other modules, e.g. std.net.curl might use std.conv.
That seems actually like a saner model to work with, but might be problematic for unittests in external files (do we have that in phobos). That also explains why coverage for some modules decreased a bit.

[wilzbach]

Btw in these modules it decreases the coverage:

and in these it increases the coverage:

(not a complete screenshot)

I will have a look if coverSetMerge can help here to get the best from both worlds.

[CyberShadow]

I will have a look if coverSetMerge can help here to get the best from both worlds.

That doesn't sound like the right approach.

If the change increases coverage in some places but decreases it in others, isn't it essentially replacing one broken solution with another broken solution?

[wilzbach]

That doesn't sound like the right approach.

Better ideas are welcome!

If the change increases coverage in some places but decreases it in others, isn't it essentially replacing one broken solution with another broken solution?

Yes.

[CyberShadow]

@rainers Perhaps you could help us out with this one?

[jmdavis]

How on earth would only running the tests from a module when measuring its code coverage increase its code coverage? Does it have something to do with having fewer template instantiations? Regardless, it makes seem like there's a problem with the coverage analyzer.

[rainers]

@rainers Perhaps you could help us out with this one?

I'm not sure I can. I think Martin's assessment in https://issues.dlang.org/show_bug.cgi?id=16397#c1 sounds correct. I don't have a solution to that.

Always generating coverage instrumentation of templated code will only work if you cover all linked libraries, though. For example, coverage from druntime modules might leak into the report, just as well as that it might provide template instances without coverage instrumention.

[MartinNowak]

@wilzbach, @CyberShadow, @jmdavis, @rainers
Seems like noone read #5579 (comment), explains the decrease in some places. We no longer account for coverage as a collateral of running other module's tests. I think that's a sane behaviour but might run into issues for externalized tests such as std.regex.internal.tests.

[jmdavis]

Seems like noone read #5579 (comment), explains the decrease in some places.

The decrease makes sense to me (e.g. the tests for SysTime are primarily in std.datetime.timezone because of how tied they are to the time zones, and they were all in one module originally, so a decrease in std.datetime.systime's coverage wouldn't surprise me). It's the increase that doesn't make sense to me.

[MartinNowak]

It's the increase that doesn't make sense to me.

Read the explanation in the bug report https://issues.dlang.org/show_bug.cgi?id=16397#c1, the linker picked non-instrumented template instantiations in various places.

[jmdavis]

Read the explanation in the bug report https://issues.dlang.org/show_bug.cgi?id=16397#c1, the linker picked non-instrumented template instantiations in various places.

Ah, okay. That makes sense. LOL. Some of this stuff is just way more complicated than it seems like it should be.

[CyberShadow]

Seems like noone read #5579 (comment),

Sorry Martin, you're right, I only skimmed over the explanations. At the moment I was a bit overwhelmed and didn't grok it.

I guess then we have two three(?) ways going forward:

• Change DMD's coverage behaviour for external modules as described in issue 16397
• Improve direct test coverage and don't rely on other modules' tests for coverage
• Build a libphobos.a with -cov so that we measure indirect test coverage as well? Would that work?