Inconsistent coverage information when loaders used

[coderaiser]

Version

v14, v16, v17 latest

Platform

Linux, Darwin

Subsystem

No response

What steps will reproduce the bug?

🐛 What is bug related to?

The bug is related to coverage tool c8 it used to build coverage report for projects that are tested using mock-import.

Loaders

mock-import is a library built on top of loaders, it supports both versions of API. I know that this is experimental technology, anyways I'm using it on my projects, it works amazing and I'm ready for any API changes. By the way second version is better of Loaders API and it is much simpler, BUT would be great if both versions can co-exist without the need of passing NODE_OPTIONS. You don't need to support both versions, but please give ability for userland modules to smooth this differences, and have ability to write polyfil for it.

Why use loaders to mock imports?

Because it feats very well for using mocks in tests in the same way we used mock-require in good old days we had only CommonJS, since we don't have access to cache of EcmaScript Modules. So it's:

• ✅ ESM friendly;
• ✅ has ability to by-pass cache using query suffix: ?mock-import-count=x;
• ✅ changes ImportDeclaration to VariableDeclaration to get mocked code from predefined storage

To get transformations working 🐊Putout code transformer used, it based on Babel.

What is problem with transforming files on-a-fly using Loaders?

The coverage is the problem with mocked files transformed by loading. Since c8 uses NODE_V8_COVERAGE to put coverage on screen.

More low level details

You can see here all the story. The thing is for similar code fragments generated different coverage information.

So for code fragment with no transformation:

import {
    readFile,
} from 'fs/promises';

import {
    execSync,
} from 'child_process';

export default (a, b, c) => {
    if (a)
        return readFile();

      if (c)
          return execSync();

      return 'd';
};

With mocked first import:

const {
    readFile: readFile
} = global.__mockImportCache.get('fs/promises');

import {
    execSync,
} from 'child_process';

export default (a, b, c) => {
    if (a)
        return readFile();

      if (c)
          return execSync();

      return 'd';
}

And with mocking second-one

import {
    readFile,
} from 'fs/promises';

const {
    execSync: execSync
} = global.__mockImportCache.get('child_process');

export default (a, b, c) => {
    if (a)
        return readFile();

      if (c)
          return execSync();

      return 'd';
};

I have such coverage results that c8 get from coverage directory generated by node:
ranges for functionNam=default is different: two positions, three and three. This is conditions, third-one is absent in first report. Why is so?

{
    "result": [
        {
            "scriptId": "0",
            "url": "file:///Users/coderaiser/c8-reproduce/lib/changelog.js?mock-import-count=1",
            "functions": [
                {
                    "functionName": "",
                    "ranges": [
                        {
                            "startOffset": 0,
                            "endOffset": 1824,
                            "count": 1
                        }
                    ],
                    "isBlockCoverage": true
                },
                {
                    "functionName": "default",
                    "ranges": [
                        {
                            "startOffset": 144,
                            "endOffset": 271,
                            "count": 1
                        },
                        {
                            "startOffset": 196,
                            "endOffset": 270,
                            "count": 0
                        }
                    ],
                    "isBlockCoverage": true
                }
            ]
        },
        {
            "scriptId": "1",
            "url": "file:///Users/coderaiser/c8-reproduce/lib/changelog.js?mock-import-count=2",
            "functions": [
                {
                    "functionName": "",
                    "ranges": [
                        {
                            "startOffset": 0,
                            "endOffset": 1820,
                            "count": 1
                        }
                    ],
                    "isBlockCoverage": true
                },
                {
                    "functionName": "default",
                    "ranges": [
                        {
                            "startOffset": 144,
                            "endOffset": 271,
                            "count": 1
                        },
                        {
                            "startOffset": 178,
                            "endOffset": 196,
                            "count": 0
                        },
                        {
                            "startOffset": 244,
                            "endOffset": 270,
                            "count": 0
                        }
                    ],
                    "isBlockCoverage": true
                }
            ]
        },
        {
            "scriptId": "2",
            "url": "file:///Users/coderaiser/c8-reproduce/lib/changelog.js?mock-import-count=3",
            "functions": [
                {
                    "functionName": "",
                    "ranges": [
                        {
                            "startOffset": 0,
                            "endOffset": 1929,
                            "count": 1
                        }
                    ],
                    "isBlockCoverage": true
                },
                {
                    "functionName": "default",
                    "ranges": [
                        {
                            "startOffset": 109,
                            "endOffset": 236,
                            "count": 1
                        },
                        {
                            "startOffset": 143,
                            "endOffset": 161,
                            "count": 0
                        },
                        {
                            "startOffset": 191,
                            "endOffset": 209,
                            "count": 0
                        }
                    ],
                    "isBlockCoverage": true
                }
            ]
        }]
}

Looks like this is the reason why c8 shows that lines are not covered. But they are.

To reproduce bug clone repository with

git clone https://github.com/coderaiser/c8-reproduce
cd c8-reproduce
npm install
npm run coverage

What is the bug?

The bug is different coverage information for the same file content.

How often does it reproduce? Is there a required condition?

Always

What is the expected behavior?

Would be great if coverage information was the same for the similar type of content.

What do you see instead?

I see that coverage is missing information about code running code parts when Loaders are used to transform the source.

Additional information

Happy Holidays 🎄! And thank you for the hard work you are doing, node.js is the greatest platform 🎉 !

[aduh95]

@nodejs/loaders

[simlu]

Maintainer of test library here. Also running into this when hot-reloading for individual tests.

Would be great to get this looked at!

[GeoffreyBooth]

Would be great to get this looked at!

Pull requests are welcome!

[simlu]

Would be great to get this looked at!

Pull requests are welcome!

I would love to! But while I can reproduce the issue, I have not the faintest idea where the problem really is. It seems to be something pretty low level if I read these tickets correctly