[ESM] Jest crashes with `request for ... is not yet fulfilled` linking error

[nicolo-ribaudo]

🐛 Bug Report

When two modules dynamically imported in parallel depend on the same file*, Jest crashes with a request for ... is not yet fulfilled error. The error is thrown by https://github.com/nodejs/node/blob/e46c680bf2b211bbd52cf959ca17ee98c7f657f5/src/module_wrap.cc#L535, so it's possible that this is a Node.js bug and not a Jest bug.

In my repository example below, I import two files in parallel (if I import them serially the test passes):
https://github.com/nicolo-ribaudo/babel/blob/31dac517b7ce0d504bda794075f0a88029cedd58/packages/babel-plugin-proposal-private-methods/test/repro.js#L6-L7

Those two files both depend on packages/babel-helper-create-class-features-plugin/lib/index.js. packages/babel-helper-create-class-features-plugin/lib/index.js depends on packages/babel-helper-create-class-features-plugin/lib/fields.js, and the error I get is about that file:

request for './fields.js' is not yet fulfilled

To Reproduce

Repository link: https://github.com/nicolo-ribaudo/babel/tree/jest-esm-request-not-yet-fulfilled-error

Steps to reproduce:

make bootstrap
node --experimental-vm-modules ./node_modules/.bin/jest babel-plugin-proposal-private-methods/test/repro

Expected behavior

It shouldn't crash

envinfo

  System:
    OS: Linux 5.8 Ubuntu 20.10 (Groovy Gorilla)
    CPU: (12) x64 Intel(R) Core(TM) i7-8750H CPU @ 2.20GHz
  Binaries:
    Node: 14.17.0 - /usr/local/bin/node
    Yarn: 2.4.1 - /usr/bin/yarn
    npm: 7.9.0 - ~/.npm-global/bin/npm
  npmPackages:
    jest: 27.0.0-next.11 => 27.0.0-next.11 

It also fails with Node.js 16.2.0

[nicolo-ribaudo]

I know nothing about this other than what the GH search surfaced, but:

• I think this error is linking error, dependency promises must be resolved on instantiate, just with a new error message
• From the last messages in Node.js 15.9 regression with Jest ESM nodejs/node#37426, maybe the problem is that the two imports try to link()/evaluate() the same file at the same time?

[SimenB]

Interesting find! The error is thrown when await the promise returned by module.link here: https://github.com/facebook/jest/blob/22267424a824310779bad3cafaae59485ce2a6c7/packages/jest-runtime/src/index.ts#L530. module.status at this point is unlinked, but the module which node says is "not yet fulfilled" is linking. I would have thought node just waited for modules in status linking to complete when it's returned from module.link callback, but maybe not?

I've tried for about an hour to understand why it goes boom without luck... Might need to ask in the node repo. I think we currently do only call link and evaluate on entry points and on dynamic imports, but I might be wrong 😛

Do you think you would be able to create a reproduction outside of the babel repo? I tried to add a failing test to this repo without success.

In https://github.com/facebook/jest/blob/22267424a824310779bad3cafaae59485ce2a6c7/e2e/native-esm/__tests__/native-esm.test.js

test('import same file in parallel indirectly', async () => {
  await Promise.all([import('../file1.js'), import('../file2.js')]);
});

and then both file1.js and file2.js being

import func from './sharedImport.js';

func();

and sharedImport.js

export default function func() {
  console.log('woop');
}

This passes instead of failing with the same error, though.

[nicolo-ribaudo]

Sure, I can try 👍

[nicolo-ribaudo]

https://github.com/nicolo-ribaudo/jest-bug-not-yet-fulfilled

However, I don't see the error if I add a test similar to my repository to https://github.com/facebook/jest/blob/22267424a824310779bad3cafaae59485ce2a6c7/e2e/native-esm/__tests__/native-esm.test.js 🤔 EDIT: #11437

[nicolo-ribaudo]

Node throws if the promise isn't already resolved: https://github.com/nodejs/node/blob/e46c680bf2b211bbd52cf959ca17ee98c7f657f5/src/module_wrap.cc#L534

However, it seems in contrast with the idea behind promises: the "settled" status shouldn't be observed, a promise already resolved to x should behave exactly like a promise that will resolve to x.

[axmad386]

same issue here, but I can't tell the detail. It's to complicated lol
I use polka (like express) and map the routes with middleware. I use Promise.all to map the routes and middleware. Some middleware import multer (and I mock it). This middleware is attached twice to the routes. I found that the first route is fine, but the second one is failed to run, and throw this error message
FATAL request for 'multer' is not yet fulfilled
{"code":"ERR_VM_MODULE_LINK_FAILURE"}
Something like multer module cannot load for the second time

It's only happen when I test my app with jest. In production is fine

this is the copy of my code

const runServices = async (routes, routeMiddleware) => {
    await Promise.all(["post", "get", "patch", "put", "delete"].map(async (method) => {
        await Promise.all((routes[method] || []).map(async (r) => {
            let service;
            let localMiddleware = [];
            try {
                const instance = await import(servicePath + r.service);
                service = instance.default || instance;
                (service.middleware || []).forEach((m, i) => {
                    let mInstance = m.default || m; //multer imported here
                    localMiddleware[i] = async (req, res, next) => {
                        try {
                            await mInstance(req, res, next);
                        }
                        catch (err) {
                            handleError(req, res, err);
                        }
                    };
                });
                router[method](routes.prefix + r.path, ...routeMiddleware, ...localMiddleware, async (req, res) => {
                    try {
                        await serviceExec(req, res, service);
                    }
                    catch (err) {
                        handleError(req, res, err);
                    }
                });
            }
            catch (err) {
                Log.fatal(err);
            }
        }));
    }));
};

System:
OS: Linux 5.11 elementary OS 6 Odin
CPU: (4) x64 Intel(R) Core(TM) i5-5300U CPU @ 2.30GHz
Binaries:
Node: 16.6.2 - ~/.nvm/versions/node/v16.6.2/bin/node
Yarn: 1.22.11 - ~/.nvm/versions/node/v16.6.2/bin/yarn
npm: 7.20.3 - ~/.nvm/versions/node/v16.6.2/bin/npm
npmPackages:
jest: ^27.1.0 => 27.1.0