Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

How to run the Node built-in testrunner for TypeScript files inside a specific directory? #3902

Closed
jtuchel opened this issue Jun 21, 2022 · 37 comments

Comments

@jtuchel
Copy link

jtuchel commented Jun 21, 2022

Details

I want to replace Mocha tests with the built-in testrunner. All tests are inside a test directory and follow the pattern

...Tests.ts

I started with a fooTests.ts file

import assert from 'assert/strict';
import test from 'node:test';

test('1 is equal to 1.', () => {
  assert.strictEqual(1, 1);
});

and added the npm script

"test": "node --test ./test/**/*Tests.ts",

The script fails with the message

Could not find '/home/.../test/**/*Tests.ts'

How can I fix the script to

  • test files matching the path ./test/**/*Tests.ts only
  • run with TS files ( I'm pretty sure I must use ts-node for this )

? Thanks in advance!

Node.js version

v18

Operating system

Linux ( Ubuntu 22.04 )

Scope

Built-in testrunner

Module and version

Not applicable.

@adrienjoly
Copy link

Did you try injecting a TypeScript processor as a command line parameter for node, yet?

E.g. node --require @swc/register [...] or node --require ts-node/register [...]

@neodon
Copy link

neodon commented Aug 16, 2022

In case it helps anyone who runs across this issue, here is how I currently execute tests written in TypeScript with the new node test runner and esbuild-kit/tsx:

npm install -D tsx

Add the following npm script entry:

"test": "node --loader tsx --test test/**/*Test.ts"

Let me know if this is helpful.

Edit: See the reply below by @scottwillmoore for an excellent tip to make this work more robustly across platforms.

@MoLow MoLow closed this as completed Aug 17, 2022
@jtuchel
Copy link
Author

jtuchel commented Nov 8, 2022

@neodon I tried to reproduce your approach but I get a "not found" error ( I'm using Node 19 )

tsconfig.json

{
    "compilerOptions": {
        "baseUrl": ".",
        "declaration": true,
        "esModuleInterop": true,
        "lib": ["es2022", "dom"],
        "module": "commonjs",
        "outDir": "build",
        "resolveJsonModule": true,
        "strict": true,
        "target": "es2022"
    },
    "include": ["./**/*.ts"],
    "exclude": ["./build"]
}

package.json

{
    "scripts": {
        "test": "tsc --noEmit && node --loader tsx --test test/**/*Tests.ts"
    },
    "dependencies": {
        "@types/node": "18.11.9"
    },
    "devDependencies": {
        "tsx": "3.11.0",
        "typescript": "4.8.4"
    }
}

./test/someTests.ts

import assert from 'assert/strict';
import { describe, it } from 'node:test';

describe('tests', () => {
    it('passes', () => {
        assert.ok(true);
    });
});

The test script fails with

Could not find '/home/.../repository/test/**/*Tests.ts'

Did I miss something?

@scottwillmoore
Copy link

scottwillmoore commented Nov 8, 2022

EDIT:

See this example repository which I have created.

Problem

I haven't tested your environment, but if I had to guess I believe it is your npm script which is incorrect:

tsc --noEmit && node --loader tsx --test test/**/*Tests.ts

On a POSIX system npm will execute the script with /bin/sh. The pattern test/**/*Tests.ts is not expanded by node or npm it is up to /bin/sh to expand it. However, **, also known as globstar is not supported by /bin/sh, it is not being expanded as you would expect. The globstar is supported by /bin/bash, but must be enabled.

Solution

I could be wrong, but I believe since globstar is not supported, the pattern test/**/*Tests.ts becomes test/*/*Tests.ts which will only match files in a subdirectory of your test folder, such as test/abc/xyzTests.ts. If you just wanted to match tests in the root of the test folder, you could rewrite the pattern as test/*Tests.ts, which would match tests/xyzTests.ts, but would not match files in a subdirectory of your test folder.

Better Solution

It would be better to rewrite the script in a cross-platform manner, instead of relying on platform dependent features in your scripts which is may be unreliable. This way your script should even work on Windows.

There might be an easier way to leverage another package, but I think I would just move it into a JavaScript or TypeScript script.

Change your script to:

tsc --noEmit && node --loader tsx runTests.ts

Then create a file named runTests.ts.

I don't have the time to experiment with a full script, but I expect you would want to use the glob package to get the list of files and the child_process Node API to then call node --loader tsx --test ....

Related

Why can't ** be used (for recursive globbing) in a npm package.json script, if it works in 'npm shell'?

Why you should always quote your globs in NPM scripts.

Follow Up

I originally wrote this answer of StackOverflow, but perhaps someone here has an easier solution than writing your own script. Also, I hope I've actually diagnosed the issue, otherwise that would be a little awkward 😬 .

@webpro
Copy link

webpro commented Dec 30, 2022

Another option to run the same command on Windows as well is to use globstar:

{
  "scripts": {
    "test": "globstar -- node --loader tsx --test \"test/**/*.spec.ts\""
  }
}

(Last release of this package seems to be 8 years ago, but still does the job.)

Offroaders123 added a commit to Offroaders123/jsmediatags that referenced this issue Apr 9, 2023
Did some general fixer-uppers to the codebase! Now it's more ES6-friendly in some spots, it consistently uses double-quotes now, and part of the ES6-friendly bits, now arrow functions are used a bit more, and any var declaration that's modified once is now a const declaration :)

Forgot to include some links that helped out for the last few commits:
nodejs/help#3902
https://stackoverflow.com/questions/35470511/setting-up-tsconfig-with-spec-test-folder
https://jestjs.io/docs/ecmascript-modules
https://khalilstemmler.com/blogs/typescript/abstract-class/ (Going to look into if this is what MediaFileReader should instead be made with. Haven't used this before!)
https://stackoverflow.com/questions/45251664/derive-union-type-from-tuple-array-values (THIS IS FREAKING EPIC, big thanks to this post)
@Ethan-Arrowood
Copy link

Ethan-Arrowood commented Apr 21, 2023

@scottwillmoore I was trying out your example and in Node v20, it now breaks with:

node-test-with-typescript on  main [!] via 🤖 v20.0.0 on ☁️  production 
❯ npm test  

> test
> node --loader tsx --no-warnings script/test.ts

/Users/ethanarrowood/Documents/github/vercel/vercel-azure-devops-extension/node-test-with-typescript/script/test.ts:1
import { execSync } from "node:child_process";
^^^^^^

SyntaxError: Cannot use import statement outside a module
    at internalCompileFunction (node:internal/vm:73:18)
    at wrapSafe (node:internal/modules/cjs/loader:1187:20)
    at Module._compile (node:internal/modules/cjs/loader:1231:27)
    at Module._extensions..js (node:internal/modules/cjs/loader:1321:10)
    at Module.load (node:internal/modules/cjs/loader:1125:32)
    at Module._load (node:internal/modules/cjs/loader:965:12)
    at ModuleWrap.<anonymous> (node:internal/modules/esm/translators:165:29)
    at ModuleJob.run (node:internal/modules/esm/module_job:192:25)

Node.js v20.0.0

The simplest solution I found was to add "type": "module", to my project's package.json

@scottwillmoore
Copy link

scottwillmoore commented Apr 23, 2023

You're right! This does not happen in Node 19 with 19.0.1, or even 19.9.0, but it does happen in Node 20 with 20.0.0. The error message appears to indicate that the TypeScript is not getting transpiled into CommonJS, but instead to ModuleJS. But, with no changes to the loader tsx and other dependencies, this must have been caused by a change in Node 20? However, I couldn't find any concrete reason in the changelog for Node 20.

The solution, as indicated with @Ethan-Arrowood is to just add "type": "module" to your package.json, therefore Node will accept the TypeScript that is transpiled to ModuleJS. I've updated my example repository to reflect these changes. To be honest, if you are using the latest version of Node, you probably should use ModuleJS anyway.

I've also double checked, and a custom script is still required to run TypeScript with the Node test runner at the moment. This is due to the current Node test runner execution model. However, there is a feature request to fix this issue: nodejs/node#46292.

@reedHam
Copy link

reedHam commented Apr 24, 2023

Hmm when I try node --loader tsx --test with node 20 on windows.
I'm getting this error.

node:internal/worker:211
    this[kHandle] = new WorkerImpl(url,
                    ^
This Environment was initialized without a V8::Inspector
Thrown at:
    at Worker (node:internal/worker:211:21)
    at InternalWorker (node:internal/worker:464:5)
    at HooksProxy (node:internal/modules/esm/hooks:496:20)
    at CustomizedModuleLoader (node:internal/modules/esm/loader:351:18)
    at createModuleLoader (node:internal/modules/esm/loader:421:14)
    at get esmLoader (node:internal/process/esm_loader:15:26)
    at node:internal/modules/cjs/loader:119:47
    at node:internal/util:764:15
    at Module.load (node:internal/modules/cjs/loader:1128:26)
    at Module._load (node:internal/modules/cjs/loader:965:12)

@MoLow
Copy link
Member

MoLow commented Apr 24, 2023

Hmm when I try node --loader tsx --test with node 20 on windows. I'm getting this error.

node:internal/worker:211
    this[kHandle] = new WorkerImpl(url,
                    ^
This Environment was initialized without a V8::Inspector
Thrown at:
    at Worker (node:internal/worker:211:21)
    at InternalWorker (node:internal/worker:464:5)
    at HooksProxy (node:internal/modules/esm/hooks:496:20)
    at CustomizedModuleLoader (node:internal/modules/esm/loader:351:18)
    at createModuleLoader (node:internal/modules/esm/loader:421:14)
    at get esmLoader (node:internal/process/esm_loader:15:26)
    at node:internal/modules/cjs/loader:119:47
    at node:internal/util:764:15
    at Module.load (node:internal/modules/cjs/loader:1128:26)
    at Module._load (node:internal/modules/cjs/loader:965:12)

it seems to be caused by https://github.com/nodejs/node/blob/8becacb25d3c275341a9fef1a94581de3bd60c3d/src/env-inl.h#L638-L641

@nodejs/loaders @nodejs/workers do workers require a v8 inspector in order to work?

@JakobJingleheimer
Copy link

This sounds familiar

nodejs/node#44710 (comment)

I think it ended up being a red-herring.

@webpro
Copy link

webpro commented Apr 25, 2023

A potential solution to the OP is to pre-compile TS to JS. Then you don't need a loader and with more than two unit tests it becomes a lot faster too, since you compile everything only once and no further overhead. (example commit that does this using swc).

@GeoffreyBooth
Copy link
Member

I think it ended up being a red-herring.

Are you sure? I think it was a red herring for that PR in that Node’s internal test runner doesn’t use --test, so the command you were using for one-off testing in that comment wasn’t working but the actual Node tests were passing. But I think maybe it really is a bug that --test and --loader can’t be used together at the moment, because of https://github.com/nodejs/node/blob/8becacb25d3c275341a9fef1a94581de3bd60c3d/src/env-inl.h#L640. cc @MoLow

@scottwillmoore
Copy link

Weird. I have Windows 11 Pro (Build 22621) and in PowerShell with Node 20 installed with Volta I don't have any issues.

git clone git@github.com:scottwillmoore/node-test-with-typescript
cd ./node-test-with-typescript

volta --version # 1.1.1
node --version # v20.0.0
npm --version # 9.6.4

npm clean-install

# Indirect
npm run test # node --loader tsx --no-warnings ./scripts/test.ts

# Direct
node --loader tsx --no-warnings --test ./tests/add.test.ts

Could @MoLow try with this example, or make their example reproducible.

@MoLow
Copy link
Member

MoLow commented Apr 26, 2023

But I think maybe it really is a bug that --test and --loader can’t be used together

@GeoffreyBooth is v8 inspector a requirement for using workers?

@GeoffreyBooth
Copy link
Member

But I think maybe it really is a bug that –test and –loader can’t be used together

@GeoffreyBooth is v8 inspector a requirement for using workers?

Well I think most users would assume that they can attach the debugger to worker threads, so when we run v8 for a worker it expects to have that feature enabled?

@Tobbe
Copy link

Tobbe commented Apr 29, 2023

https://github.com/isaacs/node-glob recently got cli capabilities, so this is what I do now

"test:node": "yarn glob './src/**/__tests__/*.test.mts' --cmd='yarn node --loader tsx --test-reporter spec --test'",

@scottwillmoore
Copy link

Awesome, thanks for the heads up. I've updated my example to use this approach. You can use the command below if you don't want to require Yarn.

"test": "glob -c \"node --loader tsx --no-warnings --test\" \"./tests/**/*.test.ts\""

@nohehf
Copy link

nohehf commented May 25, 2023

Hi, following back on this, is it possible to pass flags to tsx when using it as a node loader? I need to specify a different tsconfig for testing. Thanks!
Edit: you can pass the ts config to tsx via ESBK_TSCONFIG_PATH env var. However, I still wonder if passing flags to the loader is possible.

@joshunrau
Copy link

Also another solution for those for non-windows users who don't want to install additional dependencies:

"test": "find ./src -name '*.spec.ts' -exec node --loader @swc-node/register/esm --test {} \\;"

kachick added a commit to kachick/renovate-config-asdf that referenced this issue Jul 23, 2023
kachick added a commit to kachick/renovate-config-asdf that referenced this issue Jul 23, 2023
* `npm install --save-dev @tsconfig/strictest @tsconfig/node18`

* Update tsconfig with using tsconfig/base again

* `npm install --save-dev tsx`

* `npm uninstall ts-node-test`

* Use glob to run tsx with test runner for TS files

nodejs/help#3902 (comment)

* Replace ts-node-test with tsx

* Update docs around repl
@jasonwilliams
Copy link

jasonwilliams commented Aug 4, 2023

In case it helps anyone who runs across this issue, here is how I currently execute tests written in TypeScript with the new node test runner and esbuild-kit/tsx:

npm install -D tsx

Add the following npm script entry:

"test": "node --loader tsx --test test/**/*Test.ts"

Let me know if this is helpful.

Edit: See the reply below by @scottwillmoore for an excellent tip to make this work more robustly across platforms.

I've followed this and I get SyntaxError: Cannot use import statement outside a module, am i doing something wrong?

@mintyPT
Copy link

mintyPT commented Aug 16, 2023

Also another solution for those for non-windows users who don't want to install additional dependencies:

"test": "find ./src -name '*.spec.ts' -exec node --loader @swc-node/register/esm --test {} \\;"

This sent me in the right direction but I wanted to pass all the files to the command instead of one by one. So here's what worked for me:

"test": "find ./test -name '*Tests.ts' | tr '\\n' ' ' | xargs node --require ts-node/register --test "

@jasonwilliams
Copy link

jasonwilliams commented Aug 17, 2023

Just seen above #3902 (comment) that the fix for the syntax issue is to set “type”:”module” in package.json

Is there a workaround for those of us who can’t do that? This is a big project, and most of the files in this project are still commonjs and that can’t really change due to other reasons (upstream electron not supporting ESM right now).

is there not a way to have tsx output commonjs instead? Or to use .mjs if it’s insistent on using esm.

@scottwillmoore

@scottwillmoore
Copy link

@jasonwilliams

The whole CJS and ESM migration is kind of ugly at the moment... As mentioned in my previous comment, I did investigate this problem, but never found a good answer. I don't have the time at the moment to continue looking into this specific issue, but I'll leave some notes below which might help someone more motivated.


At the moment Node has two module loaders. There is the CommonJS module loader and the ECMAScript module loader. You can read how Node determines which module loader to use in the documentation.

To load an unsupported file type such as TypeScript the module loader has to be overridden. However, the method used to achieve this for each module loader is quite different.

  • The older CommonJS loader (require) could be monkey-patched? It was common to use Node with the --require argument to load a package/script which would modify the module object to change the loader behaviour?

  • The newer ECMAScript loader (import) can not be monkey-patched? Instead Node now provides the Loaders API to override its behaviour. A package/script which implements this API is loaded with the experimental --loader argument. I guess the advantage of this approach is that it is more composable, a it doesn't expose the internal implementation?

Now, the tsx package uses both cjs-loader and esm-loader under the hood.

  • cjs-loader is loaded with the --require argument and will patch the only the CommonJS loader.

  • esm-loader is loaded with the --loader argument and will use the loaders API which will adjust only the ECMAScript loader.

So, there are many factors which determine what may happen in your case.

  1. You execute a script with Node.
  2. Node decides what module loader to use.
  3. Any package/script with --loader is used to adjust the ECMAScript module loader with the Loaders API.
  4. Any package/script with --require is executed, which may monkey-patch the CommonJS loader with the module object.
  5. Your script is executed then loaded that module loader.

It will depend on the module loader determined by Node. It will depend on how tsx is invoked, either with --require or with --loader. And, it continues to get more complicated...

Both cjs-loader and esm-loader will then use esbuild to transform JavaScript and TypeScript into a format that their patched module loader can process.

  • cjs-loader will transform ESM and TypeScript (CommonJS and ESM) into CommonJS JavaScript?

  • esm-loader will transform CommonJS and TypeScript (CommonJS and ESM) into ESM JavaScript?

Anyway, so I don't know if that makes much sense. This comment is far longer than I anticipated. I don't know how accurate my description of this process is, but the overall process is clearly complicated and hard to understand and fix. It was at this point in the past I kind of gave up.


@jasonwilliams

I don't know your exact problem, but I suspect you could try to use --require instead of --loader as in your case you want to adjust the behaviour of the CommonJS module loader. Otherwise, you are going to have to use the information above to try and narrow down your exact problem.

@jasonwilliams
Copy link

jasonwilliams commented Aug 18, 2023

Thanks @scottwillmoore

In case anyone else comes here, I've managed to get it working by doing node --require @esbuild-kit/cjs-loader --test test/**/xt.ts instead. This at least does work for now.

It seems like the underlying issue is a Node v20 breaking change

If I run node --loader @esbuild-kit/esm-loader --test test/**/x.t.ts it also fails, so the problem is narrowed down there. Maybe the esm loader is doing something which was previously dependent on environment information and now it fails.

@robertsLando
Copy link

robertsLando commented Sep 19, 2023

Giving that this costed me hours of time I will share my finding.

Easy setup:

npm install -D esbuild-register
node --test -r esbuild-register test/*.ts

Using custom runTests.ts see here (more updated example on link):

import { readdirSync } from 'node:fs'
import { run } from 'node:test'
import process from 'node:process'
import { spec as Spec } from 'node:test/reporters'
import { basename } from 'node:path'
import { cpus } from 'node:os'

const spec = new Spec()

let exitCode = 0

const files = readdirSync(__dirname)
	.filter((f) => f.endsWith('.ts') && f !== basename(__filename))
	.map((f) => `${__dirname}/${f}`)

const start = Date.now()

const testStream = run({
	files,
	timeout: 60 * 1000,
	concurrency: cpus().length,
})

testStream.compose(spec).pipe(process.stdout)

const summary: string[] = []

testStream.on('test:fail', (data) => {
	exitCode = 1
	const error = data.details.error

	summary.push(
		`${data.file} - "${data.name}" (${Math.round(
			data.details.duration_ms,
		)}ms)\n${error.toString()} `,
	)
})

testStream.on('test:stderr', (data) => {
	summary.push(`${data.file} - Error:\n${data.message} `)
})

testStream.once('end', () => {
	const duration = Date.now() - start
	// print duration in blue
	console.log(
		'\x1b[34m%s\x1b[0m',
		`\nℹ Duration: ${duration / 1000}s\n`,
		'\x1b[0m',
	)
	if (summary.length > 0) {
		console.error('\x1b[31m%s\x1b', '\n✖ failing tests:\n')
		console.error(summary.join('\n'))
		console.error('\n------', '\x1b[0m\n')
	}
	process.exit(exitCode)
})

The pro of using run is that you have access to more options like concurrency that are not yet available in cli, also you can build some nice looking summary with errors that makes it easy to find them

@JustinGrote
Copy link

JustinGrote commented Sep 22, 2023

Quick note that @robertsLando's example does not work on NodeJS 18.15 due to the reporters not being exposed. This is what is currently used in Electron 25 and subsequently, VSCode 1.82. A similar method totally works tho, I'll publish here once I refine it a bit.

@robertsLando
Copy link

@JustinGrote could you explain me what specifically doesn't work with my example in nodejs 18? I have nodejs 18 and 20 in my ci and they both work

@JustinGrote
Copy link

@robertsLando 18.15 specifically, some of the items like the compose() function on stream and test/reporters being accessible publicly were not available yet, mostly added in 18.17

@umstek
Copy link

umstek commented Oct 8, 2023

I really tried but without any luck. Finally resorted to run-tests.ts

import { run } from "node:test";
import { spec } from "node:test/reporters";
import process from "node:process";

import { glob } from "glob";

const tsTests = await glob("src/**/*.test.ts");

run({ files: tsTests }).compose(new spec()).pipe(process.stdout);

.swcrc

{
  "$schema": "https://json.schemastore.org/swcrc",
  "jsc": {
    "parser": {
      "syntax": "typescript",
      "tsx": false
    }
  }
}

and package.json

{
  // ...
  "type": "module",
  "scripts": {
    // ...
    "test": "SWCRC=true node --loader @swc-node/register/esm run-tests.ts"
    // ...
  }
  // ...
}

Was it worth the effort? IDK.

Update:

  • I'm using node v20.7.0
  • @robertsLando's answer is better, I didn't see it. You can mix that and this to get a better result if you're already using swc.

@SanderPs
Copy link

After a lot of trial and error I got this to work on Windows:

  "scripts": {
    "test": "glob -c \"node --test --require ts-node/register\" \"./tests/**/*.ts\""
  },

@Dias1c
Copy link

Dias1c commented Oct 31, 2023

IMPORTANT: This command works only on linux

For run all test files in you project by "node" on typescript, run command bellow:

find -name "*test.ts" | xargs node --loader tsx --test

If it works, add it to scripts in package.json

I tried to run all tests with node --loader tsx --test ./**/*.test.ts but it isn't worked for me, it isnot tests nested test files, only selected layer.

@JustinGrote
Copy link

@Dias1c pretty sure that's why people are using glob, it's cross platform :)

@maestart
Copy link

For new Node versions. The --loader flag was deprecated in Node v20.6.0, use --import flag.

This worked for me: node --import tsx --test ./tests/*test.ts

@jensbodal
Copy link

You can just use

"test": "tsx --test src/*.test.ts"

The real misunderstanding seems to be how default globbing works with nodejs. If you're using zsh for example and run

tsx --test src/**/*.test.ts

It will run fine. But if you use bash the shell it will fail. I assume the default globbing in nodejs uses the same backwards compatible syntax in linux where * and ** are equivalent.

So to run all tests under src and all subfolders is not possible without either

  1. Using a glob package that supports **
    "test": "glob src/**/*.test.ts -c 'tsx --test'"
    
  2. Using something like find to find all the test files
    "test": "tsx --test $(find src -type f -name '*.test.ts')"
    
  3. Manually including all possible nesting levels for folders
    "test": "tsx --test src/*.test.ts src/*/*.test.ts src/*/*/*/*.test.ts"
    

@varunkamra
Copy link

varunkamra commented Dec 13, 2023

Hey guys, is anyone having issues with environment variables not being defined with node test runner? I am running my test with this command node --experimental-test-coverage --no-warnings --test -r ts-node/register tests/api/base.test.ts
When I run this and try to console log process.env it's not defined. However, when I try to run a normal .ts script with command node -r ts-node/register testenv.ts and log process.env, it is able to log all environment variables.

@zauni
Copy link

zauni commented Apr 23, 2024

Awesome, thanks for the heads up. I've updated my example to use this approach. You can use the command below if you don't want to require Yarn.

"test": "glob -c \"node --loader tsx --no-warnings --test\" \"./tests/**/*.test.ts\""

Currently you can even shorten it to (on Node.js 20.12.1)

"test": "glob -c \"tsx --test\" \"./src/**/*.test.ts\""

@BoscoDomingo
Copy link

BoscoDomingo commented Apr 25, 2024

I just made a succinct guide to tie all of the teachings mentioned here together, including .env file and correct TS language server support in test files, all compatible with monorepos.

Feel free to check my blog post and Gist that explain it all (both free, I make no money from either, it was just easier to share the info there for whomever may be interested :) )

TL;DR:

"scripts": {
    //...
    "test": "glob -c \"tsx --env-file .env --env-file env.test --test --test-reporter spec \" \"./test/**/*.test.ts\"",
    "build": "tsc -p tsconfig.build.json",

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests