Switch to ESM modules

[lotas]

Motivation

In node.js ESM ("type": "module") package can import both CommonJS packages, and ESM packages. However, commonjs package cannot import newer ESM packages.

As more and more dependencies switch to ESM-only distribution, making upgrades (especially security upgrades) become hard.

Taskcluster platform consists of multiple utility (library) packages, services themselves, and clients.

Library packages: db, api, app, config, iterate, loader, monitor, postgres, pulse, references, testing, validate

Services: auth, built-in-workers, github, hooks, index, notify, object, purge-cache, queue, secrets, web-server, worker-manager

Clients that are published: taskcluster-client, taskcluster-client-web.

It is not possible to only convert services packages to ESM, as they depend on library packages, which also consume 3rd-party dependencies.

Clients, however, do not depend on those, so could possibly remain in CommonJS format to avoid shipping both ESM/CJS builds.

taskcluster-client is a published standalone package, but uses unlisted dependencies for testing: taskcluster-lib-*, taskcluster-testing. Those are not listed in package dependencies, as those packages are not installed, but are internal.
This was refactored, and implicit dependencies were removed, and tests that depends on those were moved into separate client/client-test folder

importing

ESM requires that files would be imported with extensions.

// This worked fine in cjs
const { util } = require('./util')

// this wouldn't work anymore
import { util } from './util'

// this will
import { util } from './util.js'
// or if it is a folder
import { util } from './util/index.js'

For the same reason (I believe) running node services/auth/src/main server will not work, but node services/auth/src/main.js server will work

__dirname, __filename

Can now be used with the help of import.meta.url:

const __filename = new URL('', import.meta.url).pathname;
const __dirname = new URL('.', import.meta.url).pathname;

module.parent no longer available

Not possible to do:

if (!module.parent) {
  load.crashOnError(process.argv[2]);
}

This will work:

import { fileURLToPath } from "url";
if (process.argv[1] === fileURLToPath(import.meta.url)) {
  load.crashOnError(process.argv[2]);
}

Testing

Since ESM require exports to be statically analyzable, test helper functions are being rewritten:
instead of exports.xx and dynamic injections of helper functions, helper.js now exposes single helper = {} object that is being modified by utility functions like withDb(), withPulse()

Exports

Taskcluster internal libraries expose everything as named export and as default to allow mixed usage:

export * from "./upgrade.js";
import { upgrade, downgrade } from './upgrade.js';
export default { upgrade, downgrade };

Can be used both ways:

import { upgrade } from 'taskcluster-db';
upgrade()
// or
import db from 'taskcluster-db';
db.upgrade()

Import assertions

Node.js allows to specify import assertions since v17:

// load json module
import jsonSchemaDraft06 from 'ajv/lib/refs/json-schema-draft-06.json' assert { type: 'json' };

// equivalent of 
const jsonSchemaDraft06 = require('ajv/lib/refs/json-schema-draft-06.json');

However, it is still experimental and comes with:

(node:40419) ExperimentalWarning: Importing JSON modules is an experimental feature and might change at any time

Also eslint doesn't implement it yet, waiting for it to be analysed.

Alternatives:

const dataDir = new URL('./data/yml/', import.meta.url).pathname;
const loadJson = filename => JSON.parse(fs.readFileSync(path.join(dataDir, filename), 'utf8'));
// or
const schemaPath = new URL('../../../node_modules/ajv/lib/refs/json-schema-draft-06.json', import.meta.url).pathname;
const jsonSchemaDraft06 = JSON.parse(fs.readFileSync(schemaPath, 'utf-8'));

---

Files were migrated in dumb way npx cjs-to-es6 path/to/files and does not result in a working code, there are still many steps to follow:

TODO

• db/
• infrastructure/tooling
• libraries/api
• libraries/app
• libraries/config
• libraries/iterate
• libraries/loader
• libraries/monitor
• libraries/postgres
• libraries/pulse
• libraries/references
• libraries/testing
• libraries/validate
• services/auth
• services/built-in-workers
• services/github
• services/hooks
• services/index
• services/notify
• services/object
• services/purge-cache
• services/queue
• services/secrets
• services/web-server
• services/worker-manager

Fixes #4260

[lotas]

although clients are mostly untouched, and are still CJS, worth making a major

[lotas]

this is copied from lib/testing to avoid having to implicitly depend on package from outside.

[lotas]

as this whole test depends on "internal" libraries, it shouldn't be here.
figuring out where to put it

[lotas]

this was added to keep clients/client clean, and separate tests that used internal libraries from the client itself.

[lotas]

closed in favor of #6627