feat: ESM support #759

mdonnalley · 2023-08-08T20:01:27Z

Adds full support for ESM and CJS plugin interoperability
Adds integration tests for ESM/CJS interoperability

ESM/CJS Interoperability

CJS Root Plugin

✅ Install CJS plugin
✅ Install ESM plugin
✅ Link CJS plugin
⚠️ Link ESM plugin

Auto-compilation will NOT work. ESM plugin source must be compiled before runtime

ESM Root Plugin

✅ Install CJS plugin
✅ Install ESM plugin
✅ Link CJS plugin
⚠️ Link ESM plugin

Auto-compilation will work if NODE_OPTIONS="--loader=ts-node/esm" is set in the environment. Otherwise, it will fallback to compiled source code

QA

OCLIF_ALLOW_ESM=true oclif generate my-esm-cli
cd my-esm-cli
yarn link @ocilf/core

Can I install CJS plugin? (@oclif/plugin-test-cjs-2)
Can I install ESM plugin? (@oclif/plugin-test-esm-1)
Can I link CJS plugin? (https://github.com/oclif/plugin-test-cjs-2)
- Do changes to linked CJS plugin get transpiled at runtime?
Can I link ESM plugin? (https://github.com/oclif/plugin-test-esm-1)
- Do changes to linked ESM plugin get transpiled at runtime with NODE_OPTIONS="--loader=ts-node/esm"?
- Does it fall back to compiled source if NODE_OPTIONS="--loader=ts-node/esm" is not set?

oclif generate my-cjs-cli
cd my-cjs-cli
yarn link @ocilf/core

Can I install CJS plugin? (@oclif/plugin-test-cjs-1)
Can I install ESM plugin? (@oclif/plugin-test-esm-2)
Can I link CJS plugin? (https://github.com/oclif/plugin-test-cjs-1)
- Do changes to linked CJS plugin get transpiled at runtime?
Can I link ESM plugin? (https://github.com/oclif/plugin-test-esm-2)
- Can I compiled ESM plugin and then run from CJS root? (NOTE: transpiling is not expected to work in this scenario)

git clone https://github.com/cristiand391/hello-world.git
cd hello-world
yarn install && yarn build
sf plugins link
sf hello world

Does command work as expected (no require() of ES Module error)

Deployment plan

Fixes

Fixes oclif/oclif#333
Fixes oclif/oclif#695
Fixes oclif/oclif#712
Fixes oclif/oclif#906
Fixes oclif/oclif#1139
Fixes oclif/oclif#1143
Fixes #748
Fixes #755
Fixes #757

@W-13883004@
@W-13856380@
@W-13915423@
@W-13825218@

iowillhoit · 2023-08-30T20:22:45Z

QA NOTES

ESM CLI

🟢 Can create an ESM cli
🟢 Linked @oclif/core
🟢 Can run generated commands with bin/dev.js in esm cli
🟢 Can run generated commands with bin/run.js in esm cli
🟢 Can run commands with bin/dev.js in plugin-test-cjs-2
🟢 Can run commands with bin/run.js in plugin-test-cjs-2
🟢 Can run commands with bin/dev.js in plugin-test-esm-1
🟢 Can run commands with bin/run.js in plugin-test-esm-1

Installing CJS

🟢 bin/dev.js plugins install @oclif/plugin-test-cjs-2 installs
🟢 bin/dev.js cjs2 works
🟢 bin/run.js cjs2 works
🟢 bin/dev.js plugins uninstall @oclif/plugin-test-cjs-2 uninstalls

Installing ESM

🟢 bin/run.js plugins install @oclif/plugin-test-esm-1 installs
🟢 bin/dev.js esm1 works
🟢 bin/run.js esm1 works
🟢 bin/run.js plugins uninstall @oclif/plugin-test-esm-1 uninstalls
🟡 The plugin's init-hooks run when I run any command. Guessing this is expected

Linking CJS

🟢 bin/dev.js plugins link ../plugin-test-cjs-2 links
🟢 bin/dev.js cjs2 works
🟢 bin/run.js cjs2 works
🟢 Made changes to plugin-test-cjs-2/src/commands/cjs2.ts
- 🟢 bin/dev.js cjs2 compiles and shows changes
- 🟢 bin/run.js cjs2 compiles and shows changes
🟢 bin/dev.js plugins unlink plugin-test-cjs-2 unlinks
- 🟢 Can also unlink from path bin/dev.js plugins unlink ../plugin-test-cjs-2

Linking ESM

🟢 bin/dev.js plugins link ../plugin-test-esm-1 links
🟢 bin/dev.js esm1 shows warning about not being able to auto-compile
🟢 Make changes to esm-1, yarn build, bin/dev.js esm1 shows changes
🟢 Made changes to plugin-test-ems-1/src/commands/esm1.ts
- 🟢 NODE_OPTIONS="--loader=ts-node/esm" bin/dev.js esm1 compiles and shows changes
- 🟢 NODE_OPTIONS="--loader=ts-node/esm" bin/run.js esm1 compiles and shows changes
🟡 I tried relinking cjs while ems was linked and got this message. Linking was successful though
- SyntaxError Plugin: @oclif/plugin-test-cjs-2: Cannot use import statement outside a module
🟢 Both plugins work (and update) side-by-side
🟢 NODE_OPTIONS="--loader=ts-node/esm" bin/dev.js cjs2 makes no difference
🟢 Made changes to plugin-test-esm-1/src/hooks/init/init.ts
- 🟢 NODE_OPTIONS="--loader=ts-node/esm" bin/dev.js esm1 compiles and shows changes in hook
- 🟢 NODE_OPTIONS="--loader=ts-node/esm" bin/run.js esm1 compiles and shows changes in hook
🟢 bin/dev.js plugins unlink plugin-test-esm-1 unlinks

CJS CLI

🟢 Can create an CJS cli
🟢 Linked @oclif/core
🟢 Can run generated commands with bin/dev in cjs cli
🟢 Can run generated commands with bin/run in cjs cli
🟡 A CJS CLI still uses bin/dev instead of bin/dev.js (same with run). Is this desired?
- I noticed that the CJS plugin is using the new bin/*.js files, that makes me wonder
- Will there be any other consequences of renaming these files?

Installing CJS

🟢 bin/dev plugins install @oclif/plugin-test-cjs-2 installs
🟢 bin/dev cjs2 works
🟢 bin/run cjs2 works
🟢 bin/dev plugins uninstall @oclif/plugin-test-cjs-2 uninstalls

Installing ESM

🟢 bin/dev plugins install @oclif/plugin-test-esm-1 installs
🟢 bin/dev esm1 works
🟢 bin/run esm1 works
🟢 bin/dev plugins uninstall @oclif/plugin-test-esm-1 uninstalls

Linking CJS

🟢 bin/dev plugins link ../plugin-test-cjs-2 links
🟢 bin/dev cjs2 works
🟢 bin/run cjs2 works
🟢 Made changes to plugin-test-cjs-2/src/commands/cjs2.ts
- 🟢 bin/dev cjs2 compiles and shows changes
- 🟢 bin/run cjs2 compiles and shows changes

Linking ESM

🟢 bin/dev plugins link ../plugin-test-esm-1 links
🟢 bin/dev esm1 shows warning about not being able to auto-compile
🟢 Made changes to plugin-test-ems-1/src/commands/esm1.ts
- 🟢 Running bin/dev esm1 did not show updates
- 🟢 Running yarn build then bin/dev esm1 does shows changes
🟢 Both plugins work (and update) side-by-side
🟢 bin/dev plugins unlink plugin-test-esm-1 unlinks
🟡 NIT: The stack track on bin/dev when you have an esm plugin linked is noisy. In this case, I'm running the linked cjs plugin (not esm):

bin/dev cjs2
Warning: @oclif/plugin-test-esm-1 is a linked ESM module and cannot be auto-compiled from a CommonJS root plugin. Existing compiled source will be used instead.
    at warn (/Users/ewillhoit/dev/core/lib/errors/index.js:49:15)
    at memoizedWarn (/Users/ewillhoit/dev/core/lib/errors/index.js:66:9)
    at tsPath (/Users/ewillhoit/dev/core/lib/config/ts-node.js:109:39)
    at Plugin.get commandsDir [as commandsDir] (/Users/ewillhoit/dev/core/lib/config/plugin.js:153:50)
    at Plugin.get commandIDs [as commandIDs] (/Users/ewillhoit/dev/core/lib/config/plugin.js:157:19)
    at Plugin._manifest (/Users/ewillhoit/dev/core/lib/config/plugin.js:250:47)
    at async Plugin.load (/Users/ewillhoit/dev/core/lib/config/plugin.js:136:25)
    at async /Users/ewillhoit/dev/core/lib/config/config.js:532:17
    at async Promise.all (index 0)
    at async Config.loadPlugins (/Users/ewillhoit/dev/core/lib/config/config.js:519:9)
Greetings-weee! from plugin-test-cjs-2 init hook
Greetings-new2! from plugin-test-esm-1 init hook
Hola I am a CJS plugin from /Users/ewillhoit/dev/sandbox/my-cjs-cli!

mdonnalley · 2023-08-30T22:32:26Z

@iowillhoit Thanks for doing such thorough QA!

🟡 The plugin's init-hooks run when I run any command. Guessing this is expected

Yes, that's expected

🟡 NIT: The stack track on bin/dev when you have an esm plugin linked is noisy. In this case, I'm running the linked cjs plugin (not esm):

This is the default for warnings when running in development. If you use bin/run.js it'll be a clean, single line warning

🟡 A CJS CLI still uses bin/dev instead of bin/dev.js (same with run). Is this desired?
I noticed that the CJS plugin is using the new bin/*.js files, that makes me wonder
Will there be any other consequences of renaming these files?

It's not a hard requirement for plugins that are still in CJS and they don't intend to bundle/install any ESM plugins. If they intend to be interoperable, then yes they will need to start using the new bin scripts.

🟡 I tried relinking cjs while ems was linked and got this message. Linking was successful though
SyntaxError Plugin: @oclif/plugin-test-cjs-2: Cannot use import statement outside a module

This only happens when using bin/dev.js but it'll go away once plugin-plugins uses this version of core.

* chore: v3 prerelease branch * chore(release): 3.0.0-beta.1 [skip ci] * feat: ESM support (#759) * fix: set moduleResolution to Node16 * fix: use file urls * chore: add types * feat: support linking CJS plugins into ESM plugins * chore: integration tests for module interoperability * test: add CJS/ESM interoperability tests * fix: e2e tests * test: isolate cache, config, and data dir for e2e tests * test: test bug * test: make less noisy * test: esm/cjs hooks * chore: parallelize e2e tests * chore: typo * test: parallelize e2e tests * test: no more hanging tests * chore: un-parallelize esm-cjs tests * chore: add DEBUG to esm-cjs interop tests * chore: add DEBUG to esm-cjs interop tests * chore: try enabling debug again * test: fix node 20 tests * chore: update DEBUG env var * chore: debug tests * chore: debug tests * test: more debugging * test: more debugging * test: more debugging * test: more debugging * test: more debugging * test: more debugging * test: more debugging * test: use path.join * test: stop using replace-in-file * chore: more test debugging * fix: use path.join when registering ts-node * test: run in parallel * test: run tests serially * fix: dont remove undefined values from tsconfig * fix: further isolate ts-configs * feat: throw error when loading ESM paths from linked plugins * fix: default esm to true * test: compilation errors * feat: better developer experience * fix: add getPluginsList * chore(release): 2.11.9 [skip ci] * fix: add getPluginsList to Config interface * chore(release): 2.11.10 [skip ci] --------- Co-authored-by: svc-cli-bot <svc_cli_bot@salesforce.com> * chore(release): 3.0.0-beta.2 [skip ci] * More v3 features (#772) * fix: set moduleResolution to Node16 * fix: use file urls * chore: add types * feat: support linking CJS plugins into ESM plugins * chore: integration tests for module interoperability * test: add CJS/ESM interoperability tests * fix: e2e tests * test: isolate cache, config, and data dir for e2e tests * test: test bug * test: make less noisy * test: esm/cjs hooks * chore: parallelize e2e tests * chore: typo * test: parallelize e2e tests * test: no more hanging tests * chore: un-parallelize esm-cjs tests * chore: add DEBUG to esm-cjs interop tests * chore: add DEBUG to esm-cjs interop tests * chore: try enabling debug again * test: fix node 20 tests * chore: update DEBUG env var * chore: debug tests * chore: debug tests * test: more debugging * test: more debugging * test: more debugging * test: more debugging * test: more debugging * test: more debugging * test: more debugging * test: use path.join * test: stop using replace-in-file * chore: more test debugging * fix: use path.join when registering ts-node * test: run in parallel * test: run tests serially * fix: dont remove undefined values from tsconfig * fix: further isolate ts-configs * feat: throw error when loading ESM paths from linked plugins * fix: default esm to true * test: compilation errors * feat: better developer experience * fix: add getPluginsList * chore(release): 2.11.9 [skip ci] * fix: add getPluginsList to Config interface * chore(release): 2.11.10 [skip ci] * feat: use exports * fix: ux class * fix: add exports for help and perf * chore: tests * feat: use Map for config.plugins * feat: add noCacheDefault * feat: add exports for Args and Flags * chore: fix tests * feat: put permutations into manifest * fix: add return to execute * fix: add version to PJSON interface * feat: drop node 14 and 16 support * fix: use CLIError for integer flag errors * fix: skip findRoot for linked plugins * chore: remove console.log * fix: dont load core plugin if already loaded as linked or user plugin * fix: ux export * chore: making things readonly * feat: add config export * test: make tests passing again * feat: rename cli-ux global to ux global * chore: go back to node 16 support * chore: migration guide * chore: remove source-map register * test: remove resolution * chore: update migration guide * fix: revert link findRoot optimization * fix: remove unneeded exports * feat: add PluginLoader * chore: update migration guide * chore: merge conflict * chore: more merge conflicts * test: recursive mkdir --------- Co-authored-by: svc-cli-bot <svc_cli_bot@salesforce.com> * chore(release): 3.0.0-beta.3 [skip ci] * chore(release): 3.0.0-beta.4 [skip ci] * fix: add main to package.json * chore(release): 3.0.0-beta.5 [skip ci] * fix: export CustomOptions * chore(release): 3.0.0-beta.6 [skip ci] * fix: interface exports * chore(release): 3.0.0-beta.7 [skip ci] * feat: remove ux and interfaces export * chore(release): 3.0.0-beta.8 [skip ci] * fix: revert ux changes * chore(release): 3.0.0-beta.9 [skip ci] * feat: revert exports changes (#776) * feat: revert exports changes * fix: run e2e tests even if linting error * fix: revert cli-ux dir renaming * chore(release): 3.0.0-beta.10 [skip ci] * fix: revert styledJSON change * chore(release): 3.0.0-beta.11 [skip ci] * fix: update imports in Config * chore(release): 3.0.0-beta.12 [skip ci] * feat: skip ts-node register for ESM plugins (#778) * feat: skip ts-node register for ESM plugins * chore: update debug message * chore(release): 3.0.0-beta.13 [skip ci] * feat: final v3 features (#779) * feat: add charAliases * feat: add Flags.option * fix: undo default options * fix: allow bin/dev.js to auto-transpile ESM plugin * chore: update execute examples * fix: update tsnode skip logic * chore: test debugging * fix: ts-node skip logic * fix: ts-node skip logic * feat: cache relativePath and isESM in manifest * fix: calculate id permutations at runtime to support backwards compatability * perf: avoid findLegacyRoot for linked plugins * chore: remove env var * fix: improve perf metrics * perf: improve perf debug output * perf: more debug improvements * test: compilation errors * fix: make relativePath OS safe * test: use sf esm branch * perf: give full hook path * chore: test debugging * chore: test debugging * chore: test debugging * test: set shell for e2e tests * fix: flag types respect defaults * feat: node protocol * test: move windows sf integration tests into separate job * test: update assertion on plugins install test * test: use correct file path * test: use -Force * test: use bash shell * test: clean up * test: remove shell option * test: oclif/config, core v1, and core v2 interop tests * chore: drop node 16 * test: remove lts/-1 * chore: code review * chore: add pre core migration guide * refactor: break up ModuleLoader * chore: remove unicorn/consistent-function-scoping * chore: remove unicorn/no-missing-imports * chore: remove @typescript-eslint/no-empty-function * chore: remove ban-ts-comment and ban-ts-ignore * chore: add sort-import rule * fix: throw error if non-multiple flag provided more than once * test: mutliples of non-multiple flag test * test: ut for charAliases (#791) * test: duplicate aliases tests * test: extend timeout * test: split windows esm-cjs tests * test: parallelize linux interop tests too * test: typo * test: use right executor * test: use right executor for clean up * chore: update eslint libs (#792) * chore: update eslint libs * chore: clean up * chore: tests and linting * test: windows paths * fix: exports * test: update run import * chore: replaceAll * fix: error exit codes * fix: use es2021 * feat: use ES2022 * fix: use ES2021 again * test: incorporate flags and args in esm-cjs tests * test: use correct expected values --------- Co-authored-by: Shane McLaughlin <shane.mclaughlin@salesforce.com> * chore(release): 3.0.0-beta.14 [skip ci] * fix: allow undefined from flag parser * chore(release): 3.0.0-beta.15 [skip ci] * feat: set spinner style * chore(release): 3.0.0-beta.16 [skip ci] * test: unset config vars after setting them * feat: remove ts-node (#794) * feat: drop ts-node dependency * test: use --json for config unset * chore(release): 3.0.0-beta.17 [skip ci] * fix: use ES2022 (#793) * fix: use ES2022 * test: use --json for config unset * feat: stop using getters and setters for flags * chore: clean up * feat: expose json flag * feat: remove pass through getter and setter * fix: correct order of flags in toCached * chore: clean up * fix: flag merge order * chore: documentation * test: use oclif/test v3 * feat: set spinner style on windows too * fix: handle cmd with baseFlags but no flags * fix: some circular deps * fix: circular deps in help * fix: ts-node and config circular deps * fix: toCached circular dep in help * chore: organize utils * test: enforce no circular deps * chore: remove Flags.json * chore: add prettier config * test: add nyc * test: improve test coverage * test: windows unit tests * chore: revert change to automerge.yml * chore: code review * perf: parallelize cacheCommand * chore(release): 3.0.0-beta.18 [skip ci] * chore: use prettier and lint-staged (#798) * chore: update prettier config * chore: prettier and lint-staged * chore: reformat everything * chore: deal with reformat aftermath * chore: update configs and deps * perf: move ansi-escapes require to top level * chore: automerge.yml * chore: unpin eslint-plugin-prettier * chore: remove prettier plugin * chore: update lint-staged config * fix: update util import * chore(release): 3.0.0-beta.19 [skip ci] * chore: remove util/index.ts (#802) * fix: remove tsPath export * fix: move @types/cli-progress to devDeps * chore(release): 3.0.0-beta.20 [skip ci] * fix: force release * chore(release): 3.0.0-beta.21 [skip ci] * fix: move styledJson to index * chore(release): 3.0.0-beta.22 [skip ci] * feat: collect perf results from outside oclif (#797) * feat: collect perf results from outside oclif * chore: pr feedback * refactor: naming for non-core perf * chore(release): 3.0.0-beta.23 [skip ci] * fix: remove OCLIF_NEXT_VERSION * chore: add prepare for husky install * test: try false for ignoreScripts * chore(release): 3.0.0-beta.25 [skip ci] * test: ignore scripts for external nuts * chore: use eslint-plugin-perfectionist (#807) --------- Co-authored-by: svc-cli-bot <svc_cli_bot@salesforce.com> Co-authored-by: Shane McLaughlin <shane.mclaughlin@salesforce.com>

mdonnalley added 30 commits July 31, 2023 09:40

fix: set moduleResolution to Node16

bb489fb

fix: use file urls

add1235

chore: add types

1c810d0

feat: support linking CJS plugins into ESM plugins

c697617

chore: integration tests for module interoperability

1396b41

Merge branch 'main' into mdonnalley/esm-support

b09e6d0

test: add CJS/ESM interoperability tests

4e76304

fix: e2e tests

956015e

test: isolate cache, config, and data dir for e2e tests

e8c4b2f

test: test bug

63f34c5

test: make less noisy

0f45299

test: esm/cjs hooks

e0d69bb

chore: parallelize e2e tests

1ae82f2

chore: typo

c84e1b6

test: parallelize e2e tests

fec3c69

test: no more hanging tests

8e5157c

chore: un-parallelize esm-cjs tests

775c28a

chore: add DEBUG to esm-cjs interop tests

0b8e332

chore: add DEBUG to esm-cjs interop tests

96800d4

chore: try enabling debug again

1be16fc

test: fix node 20 tests

83b9911

chore: update DEBUG env var

47f204b

chore: debug tests

8c18164

chore: debug tests

36ae625

test: more debugging

b1ba804

test: more debugging

d5e9b7b

test: more debugging

cfa9053

test: more debugging

6096973

test: more debugging

24c6ca1

test: more debugging

ef19b8f

mdonnalley mentioned this pull request Aug 14, 2023

Bug: TypeError [ERR_UNKNOWN_FILE_EXTENSION]: Unknown file extension "" oclif/oclif#712

Closed

fix: further isolate ts-configs

bb68d78

This was referenced Aug 15, 2023

feat: new esm docs oclif/oclif.github.io#209

Merged

feat: remove OCLIF_ALLOW_ESM env var oclif/oclif#1171

Merged

mdonnalley added 3 commits August 15, 2023 10:57

feat: throw error when loading ESM paths from linked plugins

215a0c4

fix: default esm to true

c3f7d54

test: compilation errors

2d18e45

mdonnalley mentioned this pull request Aug 15, 2023

Migration command to make plugins ESM-compatible oclif/oclif#1172

Closed

feat: better developer experience

cf8b0c0

mdonnalley changed the base branch from main to prerelease/v3 August 21, 2023 21:02

mdonnalley and others added 5 commits August 23, 2023 09:20

fix: add getPluginsList

b01083f

chore(release): 2.11.9 [skip ci]

56245c4

fix: add getPluginsList to Config interface

cba7a75

chore(release): 2.11.10 [skip ci]

0e13bbd

Merge branch 'main' into mdonnalley/esm-support

9625fb8

mdonnalley mentioned this pull request Aug 29, 2023

ERR_UNKNOWN_FILE_EXTENSION oclif/hello-world-esm#28

Closed

Merge branch 'prerelease/v3' into mdonnalley/esm-support

19bbab3

iowillhoit approved these changes Aug 31, 2023

View reviewed changes

mdonnalley merged commit b9c2da4 into prerelease/v3 Aug 31, 2023
28 checks passed

mdonnalley deleted the mdonnalley/esm-support branch August 31, 2023 16:53

mdonnalley mentioned this pull request Feb 5, 2024

Bug: registering ts-node multiple times oclif/oclif#333

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: ESM support #759

feat: ESM support #759

mdonnalley commented Aug 8, 2023 •

edited

iowillhoit commented Aug 30, 2023

mdonnalley commented Aug 30, 2023 •

edited

feat: ESM support #759

feat: ESM support #759

Conversation

mdonnalley commented Aug 8, 2023 • edited

ESM/CJS Interoperability

CJS Root Plugin

ESM Root Plugin

QA

Deployment plan

Fixes

iowillhoit commented Aug 30, 2023

QA NOTES

ESM CLI

Installing CJS

Installing ESM

Linking CJS

Linking ESM

CJS CLI

Installing CJS

Installing ESM

Linking CJS

Linking ESM

mdonnalley commented Aug 30, 2023 • edited

mdonnalley commented Aug 8, 2023 •

edited

mdonnalley commented Aug 30, 2023 •

edited