Explore AMD to ESM migration

[jrieken]

The VS Code project uses AMD (asynchronous module definition) which is somewhat of a pre-runner of native JS modules (ESM). AMD is mostly dead technology and we should migrate away from it. Our TypeScript sources are written with ESM imports/exports but we took a few dependencies on AMD as a "runtime" and we need to understand what it takes to move towards ESM.

High-level topics:

• we have a special AMD loader plugin that supports nls
• we also have a special plugin for CSS
• we have special bundle logic for both NLS and CSS dependencies
• we use require.toUrl to locate sibling resources, like images, this needs to migrate onto import.meta.url

[jasonwilliams]

Hey @jrieken where would this leave extensions? Would there be a path for them to export esm? I believe right now they still export to commonjs because electron does not support esm

[jrieken]

Hey @jrieken where would this leave extensions? Would there be a path for them to export esm? I believe right now they still export to commonjs because electron does not support esm

@jasonwilliams For now extensions aren't affected by this and for the time being commonjs is how they roll. Tho, I am learning a ton and there will be a path for ESM extensions. We ignore this to keep this already large effort smaller. Also, if you wonder why this is complex: when extensions call require('vscode') something very special happens, there is no file for vscode but each extension gets an instance of a the API. This is only possible with commonjs loader tricks and we need to find similar ways in the ESM world.

[jasonwilliams]

@jrieken thanks for your response on this, looking forward to seeing what solution you come up with. I agree keeping extensions out makes things a lot easier.

Also, if you wonder why this is complex: when extensions call require('vscode') something very special happens, there is no file for vscode but each extension gets an instance of a the API. This is only possible with commonjs loader tricks and we need to find similar ways in the ESM world.

Would import maps not help with this? Extensions already don’t bundle “vscode” and leave it as an external reference.
I assume once Code has launched its extensions can be mapped to a specific location for the VSCode module. Anything more programmatic than that may take a while to be standarized.

[jrieken]

Yeah, import maps is our only viable option for this. Fingers crossed for Safari landing this in time for us

[zm-cttae]

If Safari doesn't support this, is core team blocked by the few months it takes for browser updates to percolate to userland?

See Google Chrome 110 popularity vs Google Chrome 109 here: https://www.stetic.com/market-share/browser/

[JustinGrote]

As an update, the electron version is now on an ESM-supportable version, but the loader still needs to be fixed to enable it. This is no longer an external dependency now however.

microsoft/vscode-loader#56

[metawrap-dev]

I will be very glad when this is resolved. I have a component of a project on hold until this can be resolved.

[jrieken]

Things for July

• • bring the alex/esm branch back into a good state
  • • run oss: ./scripts/code
    • • check issue reporter
    • • check process explorer
    • • check aux window
  • • run oss: ./scripts/code-cli --list-extensions (makes sure to launch to cliProcessMain)
  • • run oss: ./scripts/code-web
  • • run oss: ./scripts/code-server
  • • run oss: server-cli (has been overlooked so far)
  • • tests: ./scripts/test
  • • tests: yarn run test-browser
  • • tests: yarn run test-node
  • • tests: scripts/test-integration
  • • tests: scripts/test-web-integration
  • • tests: scripts/test-remote-integration
• • 🏃 Adopt electron support for ESM
• • Adopt node:module#register over server-loader.mjs (during dev we redirect to node_modules from the remote folder)
• • ⏫ verify if this is still needed, maybe NAPI solved this

[bpasero]

yarn test-node works, but shows some interesting few errors when loading jschardet:

  3784 passing (8s)
  40 pending
  5 failing
1) Encoding
   autoGuessEncoding (UTF8):

TypeError: Cannot read properties of undefined (reading 'iterator')
      at $jscomp.initSymbolIterator (evalmachine.<anonymous>:1:384)
      at $jscomp.makeIterator (evalmachine.<anonymous>:2:42)
      at evalmachine.<anonymous>:164:334
      at new c (evalmachine.<anonymous>:164:465)
      at evalmachine.<anonymous>:662:132
      at 42../constants (evalmachine.<anonymous>:662:395)
      at f (evalmachine.<anonymous>:5:32)
      at evalmachine.<anonymous>:5:66
      at 19../logger (evalmachine.<anonymous>:390:112)
      at f (evalmachine.<anonymous>:5:32)
      at evalmachine.<anonymous>:5:66
      at 1../src (evalmachine.<anonymous>:5:249)
      at f (evalmachine.<anonymous>:5:32)
      at a (evalmachine.<anonymous>:5:190)
      at DefineCall.callback (evalmachine.<anonymous>:5:218)
      at AMDModuleImporter.load (file:///Users/bpasero/Development/Microsoft/vscode-esm/out/vs/amdX.js:79:31)

[SimonSiefke]

@bpasero I encountered the same error. It seems the issue is that this is undefined at the top level of Ecmascript modules:

In jschardet.min.js:

var $jscomp = { scope: {}, global: this }, // this is undefined
	Symbol;
$jscomp.initSymbol = function () {
	$jscomp.global.Symbol || (Symbol = $jscomp.Symbol); // error occurs here
	$jscomp.initSymbol = function () {};
};

When accessing $jscomp.global.Symbol it is accessing undefined.Symbol.

There is a PR here microsoft/vscode-loader#58 that might fix it.

[bpasero]

Thanks for the insights, maybe I am missing something but in the alex/esm (#166033) branch where our work happens, we do not use the vscode-loader anymore I had thought 🤔

[SimonSiefke]

Thanks for the reply. I thought it might still be required, e.g. when an extension uses require in a webworker?

But I think you're also right that the same issue applies to the amdX loader.

Before, this was always a defined value. But now that EcmaScript modules are used, this is undefined, causing an error when importing jschardet.

A possible workaround could be to manually overwrite the jschardet code:

var $jscomp = { scope: {}, global: globalThis }, // change here
	Symbol;
$jscomp.initSymbol = function () {
	$jscomp.global.Symbol || (Symbol = $jscomp.Symbol);
	$jscomp.initSymbol = function () {};
};

[jrieken]

Thanks for the reply. I thought it might still be required, e.g. when an extension uses require in a webworker?

We fake require in that case. It's a single fetch and doesn't support dependencies ;-)

[jrieken]

It seems the issue is that this is undefined at the top level of Ecmascript modules:

@SimonSiefke The amdX module is actually a very simple AMD loader that we use to load our existing dependencies. This is a stepping stone so that the migration doesn't spread into all the places... When debugging the load-call, I see that this is defined and the error happens later when trying to access Symbol (which should also be defined...)

I didn't get to the bottom of this yet, it is a really weird error. Things work when using the non-minified version which I'll do for now

[SimonSiefke]

@jrieken you're right. It looks like Symbol is defined but it's running in a vm Context where accessing Symbol.iterator is not allowed because it is a different v8 context (or something like that?).

It seems only the jschardet.min.js version has this extra Symbol code for supporting older browsers that don't support for .. of loops.

[Tyriar]

With xtermjs/xterm.js#5092 xterm.js now ships an ESM module.

[bpasero]

Re jschardet: I opened aadsm/jschardet#96 to try to get the project to update to a newer version of google-closure-compiler that they use for minification. I am not able to reproduce the issue anymore with newer versions. It is possible that maybe some polyfills (such as for...of) are not applied anymore when using the newer compiler versions.

For now I think its OK to use the non-minified release. Eventually we should consider to minify all our node module dependencies in our pipeline and not depend on how others do it.

Reported as aadsm/jschardet#97