serve mode does not use ResourceLoader modules

[ChlodAlejandro]

A big issue with the library as of now is possible deviation of what's being tested when using serve mode (vite dev) and what's actually ran by users when the script is placed onto a user JS file.

Background

Currently, ResourceLoader (through mw.loader.using) only supports CommonJS imports (require("module")). On the other hand, Vite uses and encourages the use of ESM and targets ESNext. Currently, this plugin changes the output format into CJS when building (by changing Rollup settings), but this functionality is impossible to achieve when serving.

There now exists a discrepancy between ResourceLoader modules and modules sent to the browser in serve mode. One of these is due to the version: keeping the dependencies unlocked from version changes through a caret version selector means the installed version that's sent to the browser during dev could deviate from the version of the library currently on the MediaWiki core. For example, a dependent library could have Vue v3.3.6 whereas the MediaWiki core would have v3.3.9—a bug that is present in the dev server may be entirely fixed when built and saved on-wiki. The solution to this is just always pinning the package version, and that's something that must be done on a dependent projects' side (but we could add automated warnings for this prior to building).

The second, and more pressing issue, is that it is extremely difficult if not impossible to "hijack" imports in serve mode. In build mode, Rollup is set to completely avoid bundling specified external dependencies. If using is set to [ "vue" ], it doesn't bundle Vue and instead emits a require("vue") call, which can be cleanly placed inside mw.loader.using's function. In serve mode, there is no transformation to CJS occurring. Everything is still sent as ESM, and imports are used to load dependencies.

Solutions

In trying to find a way to load ResourceLoader versions instead, here are some possible solutions I thought of/tried:

Swap out the dependency

This is possible by using a custom load and resolveId hook. For consistency, let's use vue as the example library to be loaded from RL. resolveId is called, asking for a resolved version of vue. We intercept it, and replace it with a mw.loader.using call, which then exports whatever we loaded from MediaWiki.

Except this is impossible, because exports must be static. module.exports is not possible in ESM; every export must be statically analyzable. We also can't just toss require("vue") as a default export, because Vue doesn't have default exports. Once this "fake" module is loaded, the browser will try to resolve { createApp }, but won't be able to find it since there'd only be a default export.

The way around this using normal Rollup is by using this.getModuleInfo to extract the exports of the package, and re-export those. Maybe inject exports for things that we know the MediaWiki version has too (like createMwApp). But accessing exports and hasDefaultExport is banned in Vite. The reasoning behind this is placed very obscurely in the Vite documentation:

[...] some plugin hooks do not make sense in an unbundled dev server context.
— Plugin API § Rollup Plugin Compatibility. Vite Documentation.

Note that the moduleParsed hook is not called during dev, because Vite avoids full AST parses for better performance.
— Plugin API § Universal Hooks. Vite Documentation.

(n.b. moduleParsed would have received information that is returned by getModuleInfo, but getModuleInfo is bare-bones: it only returns an id, the resolved filename of the module being imported, and a blank meta object, because no AST parsing is done.)

We could attempt to resolve everything that the MediaWiki version of vue exports by loading the load.php result of the vue import, but we then have to mock a browser and pull out the Object.keys for require("vue") post-load, which greatly complicates things.

• We could do this as a build step, but this means releasing new package patch bumps whenever a MediaWiki core module is bumped (which we have to check every train branch cut).
• We could do this process remotely and ask the library to pull down data from there (e.g. run this resolution process through CI, then publish the results on Toolforge), but this means requiring additional and external resources from the internet, a no-no in security.
• Neither of the two above address the problem of needing to "mock a browser". It is possible to do static analysis of the MediaWiki source (i.e. start from resources/src/vue/index.js and work our way there in resolution, but it adds an even worse layer of complexity: module resolution (and it has to be static!) of an IIFE.
• We might not be able to do this live, as we don't have access to the browser JS when resolving modules and, thus, we can't get the Object.keys as loaded from the remote.
  • We could use client–server communication to make the browser run Object.keys(require("vue")), and then feed that back to the plugin to resolve the module. This should only be slow once: when the module is being loaded for the first time on the page. ResourceLoader caches the module, so subsequent mw.loader.using calls without navigation shouldn't require waiting for a web request, and we could also cache this on the plugin, while the dev server is loaded. This is the most ideal; but it still needs to be tested.

Force use of CommonJS imports

This essentially makes the optimizations of Vite moot as it focuses on using ESM for improved performance. We might as well have used Webpack instead, which basically does this. It also goes against the Vite philosophy of keeping things as close to the original code as possible; something that we also need to value as Wikimedia script developers.

Long-term thinking

This project was, first and foremost, an experiment. I'm really not sure if Vite is the proper way to go with this, and it seems this specific issue is a big hurdle for development. However, I went with it because (1) I was planning to use Vue anyway, so Vite was an obvious-ish choice, (2) it's much faster than Webpack, and (3) it has HMR — a must-have for userscript development, because loading Wikipedia over and over again with refreshes is a waste of bandwidth. If this is the way forward, this issue needs to be addressed, top priority.

Fix condition

The following code should be valid:

import { createMwApp } from 'vue';
import App from './App.vue';

createMwApp( App ).mount( '#app' );

<template>
	<div id="codex-test-app">
		<h1>{{ $i18n( 'test-message' ).text() }}</h1>
		<cdx-button action="progressive" weight="primary">
			{{ $i18n( 'test-message-button' ).text() }}
		</cdx-button>
	</div>
</template>

<script lang="ts">
// lang="ts" should work because transpilation is done during build
import { defineComponent } from 'vue';
import { CdxButton } from '@wikimedia/codex';

export default defineComponent( {
	components: { CdxButton }
} );
</script>

[ChlodAlejandro]

• We could do this as a build step, but this means releasing new package patch bumps whenever a MediaWiki core module is bumped (which we have to check every train branch cut).
• We could do this process remotely and ask the library to pull down data from there (e.g. run this resolution process through CI, then publish the results on Toolforge), but this means requiring additional and external resources from the internet, a no-no in security.

Eliminating these solutions. They require loading a massive amount of modules from core, which may have adverse effects on the browser running the checking. In addition, these are not extendable.

• We could use client–server communication to make the browser run Object.keys(require("vue")), and then feed that back to the plugin to resolve the module. This should only be slow once: when the module is being loaded for the first time on the page. ResourceLoader caches the module, so subsequent mw.loader.using calls without navigation shouldn't require waiting for a web request, and we could also cache this on the plugin, while the dev server is loaded. This is the most ideal; but it still needs to be tested.

Regretfully eliminating this solution. I was unable to add in a listener on the client side that seamlessly integrated with the user's code. importing a virtual file doesn't attach the listener until we've resolved everything; but the listener is required to resolve everything, so it gets stuck. Without another way to run the listener on the browser prior to importing our actual application code, this method won't work.

Besides that, this will have issues when it comes to cross-origin tests. Right now, a broadcast occurs to all tabs with the Vite client, so if we load another tab on a different origin (even worse with a different MediaWiki version), it might respond faster than the tab which is currently loading the user's script, and we could get desync between what keys we receive and what keys are actually available through mw.loader.using.

So we're back to square one here. I've run out of possible solutions to test.