proposal: separate Caching + TypeScript from Deno core into std

[oldrich-s]

I propose to separate Caching and TypeScript compilation from Deno to a separate std library. I believe that both scenarios could be solved by the following trick:

import { resolveModule } from 'http://deno.land/std/resolve-module/resolve-module.js'

import.meta.resolveModule = resolveModule

import('./my_library.js')

where resolveModule has type of (importPath: string) => Promise<string (javascript text)>. resolveModule function can throw / reject which would show an error in Deno console.

As I see it, the resolveModule function could support caching, transpilation, type checking, source maps, prettier and other things that are currently included in Deno core. It could also support the future versions of TypeScript or other compilers such as Flow etc.

You could provide a default import.meta.resolveModule implementation that would do everything Deno does today but would allow to be overwritten if desired.

What do you think, is there any reason why it cannot be done?

[ry]

I would like to simplify core as much as possible. TS support is quite a large amount of complexity. So I'm very open to such ideas.

That said, using V8 snapshots allows us to startup TS compilation very quickly - and it's not clear how that could be achieved without building it into core.

[oldrich-s]

Could you please elaborate on how V8 snapshots make the TS compilation startup faster?

I would argue that the compilation startup speed might be negligible compared to the time it takes to compile a typescript file (especially when doing remote fetch, full compilation including type checking and source map generation). Plain TypeScript transpilation is on the other hand quite fast and when js is already in local cache it is completely free.

So I would say that the benefit of having the possibility to choose a proper resolveModule function for the given task should outperform any optimization provided by V8 snapshots.

I also think that Deno core should be just a simple browser compatible JS runtime environment. Everything else should probably be implemented as std libraries.

EDIT:

When I think about it, Deno Core could allow to include a resolve.js and authorize.js javascript file:

deno.exe myfile.js --resolve=resolve.js  --authorize=authorize.js

resolve.js:

• A default resolve.js file would be bundled in the deno executable so that users usually do not need to provide it
• resolve.js cannot use any import statement as there is no module resolution at that time and must be a javascript file
• Any deno command line argument is visible in the resolve.js file - so that we can pass e.g. typescript arguments into it
• resolve.js has a single default export that is the resolveModule function

authorize.js:

• authorize.js can be a typescript file and contain import statements
• all deno command line arguments are visible in the authorize.js file
• a default authorize.js file is bundled in the Deno so that it provides the current Deno implementation
• authorize.js has a single default export that is a function which
  • grants / rejects access to privileged resources.
  • executes every time deno variable is being accessed
  • has access to context data auhtorizeContext that is defined in the user code as e.g.:

deno.auhtorizeContext = { user: 'OSV' }
deno.fs.writeFile('test.ts')

This way Deno Core remains very lightweight (without TypeScript, Prettier, cache, authorization, module resolution, remote fetch) and all these features are offloaded to external scripts that can be overwritten by the user if necessary.

[afinch7]

First a little clarification. When I referrer to "deno core" I mean the glorified v8 rust bindings(//core). Deno core is just that bindings, and doesn't include any typescript/caching support. I assume what you are referring to is what I would call deno cli(//cli) which is a implementation ontop of deno core that does include all these things.

A couple problems here:

1. Changing module resolution/loading at runtime subverts any static analysis. Setting the resolve by command line argument is the only viable way to achieve what you proposed without breaking existing security guarantees.
2. Disconnecting the compiler from deno cli changes the context of any type safety guarantee that previously existed(in a way that I'm not really a fan of).
3. It's very difficult to maintain a cohesive ecosystem when everyone is using a different compiler/loader/resolver.(You really can't avoid this one no matter how you stack it)
4. Deno is already a "simple browser compatible JS runtime" it is in no way incompatible with JS.

We have discussed custom compilers before, and I think the basic requirements boil down to:

1. It must be statically analyzable(A full dependency tree can be built without running any code).
2. We need to be able to define what compiler to use on a per source file basis. Though there maybe room for setting a default for a given file extension.
3. The compiler should be untrusted in the sense that it shouldn't be trusted to resolve a module and load it's source.
4. If we consider typescript to be the native language of deno, then we should allow for custom compilers to emit typescript or js + types.

If what you want is the ability to customize a javascript runtime(more easily than embedding v8), I'm working on a lib for embedding deno core in deno cli. I think that would be a better approach to achieve a customized runtime implemented in javascript/typescript. I have a working version, but it only works with a rebase of #2385 ontop of #2612. I hope to be able to bring my lib to deno_std in the future when the necessary prerequisites exist in released version of deno cli.

[oldrich-s]

I was thinking about it and what I value most about deno is that it aims to strictly follow browser compatibility. The way I proposed the custom module resolution is in direct conflict with how browsers do it and therefore I take it back :).

But how about to implement service workers. With service workers people can intercept any request and do their own module resolution, compilation (typescript, flow, prettier) and caching. An example use:

main.js:

await navigator.serviceWorker.register('service-worker.js')
await import('./my-file.ts')

service-worker.js

self.addEventListener('install', () => self.skipWaiting())

self.addEventListener('fetch', event => {
  const responsePromise = doSomeModuleResolutionAndCompilationAndCachingMagic(event)
  event.respondWith(responsePromise)
})

In the above example, the my-file.ts request and all the subsequent requests are intercepted by the service worker.

Even though I love typescript I see it as breaking browser compatibility. E.g. browser does not care about import file extensions but it is a crucial information for Deno to distinguish between javascript files and typescript files. So what would work in browser will not work in Deno. To assure browser compatibility, there should probably be an option in Deno to deactivate typescript. You could also consider moving the typescript compilation into a default Deno service worker. That way it would be possible to overwrite the default service worker by providing a custom service worker.

[kitsonk]

The compiler is currently implemented as a web worker, though it isn't exposed or started via the main runtime isolate.

Service Workers are a specific type of web worker geared for proxying network requests. Not really designed for what the compiler does/needs to do.

[ry]

Even though I love typescript I see it as breaking browser compatibility. E.g. browser does not care about import file extensions but it is a crucial information for Deno to distinguish between javascript files and typescript files. So what would work in browser will not work in Deno. To assure browser compatibility, there should probably be an option in Deno to deactivate typescript.

We define browser compatibility this way in the manual:

Browser compatible: The subset of Deno programs which are written completely in JavaScript and do not use the global Deno namespace (or feature test for it), ought to also be able to be run in a modern web browser without change.