Is it possible to publish common modules as third-party packages, such as rquickjs-ext similar to deno?

[ahaoboy]

Some common modules such as os, path, fs, etc., which are not closely coupled with the runtime, can they be extracted as separate third-party libraries? For example, rquickjs-ext, so that other projects that only use rquickjs can also use it

[richarddavison]

It probably can! We "semi-started" to support this by moving out runtime specific code to its own crate. Right now there are a couple of blockers tho:

1 There is some lambda specific features (currently gated) that reside in llrt_core
2 The library depends on some JS bundled code to be present (for things like ESM), this should be moved and built into core
3 The core module will at compile time create a perfect hashmap and embed modules into its own executable.

I experimented with a different approach for 3 by appending a special data format that contained an index of all embedded files + the raw bytecode at the end of the executable. This proved to slightly increase cold start even though it made the embedding more flexible so was ruled out.

[Sytten]

Hey I am pitching in since I started working on a utils crate for rquickjs (DelSkayn/rquickjs#319). The goal is to eventually recreate all the api we expect from node/deno for quickjs.

I would interested to help in whatever implementation. @richarddavison do you guys have a chat system so we can talk about that?

[richarddavison]

Hey I am pitching in since I started working on a utils crate for rquickjs (DelSkayn/rquickjs#319). The goal is to eventually recreate all the api we expect from node/deno for quickjs.

For sure! ⭐️

I would interested to help in whatever implementation. @richarddavison do you guys have a chat system so we can talk about that?

I just dropped you an email!

[gc-victor]

It would be great to know if there is any update about this 🙏

[ahaoboy]

Some problems I encountered during the development process, I hope the new module mechanism can improve

1. Node protocol support, that is, support for two module protocols at the same time

import fs from 'fs'

import fs from 'node:fs'

2. Module dependency, for example, fetch depends on the net module. Currently, we can only manually ensure the import order of the modules. Is it possible to implement a mechanism that reports an error or automatically imports the net module if only fetch is imported?

[richarddavison]

Some problems I encountered during the development process, I hope the new module mechanism can improve

1. Node protocol support, that is, support for two module protocols at the same time

import fs from 'fs'

import fs from 'node:fs'

2. Module dependency, for example, fetch depends on the net module. Currently, we can only manually ensure the import order of the modules. Is it possible to implement a mechanism that reports an error or automatically imports the net module if only fetch is imported?

Not quite sure I follow. Import order, does not matter.
For node protocol support we simply strip the "node:" prefix from import specifier.

[ahaoboy]

For node protocol support we simply strip the "node:" prefix from import specifier.

Will you consider supporting require and submodules? Many node libraries have syntax similar to the following. I don't know if there is a bundle tool that can handle it.

const { isArrayBuffer } = require('node:util/types')

[richarddavison]

For node protocol support we simply strip the "node:" prefix from import specifier.

Will you consider supporting require and submodules? Many node libraries have syntax similar to the following. I don't know if there is a bundle tool that can handle it.

const { isArrayBuffer } = require('node:util/types')

This would work if there was a a util/types.mjs present on the filesystem. For embedded and native modules we have to explicitly define the paths.

Require is already supported. You can also mix and match with imports in the same file

[richarddavison]

It would be great to know if there is any update about this 🙏

Not yet, but PR is welcome. @KaanMol maybe you want to take a stab at this? The "only thing" missing right now is that we need to embed some JS modules that is required, for instance:
https://github.com/awslabs/llrt/blob/main/llrt_core/src/modules/js/%40llrt/std.ts
and https://github.com/awslabs/llrt/blob/main/llrt_core/src/modules/js/%40llrt/init.ts

There is also some AWS specific stuff in there that should not be part of the core package. I've experimented with a different approach to embed modules into the executable by appending the bytecode to the end. It works and make building a bit simpler. However, it comes with a tiny performance cost so we might need to stick with this until I've found a better way or removed that overhead.

[KaanMol]

It would be great to know if there is any update about this 🙏

Not yet, but PR is welcome. @KaanMol maybe you want to take a stab at this? The "only thing" missing right now is that we need to embed some JS modules that is required, for instance: https://github.com/awslabs/llrt/blob/main/llrt_core/src/modules/js/%40llrt/std.ts and https://github.com/awslabs/llrt/blob/main/llrt_core/src/modules/js/%40llrt/init.ts

There is also some AWS specific stuff in there that should not be part of the core package. I've experimented with a different approach to embed modules into the executable by appending the bytecode to the end. It works and make building a bit simpler. However, it comes with a tiny performance cost so we might need to stick with this until I've found a better way or removed that overhead.

I wonder if making the core modules AWS stuff free is not a blocking piece before continuing. I could take a look next week :)

[KaanMol]

@richarddavison would it be possible I could also get an invite for the chat system?

[Sytten]

I will post it here since I am in charge of it.
Here is the discord: https://discord.com/invite/pyMppMngYU

[Sytten]

One thing I wanted to mention is that modules should be more independent than they are right now. For example the URL module cant be evaluated at the moment if the http module init is not called first because some stuff like URL constructor is put in the globals in the http init and taken back in the module eval.

[KaanMol]

One thing I wanted to mention is that modules should be more independent than they are right now. For example the URL module cant be evaluated at the moment if the http module init is not called first because some stuff like URL constructor is put in the globals in the http init and taken back in the module eval.

They share a lot of code and components, so making everything completely standalone from the start might not be the best approach. Instead, it would be better to develop something that improves over time. It may not be ideal to have everything as separate crates like Deno does. This raises the question of how standalone "standalone" should be.

Most modules consist largely of shared code or are very small. It would make more sense to have a crate like "llrt_modules" with feature flags, to be honest.