Making LLRT available as a crate

[KaanMol]

Issue #319 (if available)

closes #319

Description of changes

Currently it is pretty "ok", some naming etc is made up on the fly, like the package names or like module_builder.

Splitting AWS parts out to its own package is possibly a better option to pick up in a second PR, since this PR is getting a bit on the big side.

Tasks that might be picked up with this PR:

• Moving AWS specific parts out of the core of LLRT;

Checklist

• Created unit tests in tests/unit and/or in Rust for my feature if needed
• Ran make fix to format JS and apply Clippy auto fixes
• Made sure my code didn't add any additional warnings: make check
• Updated documentation if needed (API.md/README.md/Other)

By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

[KaanMol]

@richarddavison, could you take a look and let me know if it aligns with what you had in mind? It is more of a proof of concept. I dont like how I did the module declaration and also the bind to global part, however at the same time no other solution comes to my mind.

[richarddavison]

I like this approach! Now third party projects can pull in a lightweight runtime and use the modules they wan't. Unused modules would then be automatically removed at compile time. Just some minor changes regarding module builder

[KaanMol]

@richarddavison I had another problem, how would we allow passing values from LLRT to Rust context? Do you maybe have an idea? Deno has a Scope for example

[richarddavison]

@richarddavison I had another problem, how would we allow passing values from LLRT to Rust context? Do you maybe have an idea? Deno has a Scope for example

Can you expand a bit what you mean pass values? I'm not familiar with Scope in Deno, can you send me documentation?

[KaanMol]

@richarddavison I had another problem, how would we allow passing values from LLRT to Rust context? Do you maybe have an idea? Deno has a Scope for example

Can you expand a bit what you mean pass values? I'm not familiar with Scope in Deno, can you send me documentation?

Their documentation is not the best, they dont have their runtime really documented.
I haven't really looked at it in dept, but maybe you have an idea how one would do this.

How could one start the runtime maybe pass the JS function some params from Rust and also get the data returned in the JS function to do something with it after the execution?

[richarddavison]

@richarddavison I had another problem, how would we allow passing values from LLRT to Rust context? Do you maybe have an idea? Deno has a Scope for example

Can you expand a bit what you mean pass values? I'm not familiar with Scope in Deno, can you send me documentation?

Their documentation is not the best, they dont have their runtime really documented. I haven't really looked at it in dept, but maybe you have an idea how one would do this.

How could one start the runtime maybe pass the JS function some params from Rust and also get the data returned in the JS function to do something with it after the execution?

This is very simple, for example call JS from Rust:

globals.anyJsObjectOnGlobalScope = (arg) => {
  return arg;
}

let global_js_fn : Function = ctx.globals().get("anyJsObjectOnGlobalScope")?;
let result_string: String = global_js_fn.call(("hello",))?;

Or call Rust from JS:

rustFunction("hi")

ctx.globals().set("rustFunction",Func::from(|value:String|{
   //returns same value
   value
})

[KaanMol]

@richarddavison I had another problem, how would we allow passing values from LLRT to Rust context? Do you maybe have an idea? Deno has a Scope for example

Can you expand a bit what you mean pass values? I'm not familiar with Scope in Deno, can you send me documentation?

Their documentation is not the best, they dont have their runtime really documented. I haven't really looked at it in dept, but maybe you have an idea how one would do this.
How could one start the runtime maybe pass the JS function some params from Rust and also get the data returned in the JS function to do something with it after the execution?

This is very simple, for example call JS from Rust:

globals.anyJsObjectOnGlobalScope = (arg) => {
  return arg;
}

let global_js_fn : Function = ctx.globals().get("anyJsObjectOnGlobalScope")?;
let result_string: String = global_js_fn.call(("hello",))?;

Or call Rust from JS:

rustFunction("hi")

ctx.globals().set("rustFunction",Func::from(|value:String|{
   //returns same value
   value
})

Oh thats amazing!

[KaanMol]

@richarddavison I just implemented LLRT as a test in my own project, it brought it back from around 2/3 sec with Deno to around 100ms, I also ran LLRT's benchmarks and didn't notice a change before and after these changes :)

[richarddavison]

@richarddavison I just implemented LLRT as a test in my own project, it brought it back from around 2/3 sec with Deno to around 100ms, I also ran LLRT's benchmarks and didn't notice a change before and after these changes :)

Have you thought about moving out the Lambda specific implementations we do? For instance the console contains a lot of Lambda specific code, even at its core (for example treating newlines). I suspect the best thing right now would be to feature gate it. Another option could be to pass a config and use a closure for the exposed functions so we can pass config values down the the console argument handling (for example line endings, output etc).

[KaanMol]

@richarddavison I just implemented LLRT as a test in my own project, it brought it back from around 2/3 sec with Deno to around 100ms, I also ran LLRT's benchmarks and didn't notice a change before and after these changes :)

Have you thought about moving out the Lambda specific implementations we do? For instance the console contains a lot of Lambda specific code, even at its core (for example treating newlines). I suspect the best thing right now would be to feature gate it. Another option could be to pass a config and use a closure for the exposed functions so we can pass config values down the the console argument handling (for example line endings, output etc).

Yes, like I mentioned previously, I thought of making console utils and implementing the module within LLRT while creating a second module, but in LLRT-lambda or some other name. The thing I actually find complicated is that some JS gets compiled for improved start-up times, and it also includes AWS libraries. Do you know if it would be easy to split that?

[richarddavison]

Yes, like I mentioned previously, I thought of making console utils and implementing the module within LLRT while creating a second module, but in LLRT-lambda or some other name. The thing I actually find complicated is that some JS gets compiled for improved start-up times, and it also includes AWS libraries. Do you know if it would be easy to split that?

Yes, that can be easily removed by feature gate or in this case remove that logic from the module loader and resolver:

llrt/src/vm.rs

Line 225 in bfaa41e

impl Resolver for BinaryResolver {

llrt/src/vm.rs

Line 346 in bfaa41e

impl Loader for BinaryLoader {

Right now the binary resolver does some other things as well so that can complexity probably be refactored a bit.

[KaanMol]

Yes, like I mentioned previously, I thought of making console utils and implementing the module within LLRT while creating a second module, but in LLRT-lambda or some other name. The thing I actually find complicated is that some JS gets compiled for improved start-up times, and it also includes AWS libraries. Do you know if it would be easy to split that?

Yes, that can be easily removed by feature gate or in this case remove that logic from the module loader and resolver:

llrt/src/vm.rs

Line 225 in bfaa41e

impl Resolver for BinaryResolver {

llrt/src/vm.rs

Line 346 in bfaa41e

impl Loader for BinaryLoader {

Right now the binary resolver does some other things as well so that can complexity probably be refactored a bit.

So I wonder if it wouldn't be a better idea to start splitting this PR up in multiple parts. It already got very big.
Right now I have split LLRT up in 2 parts, one of them being the core and one is the executable itself.

How would you feel if we set the scope for this PR only to what it is now, where LLRT is 2 parts with a module selection etc. And after merging I would start splitting the AWS parts away. At least if a big refactor happens somewhere, I would not have to somehow merge that in this newer structure every time.

[KaanMol]

@richarddavison Could you review this once you have time? :)
Splitting the AWS parts is something I am going to do in its own PR

[richarddavison]

Minor nit, but LGTM!

[KaanMol]

Minor nit, but LGTM!

Should be done now! The failing test should (hopefully) also be addressed, make test-ci now works as well locally.

[richarddavison]

@KaanMol can you please resolve the last conflict, then I'll merge! I think we're good now!

[KaanMol]

@richarddavison Done!