New Public API

[Razican]

Related to #440.

Current situation
The current public API of Boa, that can be found here for Boa 0.8.0 and can be seen offline running cargo doc --open for the master branch, is far from optimal. We have 3 global functions:

• exec(): That creates a new lexer, parser, Realm and interpreter, and isolates the execution of a given JavaScript string, and return a string with the result.
• forward(): That will receive an already initialized interpreter and a JavaScript string and will lex, parse and execute this code in the initialized interpreter, and return the result as a string.
• forward_val(): That will work the same way as forward() but will return a ResultValue, which will indicate if the execution was successful or not.

Then, we have the following modules public:

• syntax: which includes:
  • ast: the AST implementation (nodes, constants, keywords, builtin types, positions, spans, tokens, operations, punctuators...), with some re-exports to make life easier.
  • lexer: The tokenizer/lexer implementation, that turns a string into a vector of tokens. (this is changing in Started with the new lexer implementation #432)
  • parser: The parser implementation, with its error and a couple of parsers (Script and ScriptBody), which turns a vector of tokens into a StatementList, that can then be executed with an interpreter.
• realm, which includes the implementation of a Realm, an interesting helper function to register a global function, and makes public the global object, environment, and the lexical environment. This in theory allows anyone to add stuff to the global environment, but it's a bit cumbersome.
• builtins: Where we have the builtin objects such as Array, Number, JSON and more, and also the implementations of Value and object properties. And the initialization function.
• environment, which has modules for each of the environments and the global trait, everything exposed because the realm has public members so that they can be modified.
• exec: We only contain the Interpreter, with just two public functions: new() and to_string(). The first creates a new interpreter with a given Realm, and the second one converts a Value to a string. It also contains the Executable trait, that allows calling run() on any AST node passing the interpreter so that it runs it.

Finally, we also have some re-exports to make things easier to use.

Problem description
When a user wants to use Boa for their projects, I can imagine the following use cases:

• They want to interpret JavaScript from a file or a stored String and get the result.
• They want to interpret JavaScript that is coming from a socket with a Read interface and get the result by streaming the bytes.
• They want to extend our engine by providing new global functions and objects, so that with the JavaScript they can either perform more rich things in their setup (for example by providing an API for a browser window), or because they want to add JavaScript helpers for common actions in their setup. This also includes things as being the scripting language of a complex platform.
• They want to change the global object, for example, because they want it to be the window or a frame.

I cannot think of anything else, but there might be others. In this context, our current API is very poor. It does allow most of this (except for the Read interface, that is being worked on in #432). But, it is very difficult to do it if you are not an expert in Boa.

Proposed solution
I think we should go for a global Boa structure. This structure should have a default() function (implementing the default() trait, that would setup a clean environment for the execution. Then a interpret() function that would receive either a &str or a u8 (Read) stream (we might need two functions) and return a ResultValue. We could then call to_string() on this if we needed to show it, for example, or on the Ok/Err variants.

It should also have a new() or with_global_obj() function that would require a custom global object. Note that we don't need to expose all the Gc stuff here, as we would be owning the global object.

One option is for it to receive the full Realm, but I think it's best if we didn't expose the complexity of a Realm. In any case, that global object should be wrapped, to provide an easy to use API:

• It should have a register_global_object() function, that would just receive a name and a NativeObject. The current register_global_func() should be kept (maybe renamining to register_global_function().

The NativeObject should just implement a trait, and should easily be boxable, or we could use dynamic dispatch. This trait should allow adding native methods to the object, and should implement a way of retrieving and setting fields. I want to be vague about this, because #419 should give us a clear path.

So, what do you think? What could we improve?

[jasonwilliams]

One option is for it to receive the full Realm, but I think it's best if we didn't expose the complexity of a Realm. In any case, that global object should be wrapped, to provide an easy to use API:

yeah we don't need to expose that really.
For ergonomics we should make it easy to add your own native function to the global object.

Are there any existing standards/interfaces we should be following?

[HalidOdat]

Are there any existing standards/interfaces we should be following?

Speaking of standard/interfaces we should follow the Rust API Guidelines

[Razican]

Are there any existing standards/interfaces we should be following?

Not that I know of. We could check how V8 and SpiderMonkey interface with browsers, for example.

[HalidOdat]

Not that I know of. We could check how V8 and SpiderMonkey interface with browsers, for example.

Neon crate has a nice API. that's were i got the idea of ctx.throw_range_error(msg)

The ctx.throw_range_error(msg) and ctx.throw_type_error(msg) should be public to the user

[HalidOdat]

In the future we might want to have some kind of procedural macro to make it easy to interoperate from native to javascript

wasm-bindgen has this and it's awesome, for example:

#[wasm_bindgen]
#[derive(Copy, Clone)]
pub struct Answer(u32);

#[wasm_bindgen]
impl Answer {
    pub fn new() -> Answer {
        Answer(41)
    }
    #[wasm_bindgen(getter)]
    pub fn the_answer(self) -> u32 {
        self.0 + 1
    }
    pub fn foo(self) -> u32 {
        self.0 + 1
    }
}

We could have something like this, instead of wasm_bindgen we could have to be boa, for example:

#[boa]
#[derive(Copy, Clone)]
pub struct Answer(u32);

#[boa]
impl Answer {
    pub fn new() -> Answer { // this will be ignored
        Answer(Self::helper_function())
    }

	fn helper_function() -> u32 { // this too
		42
	}

    #[boa(name = "theAnswer", getter)] // we could have the proc macro convert the name of the function to camelCase implicitly, so we don't have to supply it, when we don't want to rename it.
    pub fn the_answer(&mut self, args: &[Value], ctx: &mut Interpreter) -> u32 {
        self.0 + 1
    }
	
	#[boa(name = "bar", prototype)] // Reaname function to 'bar' and add it to the prototype.
    pub fn foo(&mut self, args: &[Value], ctx: &mut Interpreter) -> u32 {
        self.0 + 1
    }
}

And the boa proc macro would automatically generate an init function, so the user only has to call it (Answer::init(global)).

For the function:

#[boa(name = "theAnswer", getter)]
pub fn the_answer(&mut self, args: &[Value], ctx: &mut Interpreter) -> u32 {
    self.0 + 1
}

Where it takes Self instead of Value we could have proc macro downcast it to Self and use it as if it was a native type, this could be done with internal state so Answer would require InternalState trait, we could also implement this with trough boa proc marco.

This is for objects and methods but we can have something similar for functions.

BTW: I'm just brainstorming.

What do you think?

[Razican]

What do you think?

This would be awesome. I would automatically rename to camelCase, though, since that's the usual pattern in JavaScript, and maybe allow to specify the name if needed.

[jasonwilliams]

It should have a register_global_object() function, that would just receive a name and a NativeObject. The current register_global_func() should be kept (maybe renamining to register_global_function().

This might need to be register_global_property(), as global object usually refers to the top level global object itself window or globalThis etc.

I like how in the random crate you have an option to just use it directly without much setup https://rust-random.github.io/book/guide-start.html i think using Default() helps with this, we should be able to get users up and running pretty quickly.

Whats a simple use case? You pass in a string (or path to file) then get a value back.
we could have a fast-path function which gives you back a ValueData enum.

Then a interpret() function that would receive either a &str or a u8 (Read) stream (we might need two functions) and return a ResultValue.

I think fromString() and fromFile() would be the most explicit and easiest to understand right?

How much of this do we want to follow:
https://tc39.es/ecma262/#sec-execution-contexts

Embedding V8 example
For some inspiration this is V8
https://chromium.googlesource.com/v8/v8/+/branch-heads/5.8/samples/hello-world.cc

[HalidOdat]

Here is V8 API https://v8.dev/docs/embed (it also has some examples)

[jasonwilliams]

#461 implements Default for Realm which right now includes all the builtins.

Here's a Q
Should the Default Realm be empty or have all the builtIns Boa provides?
I don't see why anyone would want the empty realm.

[Razican]

Here's a Q
Should the Default Realm be empty or have all the builtIns Boa provides?
I don't see why anyone would want the empty realm.

I think it should have all the builtins, since that should be the "default" behaviour. We can have a Realm::empty() method, that creates an empty Realm.

[tommyZZM]

if I wants to embed boa into some native application.

I wants more featured API. so that i can call rust code from javascript and return javascript value from rust code.

maybe like nodejs's napi, https://nodejs.org/api/n-api.html

[Razican]

I wants more featured API. so that i can call rust code from javascript and return javascript value from rust code.

You currently can do this, if I'm not mistaken, but it's a bit difficult to make it happen.

[HalidOdat]

I wants more featured API. so that i can call rust code from javascript and return javascript value from rust code.

You currently can do this, if I'm not mistaken, but it's a bit difficult to make it happen.

Yes. you can do this, I have been looking at making this a bit nicer, how mlua/rlua/quickjs-rs handles functions, something like:

context.register_function("sayHello", |context, (name, lastname, age): (String, String, u8)| {
	println!("Hello, {} {}", name, lastname);
	// do something with age
	Ok(Value::undefined())
});

Right now we pass the interpreter as the context, but this is not what we want, the interpreter is only a implementation detail, we might remove the Interpreter to switch to a Vm design which is a better and faster option. so we wrap the implementation detail e.g the Interpreter, this should make switching Executors designs in the future not breaking changes and easier to implement as well something like:

struct Context<'a> {
	executor: &'a mut Interpreter,
	// maybe other fields ...
}

We could also have different contexts for different things CallContext for function call contexts:

struct CallContext<'a> {
	context: Context<'a>, 
	arguments: &'a [Value],
	this: &'a Value,
}

Something similar to neon crate Contexts

There will be a trait maybe called FromValue and ToValue that take a context and return Self.
And two other traits FromValueMulti and ToValueMulti that is implemented on tupels so we can do (name, lastname, age): (String, String, u8) what we did with the closure, we could also implement FromValueMulti on a single T: FromValue

pub trait FromValue {
	fn from_value(&self, context: Context) -> Result<Self>;
}

pub trait ToValue {
	fn to_value(&self, context: Context) -> Result<Value>;
}

so we could so something like:

context.register_function("sayHello", |context, name: String| {
	println!("Hello, {)", name);
	Ok(Value::undefined())
});

These methods of passing arguments enables us to have in a way variadic parameters.

We can pass also a function it does not have to be a closure:

// in boa have a `Result` type def
// in sre/lib.rs
type Result<T> = Result<T, Value>;

/// user code
fn say_hello(_: CallContext, (name, lastname, age): (String, String, u8)) -> boa::Result<()> {
	Ok(())
}
context.register_function("sayHello", say_hello);

Questions that need answers for the API:

• FromValue of () unit should be null or undefined? At first it seems null but at second glance in rust unit is returned in every function that has an explicit () return or no return value provided and in js when you don't return anything undefined is return, so maybe FromValue of () should be undefined?
• What happens if the amount of arguments is not suplied should it throw an exception? Or do we force Default trait on FromValue: Default so we can default construct the rust type that implememts FromValue
• What happens if the type is not convertable to the type T: FromValue do we throw an exception, or default construct it? I would say throw an exception.

Right now these are the only thing that popped into my mind, but there are probably more.

What do you think?

[Razican]

Questions that need answers for the API:

* `FromValue` of `()` unit should be `null` or `undefined`? At first it seems `null` but at second glance in rust unit is returned in every function that has an explicit `()` return or no return value provided and in js when you don't return anything `undefined` is return, so maybe `FromValue` of `()` should be `undefined`?

I would say it should be undefined. If I develop a Rust function that is callable from JS, I don't expect my Rust function to return anything valuable if I'm not returning it on purpose. I would return a Value::null() if I really need to return a null value.

* What happens if the amount of arguments is not suplied should it throw an exception? Or do we force `Default` trait on `FromValue: Default` so we can default construct the rust type that implememts `FromValue`

Good question. I would just follow whatever happens in JS when you add more arguments to any function. So for example, the new Date() constructor will receive at most 7 parameters, but if I call it like this:

var d = new Date(10, 10, 10, 10, 10, 10, 10, 10, 10, 10, 10)

Then the rest of the parameters are just ignored. So I think we should just ignore them too. There is one especial case. What if we want the Rust function to receive a variable number of arguments? In that case, I would say we need a way of calling that function with a slice of the argument list.

So, all Rust functions should have a signature such as:

fn say_hello(_context: CallContext, _args: &[Value]) -> boa::Result<()> {
	Ok(())
}

We can even create a type (NativeFunction) or something like this and expose it, so that it's easy to handle. Then, we should be able to easily transform those values to Rust with the context at hand.

* What happens if the type is not convertable to the type `T: FromValue` do we throw an exception, or default construct it? I would say throw an exception.

I think this should be solved with the idea above. There is no way in Rust to specify a variable number of arguments, so we will need to make all Rust functions accept a slice of arguments, and then the function itself can check if the number and type of arguments is correct.

Right now these are the only thing that popped into my mind, but there are probably more.

What do you think?

Overal, I think this is the way to go, and it looks easy to use. I don't think we can just use any Rust function from JavaScript, but I do think we can allow users to create wrappers with lists of arguments. If we had to create a default wrapper, I would just ignore extra arguments and give a TypeError when encountering a type that is not correct, but I do think that many of the Boa users will want to create functions with variable number of arguments, and I do think it's more spec-friendly.

Even if we provide a default wrapper for most native Rust functions, I still think we have to expose this, to allow for customizing the behaviour in case the number or types of the arguments is not the expected one.

[HalidOdat]

Then the rest of the parameters are just ignored. So I think we should just ignore them too.

Agreed. this is the way to go if we want to match what js does, in rust.

About the variadic arguments, one way of doing is as you said:

fn say_hello(_context: CallContext, _args: &[Value]) -> boa::Result<()> {
	Ok(())
}

&[Value] would implement FromValueMulti and it should definetaly be possible do this. also there is a second way remember that I defined a CallContext as:

struct CallContext<'a> {
	context: Context<'a>, 
	arguments: &'a [Value],
	this: &'a Value,
}

and we can implement on CallContext arguments() method which returns the &[Value]:

fn say_hello(context: CallContext, _: ()) -> boa::Result<()> { // we accept an () unit which is accepting no arguments
	context.arguments() // maps the first argument to the last argument
	Ok(())
}

The equivelent in js is:

function sayHello() {
	arguments
}

This matches nicely with javascript and Rust :)

What if we want to have typed arguments with variadic?
This should be possible by specializing the parameters:

impl<A, B> FromValueMulti for (A, B, &[Value])
where
	A: FromValue,
	B: FromValue,
{
	// ...
}

so we could do:

fn say_hello(context: CallContext, (_, _, _): (String, u32, &[Value])) -> boa::Result<()> {
	Ok(())
}

we would have more overloads no just (A, B, &[Value]), the implementation will include some macro magic so we don't have to write them by hand.
The &[Value] will be empty if only two arguments were supplied and I don't think it should throw it two or more are supplied. Also if we want to access the first arguments to the last one we use context.arguments(), if we want a continuation we use the &[Value] in (_, _, &[Value]) which is offset by the number of arguments that precede it.

also FromValueMulti will have a method maybe called len/length:

pub trait FromValueMulti {
	// ...

	// Return the number of arguments this function accepts
	fn length() -> usize;
}

so for (A, B, C) it should be 3 for (A, B, &[Value]) it should be 2 the number of expected values not including the catch all &[Value]. This way allows us to know the length of a function without specifying manually.

Also forgot to mention if Value is passed as a parameter (String, Value, &[Value]) it is any value.

[jcdickinson]

FWIW the current internal API is pretty awesome. I've used the ChakraCore integration API in the past, which is designed to be usable for integration (unlike V8), and Boa feels way more ergonomic than even that. The effort involved for a large public interface (Date) was minimal. Additional abstractions (e.g. #[boa], FromValueMulti) could then operate against this raw public API.

IMO simply adding a public set_data method (backed by ObjectData::Any<std::any::Any>) would go a long way.

[Razican]

we would have more overloads no just (A, B, &[Value]), the implementation will include some macro magic so we don't have to write them by hand.

My experience with overloading stuff is that you have to decide where to stop, and it will increase compile times a lot (I'm looking at you, diesel). Do we need all this type checking?

It's not easy to define a good and only way of going from Rust values to JS values and vice-versa. We can just use &[Value] (probably on the context is good), and let the users deal with value conversions.

IMO simply adding a public set_data method (backed by ObjectData::Any<std::any::Any>) would go a long way.

What would this do? We were talking about adding functions, but something similar could be done to add new objects.

[HalidOdat]

My experience with overloading stuff is that you have to decide where to stop, and it will increase compile times a lot (I'm looking at you, diesel). Do we need all this type checking?

It's not easy to define a good and only way of going from Rust values to JS values and vice-versa. We can just use &[Value] (probably on the context is good), and let the users deal with value conversions.

If I was a user of Boa I would love to have this abstraction, I have used mlua and I love it, it's so easy to get started with minimal setup. while the compilation time will increase by a small amount, I think it's worth having this abstraction. We can have a compromise for this we can put it in a feature flag (FromValue, ToValue, FromValueMulti, ToValueMulti, and all the overload, we can put these in a file and use #[cfg(feature = "name")], this way its the same as #[cfg(test)]), like boa attribte flag for #[boa] if a user does not want to have this abstraction they can disable it and use the raw API.

What do you think? @Razican

[Razican]

My experience with overloading stuff is that you have to decide where to stop, and it will increase compile times a lot (I'm looking at you, diesel). Do we need all this type checking?
It's not easy to define a good and only way of going from Rust values to JS values and vice-versa. We can just use &[Value] (probably on the context is good), and let the users deal with value conversions.

If I was a user of Boa I would love to have this abstraction, I have used mlua and I love it, it's so easy to get started with minimal setup. while the compilation time will increase by a small amount, I think it's worth having this abstraction. We can have a compromise for this we can put it in a feature flag (FromValue, ToValue, FromValueMulti, ToValueMulti, and all the overload, we can put these in a file and use #[cfg(feature = "name")], this way its the same as #[cfg(test)]), like boa attribte flag for #[boa] if a user does not want to have this abstraction they can disable it and use the raw API.

What do you think? @Razican

I like the idea of having it behind a feature flag.

[HalidOdat]

We have the Value::to_unit32 and Value::to_int32() that convert the value to a u32 or i32 should we rename it to Value::to_u32 and Value::to_i32? to match with Rusts u32 and i32 convention or stick to the specs naming. These are rust functions, so IMO we should rename them.

What is your opinion?

[Razican]

We have the Value::to_unit32 and Value::to_int32() that convert the value to a u32 or i32 should we rename it to Value::to_u32 and Value::to_i32? to match with Rusts u32 and i32 convention or stick to the specs naming. These are rust functions, so IMO we should rename them.

What is your opinion?

Yes, I think we should stay close to Rust naming conventions.