Scripting API

[afonso360]

This issue is meant to consolidate the discussion around the Scripting API.

It might be a bit early for an actual implementation to come out of this, however, starting a discussion around API design and goals should provide benefits when impl time comes.

On a discussion in gitter we threw around some ideas

@torkleyy mentioned he would like to have an alternative to the classic scripting approach. One of the ideas proposed was to have flexible scripting files that could be used as config files.

I already toyed with an idea for an alternative to the classic scripting approach. I wondered if we could just have pretty flexible config files, like we plan already for entities and other stuff of the engine. I mean, this does not conflict with scripting, but I think it would be worth to experiment with it, because looking at most scripts, these could be easily transformed into a YAML/JSON/whatever file. Additionally, these files would nicely integrate into GUI (you could probably generate a GUI dialog by some scheme) and tooling, the user is able to define custom ones (like configs for items, weapons, levels, ...). I still have to think about this idea some more, but I think it'd be worth trying it out. That said, I'm open for other ideas or for hearing situations in which you would additionally want a scripting language. I'm going to write up the advantages of this approach in a few days.

With an example

// some_script.pseudo

class MySuperItem extends Item {
    fn getTex() = "items/super_item";
    fn getBlah() = "Blah";
}

fn on_init() {
    register_item(MySuperItem);
}

Or, as config:

# super_item.yml
name: MySuperItem
tex: items/super_item
blah: Blah

---

@Xaeroxe and @lschmierer Discussed some ideas involving accesing components from a scripting language. Briefly mentioned is the feasablity of running the scripting system in parallel as an actual System.
However this seems impractial due to the way specs enforces SystemData correctness on a type level.

---

There are a lot of things that need to be addressed, however, I think we should start with:

• Are we going for a traditional scripting api, or with something newer?
• How important is it getting the scripts to run in parallel?

[torkleyy]

Thanks for the summary! The point about SystemData is not true, there is no restriction on Specs' side.

After the renderer rewrite is merged I'm going to start a game; it may be useful to have a game at hand to evaluate which parts of it could be written more efficiently.

[minecrawler]

Just want to throw in a scripting language I considered, but threw away, because it is quite dead at the moment. I might be a better start than creating a new one from scratch, though: https://github.com/jonathandturner/rhai

• Rhai is based on Rust and JS (two languages I love)
• Rhai is pure Rust code, no unsafes
• Rhai can use generic functions
• Rhai can use strict Rust types

Examples:

use std::fmt::Display;

extern crate rhai;
use rhai::{Engine, FnRegister};

fn showit<T: Display>(x: &mut T) -> () {
    println!("{}", x)
}

fn main() {
    let mut engine = Engine::new();

    engine.register_fn("print", showit as fn(x: &mut i64)->());
    engine.register_fn("print", showit as fn(x: &mut bool)->());
    engine.register_fn("print", showit as fn(x: &mut String)->());
}

extern crate rhai;
use rhai::{Engine, FnRegister};

#[derive(Clone)]
struct TestStruct {
    x: i64
}

impl TestStruct {
    fn update(&mut self) {
        self.x += 1000;
    }

    fn new() -> TestStruct {
        TestStruct { x: 1 }
    }
}

fn showit<T: Display>(x: &mut T) -> () {
    println!("{}", x)
}

fn main() {
    let mut engine = Engine::new();

    engine.register_type::<TestStruct>();

    engine.register_fn("update", TestStruct::update);
    engine.register_fn("new_ts", TestStruct::new);

    if let Ok(result) = engine.eval::<TestStruct>("var x = new_ts(); x.update(); x") {
        println!("result: {}", result.x); // prints 1001
    }
}

[afonso360]

I've been thinking about choosing a scripting language, however i think we should be able to do something similar to what gfx does with backends.
The amethyst engine could export a set of traits that client crates could implement, and provide bindings to the scripting language.
The user could then register the client crate at initialization time and use whichever language they would perfer.

From my point of view there are two main downsides to this approach:

• There is extra work involved, in designing the api, both for the engine, and the for individual implementations.
• It would almost certainly be slower than if we chose a single scripting language and implement bindings taking avantage of all the features in the language

[minecrawler]

@torkleyy I think, this is the same discussion as with the config format. Which formats should we allow? Why not all of them?

Because:

• the user has to choose, which in itself is tedious
• might lead to confusion in docs and tutorials and other places
• it's more work on all sides
• the user might not be able to leverage all features a script engine offers

That's why I am, again, of the opinion, that Amethyst should make a sane choice and be consistent.

[torkleyy]

If we want to decide for one, I cannot support your proposal for Rhai, however. I don't really like how it works; it's also considerably slow, which is unfortunate for using it in a game engine.

[afonso360]

dyon might be a good choice, it's very likeley that it could become a really fast language and it would probably interact with rust very well. However I'm a bit reluctant due to how young it is, and because its not wideley used its also another thing the user has to learn.

Lua is also a popular choice for game engines, it's really fast with luajit and even if the user hasn't used it before there's really not much to learn about it. Not sure how well we would be able to handle running it on multiple cores.

Javascript is another choice that might be good, with V8 its definitely fast, and much like hlua there is also a set of high level bindings available.

Out of all these ones I would support Lua the most.

---

However, I would still like to argue for allowing the user to choose their own scripting language. I haven't really looked into the config file discussion, so i might be make some points that already were debated there.

the user has to choose, which in itself is tedious

I disagree here, I would argue that when you are starting out a new project, having to choose the config file type might be a burden, because, everyhing is new and its another thing just before you can get started.

However, I don't think the user will start implementing stuff around the scripting language before they develop something and have a bit of experience with the engine.

But then again, they might just want to have a rust backend available if they need more performance and try to implement everthing on the scripting language.

might lead to confusion in docs and tutorials and other places

I'll need to have a look at the discussion about this on the config files, however, i would say that we could just separate the documentation / tutorials between the crates, so for example you could explain how to setup the bindings in the amethyst documentation, and also on the bindings crate you could have a more comprehensive set of tutorials on how everything is going to work.

it's more work on all sides
the user might not be able to leverage all features a script engine offers

I agree with both of these.

[alteous]

Here's a crazy idea: How about using a subset of Rust as a scripting language. It has the advantage of being already familiar, and able to be compiled as a dylib if interpreting is too slow.

[torkleyy]

Well, I don't really need a scripting language at all, since Rust is perfectly fine for these simple jobs, I mean it often even feels a bit like a scripting language with all the type inference and expressiveness.

But it's really hard to discuss about this without an actual game. I would really vote for postponing this issue.

[minecrawler]

Well, I don't really need a scripting language at all

Careful there! Just because you don't need it does not mean others don't either.

Rust is perfectly fine for these simple jobs

Rust is compiled, so it is no replacement for a proper scripting language. A script can be loaded and executed, but can be replaced or edited and re-loaded at runtime. Rust would have to be compiled and bindings would have to be de-/initialized, given that the dylib has a stable API (which usually is not the case during development)

But it's really hard to discuss about this without an actual game.

I don't think that the discussion about the scripting language is dependent on a game. It is much more dependent on preferences and compüatibility with engine architecture. Hence, I think we can already discuss the feature. Just as @afonso360 wrote, this is not about implementation, but about specification :)

[alteous]

subset of Rust

Here I mean a subset of the Rust grammar that may be interpreted or compiled at the user's discretion.

[OvermindDL1]

Just for some potential ideas that can be taken from this since it is not really a scripting language (although it could be used as such) but look at Terra. It is Luajit combined with LLVM. It itself is basically a set of enhancements on top of luajit that add specialized bindings to the code generator for LLVM including being able to add types to code and more. Basically it is a code generation library, you could, for example, make a scripting language 'compiler' in terra that compiles a scripting language to native fast and optimized code at run-time via LLVM, or you could even compile it out to a stand-alone dynamic link library (or static link) that you could then load 'instantly' next time.

The main thing I'm bringing up with this is that it would be really nice to have the engine 'export' the necessary primitives to be able to bind whatever scripting language we want, and things like Terra would use those exports directly, or something like a 'lua' binding would be a dynamic-link (or static link depending on how compiled) library that just creates the lua scripting engine with the proper bindings to the exports. It opens the entire gamut for what is possible when this is done.

/me uses terra, though not as a scripting language but rather as a generator for, say, noise generators to make user-generates noise functions, as one example

[omni-viral]

I agree with @torkleyy. We need real example (a game) to better understand what data we want to access from scripts, how to link it, how to run scripts.

[afonso360]

But it's really hard to discuss about this without an actual game. I would really vote for postponing this issue.

I think we can still make some headway before we get to the point where we need something specific.

How about using a subset of Rust as a scripting language

It sounds like a lot of work, unless there is already something that we can use.
The really good thing about a dynamic scripting language is that I don't have to worry about types, and that gives me a lot of flexibility. (It can also backfire horribly)

[torkleyy]

I've changed this to diff: hard because

• we can't send structs with lifetimes to a script
• how to work with resource ids is non-obvious
• the 'static restriction forces us to either duplicate all Specs types, using rced values or to send "commands" from the scripting layer to Rust (which isn't very nice)

[naturallymitchell]

If Amethyst could reuse WayCooler's Lua module (rlua IPC), that'd probably make your work easier and several projects' common code bases better.

(Aside: imho, to run code with speed, it should be a core engine function. To tell an engine how to glue together a bunch of those functions, any practical scripting/config language could make sense.)

Lua's interpreter seems very practical since it's a well-proven interpreter with well-established semantics, and rlua seems like a good wrapper with its safety measures and active development.

[torkleyy]

I'm working on this issue and I'm already using rlua. But I'll tell you more once I got some nice results.

[Marwes]

Shameless plug https://github.com/gluon-lang/gluon.

Since gluon is a statically typed language already, and it has full APIs for enforcing types between itself and Rust should be possible to integrate into specs without any limitations.

Types with lifetimes may be possible. I actually had that working way, way back but ended up removing it since it was so limited (any lifetime needed to strictly outlive the interpreter itself making it more trouble than its worth).

Thanks to some changes done in the last year it could be worthwhile to try it again however as it is now possible to create short lived child interpreters which are already have the capability to restrict short lived values from escaping to longer lived interpreters. Piggybacking on this system it should be possible to create a interpreter for each script call (the interpreter could easily be recycled to avoid allocating a fresh one each time), pass it some reference values to the script and then destroy/recycle the interpreter before the references lifetimes ends. No promises though :)

[torkleyy]

gluon defintely looks interesting :) I'll look into it, but the time I can use for OS is currently rather limited.

[ShalokShalom]

A very interesting discussion is going on here, thanks for all your contributions.

Gluon seems to get a lot of development and I can imagine that Mozilla is interested in it.
Sooner or later, such a scripting language is ideal for their growing Rust stack.

To me, it looks a lot like a primarily functional language that I love: OCaml.
The readme even mentions that it is directly inspired by it.

It is also in pure save Rust, so no unsafe code.

I love the clear syntax and like to remind us, that Amethyst is also meant for game designers.
They are often not educated in coding, so such an easy and concise syntax helps with the adoption.

Plus, it is type inferred and capable to display domain driven design very well.

Other benefits are the thread safety and of course its deep cooperation with Rust which made it look like an obvious choice.

Imagine all the capabilities that you can give the users. You can basically create an interface to every single function in the engine. This can be one of the features who turn out to set the project unique.

One more thing is that Amethyst aims to be a showcase for Rust and Gluon is written in it.

[torkleyy]

I started working on a scripting framework (rustgd/gsf) which abstracts the scripting language some time ago, but I didn't have time to improve and continue it. The idea is that libraries register their types and functions in a language-agnostic way and for each scripting language there is an intermediate layer which can read out those types (see these lines for a minimal example). I planned to replace the registration process with traits to make it more lightweight and I also want to add more languages (atm it's just rlua and I started a crate that translates this stuff into a C binding), but there are too many other things these months, so my time for it is rather limited.

[ShalokShalom]

About dynamic recompiling in/with Gluon: gluon-lang/gluon#454

[norman784]

I don't know if this discussion still going here, but I think theres a point of using a scripting language, and its attract users or maybe make thinks easier to other users (like an entry barrier) and lastly modding :), saying that I think the best option is a wide used scripting language, could it be JS, Lua, etc, I don't know OCalm (just by name) but to me feels a niche language, is not so wide used.

About something that read in the first post about config files for UI, I agree with that, maybe something like yaml like mentioned or some html-ish template (my preferred here is pug-lang/slim-lang subsets), I mean take that syntax and use with the corresponding tags for each component (i.e. UiText, UiButton, UiImage, etc and take as tag attributes the property of each ui element), and maybe add some sugar with a small subset of css for the styling or just a css like for styling using the corresponding name of the elements and their properties (then maybe will be more json like). Well I propose this because to me feels more natural because my background as webdeveloper.

But theres also another think, and it is real needed a scripting language at all (I mean for coding, not for the UI), because it may take a lot of resources to develop and maintain it, just take a look what Unity do, they drop the support for UnityScript (JS) and Boo, and they have the resources to maintain it, but maybe was not practical because the code redundancy (maintain 3 frontend langs on top of their c++ backend), docs maintenance, etc.

If this last one is the case I'm in with releasing binding tools and the community work on what they feel better for them (so amethyst core team itself doesn't need to deviate the resources to this and can focus on making this engine more awesome).

[ShalokShalom]

I think the best option is a wide used scripting language, could it be JS, Lua, etc, I don't know OCalm (just by name) but to me feels a niche language, is not so wide used.

I think that the competence, elegance and curtness of a language is also important.
Otherwise is there also no reason for Rust

Plus, Gluon is a real scripting language, written in and for Rust.
With JavaScript is it not possible to directly call Rust functions.

because it may take a lot of resources to develop and maintain it, just take a look what Unity do, they drop the support for UnityScript (JS) and Boo, and they have the resources to maintain it, but maybe was not practical because the code redundancy (maintain 3 frontend langs on top of their c++ backend), docs maintenance, etc.

Its also easier to integrate.

[omni-viral]

There is no need for particular scripting language support to be part of the engine though.
Engine can just expose scripting API. Binding that API to particular language should be done outside.

[OvermindDL1]

There is no need for particular scripting language support to be part of the engine though.
Engine can just expose scripting API. Binding that API to particular language should be done outside.

Exactly this. This is the pattern I took, and it is the pattern that engines like Godot and Urho3D and others take as well. Just expose a native interface that can be linked to and other libraries can implement the scripting language integrations.

[Moxinilian]

Taking into account all the insights in this issue, @kabergstrom and I have been working for a couple days now on a design for scripting support that would work with direct C bindings through language-specific drivers, and that would satisfy the following constraints:

• Support any language (the design should not assume a specific programming language)
• Enforce the safety of the ECS (it must be formally proven that no behavior in the scripting languages can impact the safety of safe code in the ECS)
• Close to no overhead (Rust as a scripting language should be as fast as Rust natively)
• Expandable (adding new scripting languages should be easy, even for 3rd parties, and possible without modifying the engine itself)
• Little maintenance for language drivers (adding new features to the scripting API should not require an update to the language drivers)
• Support the creation of components and resources from the scripting environment (without using Rust) including complex data structures
• Support hotswapping (components, systems and resources can be modified and created without restarting the engine)

This approach was built to be able to append any language to the engine.
We made sure it is compatible with LuaJIT, which would be an excellent backend for the first scripting language supported by the engine. We also checked that it would also be compatible with Rust as a scripting language, C, C++ and JavaScript.

We will open a RFC describing the details later. But before we do that, did we forget anything in those constraints?

[Marwes]

Despite the fact that it is extremely rough and has some hilariously unsafe code I think I should still bring up the specs/shred-gluon integration I have been sketching out. Still, it does sort of work and I haven't found any unsolvable blockers yet.

The main issue is that the support for dynamic systems in specs/shred is rather lackluster at the moment, part of it can be solved by exposing a few currently private function with regard to component extraction (which shouldn't affect the safety in any way). There might very well be possible to provide a better dynamic system abstraction in the crates themselves which could be another way of fixing it.

On the gluon side the main issue is that I have to cast away the lifetimes to pass in borrowed resources to the virtual machine. I think gluon can be improved to support that however.

https://github.com/Marwes/shred-example/blob/master/src/gluon_system.rs
https://github.com/Marwes/shred-example/blob/master/src/gluon_system.glu

[scorbeil]

Is it not possible to support an "official" scripting language (like Gluon) and an extension API for adding support for other languages? Sounds like best of both worlds to me. Folks who just can't live without language X can add language X support (or leverage the efforts of others who have already done so) and folks who are willing to use any language so long as it achieves maximum performance get the tighter integration of Gluon(?).

Forgive me if I'm missing something obvious. I'm not a game dev. Just a hacker who likes to say he contributed to a game engine, haha.

[Moxinilian]

Well if you can support all languages, no need to worry about implementation details for a specific language. The only debate is what language will serve as an example made by the core team and Amethyst contributors ;)
I suggest LuaJIT because it's fast (both to run and implement) and well known. But effort can be split, and then all languages would undergo natural selection up until one ends up being the more widely used.

[OvermindDL1]

If you wanted a sandboxed language as 'official', just use a webassembly JIT (V8 or Firefox's are both fantastic), it's safe and fast both with already a lot of languages able to target it and more coming all the time.

I'd still just vote for a native interface then implement everything on that.

[scorbeil]

Because I am not a game dev, can someone help me understand how a scripting language is typically implemented in the context of a game engine?

An example of what I mean: in a web app, if I wanted a button to react to user input, I would register a callback on that button's "onClick" method (typically in the HTML). Within that callback, I have access to a suite of functions exposed by the web browser to, for example, manipulate the DOM.

What I'm failing to wrap my head around in the game engine context is: when/where/how are the scripts executed? Every frame? Do you register callbacks like in the web example? If so, does the game dev expose the types of hooks they want available to the scripting engine? Seems like there's no "standard" set of hooks like in the web case. If I want to be able to fire off a registered callback when the player opens a door, for example, I would have to implement the "doorOpened" hook, right? Then register a callback to that hook in the script?

Or am I way off? Thanks :)

[omni-viral]

@scorbeil I think you are mostly correct.
Scripts are executed on event specified: next frame (or tick), some particular event (door open, pawn hit, button pressed), timer.

With specs you implement any of that as a System that runs each frame. If it must execute some code only when event occurs it checks the event channel or just see if state changes.

With scripts it is probably better to not check for events manually but let the scripting system do that.

[ShalokShalom]

If you wanted a sandboxed language as 'official', just use a webassembly JIT (V8 or Firefox's are both fantastic), it's safe and fast both with already a lot of languages able to target it and more coming all the time.

Thats interesting. It would also include all languages, who are able to compile into JavaScript.

[Moxinilian]

It's not unreasonable to consider WebAssembly as a scripting language itself however.
To support any language building to WebAssembly, it would only require creating a language driver for WebAssembly, and then everything would fall into place.
But I'm not sure of the practical interest if native builds can be supported anyway.

[scorbeil]

@ShalokShalom and any language that targets LLVM IR can be compiled to wasm with emscripten, yes?

[scorbeil]

That's several intermediate steps, but it should work?

[ShalokShalom]

Is there any language that can do so, which does not yet have a proper 'to-Javascript' compiler?

[scorbeil]

Not that I can think of off the top of my head, but it's nice that it's there. I really like the idea of using wasm for the scripting API. Just about any language is an option, then.

[jonathandturner]

I have to admit that my project Rhai is "dead", or at least very, very stable 😅

That said, I think there are a lot of ideas in there and the related ChaiScript that Rhai was inspired by. The original motivation for ChaiScript was that it would near-effortless to register information to an engine, and that you could write your own scripting language against this engine. ChaiScript started off as one of the sample languages you could write, but as it progressed, it quickly became the default. To this day, though, there's a separation in the engine and the language.

Rhai also has a similar distinction where the engine piece allows you to easily (hopefully) register functions and data types into the engine, and then a scripting language can dispatch off of these.

While Rhai not be alive, I think the ideas there are still sound and something that are worth exploring. I wrote Rhai when I was still learning Rust, so the code in places might need some updating, but an updated version of the engine approach could be quite helpful.

Some of the core ideas:

• As each function is registered, it is wrapped with an arity-matched function that has Any for each parmeter.
• Functions are then stored in the engine with these wrappings
• Later invocation by the scripting engine will invoke the wrapping, which does the test that each parameter matches the expected type. If it does, great, the function is invoked. If it doesn't, it lets the engine know it's not a match
• Any user-defined datatype that can be Any will work with the engine
• Rhai also allows functions with different ownerships in the first parameter to enable both mutable and immutable access to self for methods

The end result is one that doesn't use any unsafe code and can support Rust in a fairly natural way.

Not to try to hard sell you, but I'd love to see these ideas live on.

[torkleyy]

Yeah I've looked into Rhai some more since I wrote that comment. I think it has promising ideas :) I would personally not use it as is though.

It has the necessary features, but one thing in particular that I'm unsure about is the function overloading. The problem I see is that by checking the exact type ids, it's easy to get an error because you're using a primitve with a different size (i32 instead of i64, etc) and Rhai does not handle that very well yet. Maybe that works better than last I checked by now though. I started improving the dispatching in one PR, but I didn't have much time to continue it.

Let's see, maybe I'll look into it again in a few months.

[jonathandturner]

@torkleyy - for that kind of case, it's good to be specific about what you support in the scripting language, and why. Scripting languages are often more flexible than the static languages, naturally.

One place they tend to be flexible is with numerics. They may not tell a difference between i32 and i64. Whatever amethyst ends up using may instead convert all integers to isize/usize or i64/u64, for example.

Just to be clear, I'm not suggesting anyone use Rhai. Instead, I'm suggesting using the techniques Rhai uses to make a scripting engine that works closely with Rust.

[jojolepro]

We have an opened rfc about scripting and which languages will be supported (all of them lol). amethyst/rfcs#1

[fhaynes]

Closing this as it is subsumed by the RFC linked above.