Scripting RFC

[Moxinilian]

This RFC introduces our scripting proposal.
Rendered

@kabergstrom

[Xaeroxe]

I really like all of this. My only suggestion is a name change, rather than augmented FFI we should call it Automatically Idiomatic Foreign Function Interface. AI-FFI.

[fhaynes]

The first one is to ban scripts from handling their own threading.

What about languages that use green threads?

[Moxinilian]

Feel free to use GitHub's review features so we can have threads.
@Xaeroxe Sure, why not, I called it augmented FFI just for the context of the RFC, I wasn't planning on keeping the name.
@fhaynes I think this could fall under the "limited managed threading" kind of deal. But maybe it's been too long since I've read about green threads. I need to look at it again.
But like for example it would be perfectly fine to have Rust have threads, because Rust is smart. The constraints here are language-specific.

[fhaynes]

Feel free to use GitHub's review features so we can have threads.
@Xaeroxe Sure, why not, I called it augmented FFI just for the context of the RFC, I wasn't planning on keeping the name.
@fhaynes I think this could fall under the "limited managed threading" kind of deal. But maybe it's been too long since I've read about green threads. I need to look at it again.
But like for example it would be perfectly fine to have Rust have threads, because Rust is smart. The constraints here are language-specific.

I mean more like threads that are managed by a language runtime, goroutines or Erlang processes for example, but the OS views as just one OS thread.

[Moxinilian]

It would depend on specific cases then. I haven't thought about it yet. If you want to make an addition, please write a paragraph and PR it to my fork.

[fhaynes]

In order to expose that API in an idiomatic way to the language they drive, the drivers are also provided with additional high level metadata, along with the FFI. For example, if a type is an iterator, the language driver will be noticed of this additional information so that it can, for example, do the necessary language-specific work for this iterator to be iterated over in a for loop (see the Lua example for more details).

Something about this bugs me, but I'm not sure what. Will have to think about it more. At a minimum, I think this will need a strict schema. I just have a suspicion that the driver code responsible for the the translation will end up messy. But we won't know until we try it.

[fhaynes]

It looks pretty solid to me. I'd vote move forward with a minimal PoC using LuaJIT to get an idea of the difficulties associated with this level of genercism.

[torkleyy]

I think this is pretty similar to what was said before. I'm going to go through the details later.

[torkleyy]

I think this is pretty similar to what was said before. I'm going to go through the details later.

[Marwes]

Does this mean that script system(s) are run in a separate World(?). I don't remember where I read it but since there is no way to add or remove systems after constructing a World it was recommended to keep dynamic systems in a separate world to avoid needing to reconstruct everything everytime something is reloaded.

https://github.com/Marwes/shred-example

https://github.com/Marwes/shred-example/blob/9f6d6a62f0b4cf7f55c3e5e9ca851c76c8ec9dea/src/gluon_system.rs#L798-L800

[torkleyy]

The architecture Specs will have in the future will differ from the current quite a bit. It's already in design ;)

[torkleyy]

Regarding your comment, I think you're mixing up Dispatcher and World. Both are very likely going to change (the latter one also renamed to SystemGraph which will not need any of those workarounds).

[jchitel]

Modifying existing component types on the fly would only work on component types that were created from the scripting environment, but I think it's a reasonable constraint.

Does this mean that scripts would not be able to modify core components like Transform? That seems like it would reduce the power of scripts quite a bit. Is there any reason that component methods couldn't be exposed? Or is there another way that scripts might access static components that I'm missing?

[Moxinilian]

What I meant is that scripts can't change the type signature of components defined outside of the scripting environment, as in they can't modify them. In other words, if your project creates a Rust struct as a component, you won't be able to hot-reload its structure definition.

[jchitel]

Ah, you meant the component types, not component instances. So scripted systems will be able to read arbitrary storages from the world and modify the components in those storages, correct?

[Moxinilian]

Not the systems. Scripted systems can do exactly as much as Rust systems.
But from the scripting ecosystem, you can't edit static Rust types.

[jmqualls]

My 2c as someone who is not (yet) a contributor but a prospective user:

A bird in the hand is worth 2 in the bush. Don't design an API for all languages up front. Just the fact that there exists an API separating the engine and the language is sufficient abstraction for a first pass.

Make the first language a mature one. Don't adopt someone else's technical debt. LuaJIT seems like a fine choice.

Once people can actually write and run some kind of script at all, then start thinking about what kinds of accommodations would have to be made to support MUMPS or Befunge or whatever. Don't be afraid to say no if the language concepts just don't map well to the engine data structures.

[Moxinilian]

Could you give more details on the reasoning behind that conclusion?

[jmqualls]

Could you give more details on the reasoning behind that conclusion?

http://wiki.c2.com/?PrematureAbstraction
http://wiki.c2.com/?TracerBullets

[Moxinilian]

I published a post on the Amethyst forums containing more details on the LuaJIT implementation.

[Moxinilian]

@jmqualls I wouldn't call premature a solution to a problem all major game engines have failed at solving because of their legacy code and userbase. By only thinking about a single language, we get ourselves trapped in that language and might eventually be stuck with it. Unity is so stuck with C# that they ended up creating a custom compiler for it when they realized the language requirements had changed.

Besides, so far, this general purpose solution has not been in the way at all of the LuaJIT integration. If you want to consider this + LuaJIT as focusing on a single language first, then it's all good! It isn't very complicated either, we are just describing the essence of an ECS and the requirements for it to fit to a language.

[DianaNites]

One possibility is to have something that other languages can compile to easily.

Even Lua can fit this purpose, with several languages that can compile to it, including C#. Typescript to Lua compilers are particularly interesting to me, basically free types and other nice things, and much better IDE support than plain Lua.

[Moxinilian]

@DianaNites My issue with this is that it would introduce one more potentially costly abstraction layer. For example, say we want to integrate Go with this method, we cannot take advantage of its pretty performant static builds. Also, this does limit what languages we want to integrate, as we wouldn’t have the opportunity to maintain different compilers to Lua.

But this approach is not incompatible with the one suggested in this RFC, so if it makes sense in some casss it can definitely be used.

[fhaynes]

Merging this as @khionu is satisfied.

[zicklag]

When does the builder phase happen in this design? The build is taken care of automatically by Amethyst or other Amethyst tooling without you having to re-compile Amethyst itself, right?

I want to understand whether or not I can build Amethyst itself one time, distribute that to users, and then allow them to write games with just the scripting languages and the Amethyst binary ( even if that includes more tooling like an Amethyst CLI ).

[Moxinilian]

You are correct. The build is, at least during development, a dynamic process. If the language is a dynamic one (like Lua for example), then it would work very simarly to how it is commonly done. If the language is a more static one (like Rust for example), it’s a module of the engine that builds your code and dynamically links it without even needing to shut down the engine.

[erlend-sh]

For those following along, the implementation for this RFC is being discussed here:

• https://community.amethyst.rs/t/scripting-what-do-we-need-to-get-there/958?u=erlend_sh
• https://community.amethyst.rs/t/scripting-component-schema-language/1021?u=erlend_sh

The implementation repo lives here: https://github.com/katharostech/amethyst-scripting-lab