[RFC] Amethyst as bin

[Xaeroxe]

Rendered

[AnneKitsune]

I got a couple of questions concerning this. It still feels pretty vague.

Also I'm a bit afraid we will lose customization power by doing this.
You can't run amethyst as a thread in a bigger application when doing this for example.

Here's one of my use case for reference:
An application listens for tcp connections. When it receives one, it creates a process with the amethyst engine and passes a "game replay" as an argument to it. The process plays the "replay" and ensures it is valid (no memory manipulation, physics are correct, time is correct, no teleportations, etc...). At the end, it returns a value to the host process. 0 if it is valid, 1 if it isn't. Then the host process reports back to a backend server.

Is this still achievable under this architecture? It is a bit unclear.

Thanks for opening the RFC!

[AnneKitsune]

I'd assume this would reduce parallelism since dispatchers are run sequentially.

[Xaeroxe]

Maybe a bit, but I provide the following counterarguments

1. Rendering wasn't parallel anyways, it's a thread local system.

2. Input is pretty lightweight, mostly O(1). It doesn't get more complicated the more entities there are for example.

[AnneKitsune]

How does it do it? What does it implies? What are the steps the user has to do to compile (their game?, amethyst?).

[Xaeroxe]

How does it do it?

Intentionally left open so as to permit flexibility in implementation, but right now my first guess would be that on the user computer we have a amethyst_bin executable project whose Cargo.toml file contains a reference to a user crate, then we just use that crate like we would any other crate making some assumptions along the way about what functions and types it will provide.

What does it implies?

This question is so vague I don't know how to answer it. It implies <insert contents of the RFC here>.

What are the steps the user has to do to compile

Probably something like amethyst run wherein our tools handle the build for them.

[AnneKitsune]

yeah but also that is the section of the rfc where you are supposed to describe step by step what happens internally. What is the amethyst run command actually doing, and in which order.

[Moxinilian]

This issue is very interesting to me. It has been debated whether or not Amethyst truly is a game engine or merely a game framework. I think we could try the novel approach of being both.

As this RFC mentions, engines usually handle everything for you, while you handle everything for the framework.

What if Amethyst remained a crate that users can use like a typical game framework as they will, but as soon as they create an Amethyst project in the "official" editor, it transparently generates all of the setup code? In other words, the editor would codegen the scaffolding crate with everything to load the engine, and hook in what is modified in the editor (no real-time codegen, just a typical template crate with all the hooks to the editor).

That approach has the advantages of:

• Letting the user choose the way they want to use Amethyst. Hobbyist tech-oriented game makers tend to prefer game frameworks when artists and game studios seem to prefer engines. With this we can offer something appealing to both.
• While we will have and maintain our official game editor (Codename Atelier), this would allow third parties to create their own tooling experience as well, which was part of Amethyst's design motivations.
• Letting the user modify that generated code could let them hook deep into the editor's generated game, which can be extremely useful compared to other game engines (see Unity).

In the end you get most of the advantages of an engine-as-a-binary with customisation and static linking. The obvious drawback is that you also need the build toolchain, but I don't think it's a noticeable issue considering how automatable Rust installation can be.

[Xaeroxe]

@Moxinilian yeah I like that approach. The only thing that really concerns me about it is having the distinction between Atelier users and framework users, as framework users won't get all of the same benefits as Atelier users. My objective is to design this with enough flexibility to where people don't feel like they need to have fn main() and are content using the workflow proposed here.

[Xaeroxe]

@Jojolepro

Here's one of my use case for reference:

That should still be possible, though we'd need to provide a function for returning an exit integer which we would then pass into https://doc.rust-lang.org/std/process/fn.exit.html

Actually nevermind, the game crate could just call std::process::exit whenever it wants to.

My objective is to flesh out our interface enough that people don't feel a compelling need to have their own fn main().

[Moxinilian]

Well, I think the main appeal of Amethyst would be in its engine side anyway, as most people involved in the framework side would probably be hobbyist more curious about the technology than anything (I mean it might become the most powerful game framework available so I'm not sure about that).

I think if we focus our marketing on the Atelier experience, most people will expect the difference in experience they will get by using the framework mode.

[Moxinilian]

Also, if you want to make it more managed even in framework mode, you can make the user implement an Application trait with methods being called on each initialisation state.

[Xaeroxe]

@Moxinilian I'm legit kind of mad I didn't think of that. That's great! Using traits we could drastically reduce the amount of "magic" that happens by having everything come through a single interface. We should do that for Atelier users too!

[Xaeroxe]

That also has the added benefit that we can now have optional functions in the trait implementation.

[AnneKitsune]

Hobbyist tech-oriented game makers tend to prefer game frameworks when artists and game studios seem to prefer engines.

Wouldn't it be the opposite? I'd assume hobbyist would want to get started as quickly as possible, while game studios need a larger amount of control over the execution.

[Moxinilian]

Game studios expect to have a nice and easy to use workflow they can give to their artists. When I was referencing hobbyists, I meant the people making games for the love of the craft more than the game itself.

But it is clear that studios also want control over things. As you may have noticed, engines like Unity do not deliver, but they still believe workflow is worth more. We can fill a gap here.

[LucioFranco]

I'm all for the Application trait idea. I wonder if we can abstract stuff out so you can test things, like integration tests?

You could simulate a set of inputs coming into the engine and check for outputs.

[Xaeroxe]

I've been thinking about this some more, and I wanted to provide a few answers for some of my own questions

Will all user provided crates have the same name? Should we attempt to build a system that supports dynamic user crate names?

I don't think we should support multiple crate names, because folder names don't have to match the name of the crate inside them. Inside the Cargo.toml file we can name all user crates game and then just generate the folder with the name they've provided. The only viable alternative to this I can see is taking control of rustc ourselves and effectively re-implementing cargo. I'd like to avoid that if we can. I want to be a front end to cargo, not a front end to rustc.

What would it look like when a user needs to upgrade the version of Amethyst that they're using? We can't just use cargo anymore if we're going with this approach. Maybe we could keep most of the engine as a lib and provide amethyst_bin projects that use the amethyst crate on cargo, as well as hooking into a user provided "game crate"?

This is where we'll probably want an Amethyst.toml file. That file contains the Amethyst version as well as any other Amethyst specific metadata we may find necessary. Then we can provide a command to users that I'd call with something like amethyst upgrade 0.12 which would then alter the Amethyst.toml file to contain the new version, run any migration scripts we've provided on their code base, (which could do things such as automatically altering function interface signatures) and finally make sure they have that version of amethyst installed, if they don't we'll download and install it for them.

[Xaeroxe]

Oh and another cool thing we can do with this approach: since we're installing Amethyst anyways why not make a pre-compiled amethyst.rlib file part of that installation? Then we can just have the amethyst_bin link against that rlib file instead of asking our users to download the source and compile Amethyst.

[torkleyy]

I think this section could use more details.

I see one major drawback which is not being able to no longer call Amethyst, but now being called by Amethyst. That means a huge lost in flexibility.

[Xaeroxe]

That means a huge lost in flexibility.

That's exactly what I'm trying to avoid with this RFC process. The biggest and most important question this RFC needs to answer is "What do we need to do to make it so that none of our users miss being bin?" If we can't answer that question then this whole approach falls apart.

[torkleyy]

that are very reasonable for a game engine to do

Well, that depends a lot on what you expect a game engine to do 😄 It also depends on if you prefer facilities that are provided or frameworks. The latter imposes more structure on your code. This can have advantages sometimes (think of the ECS pattern), but it can also feel very restricting.

[minecrawler]

At the moment, I can grab my favorite application framework, write a complete application there, and if I want and need, start and stop rendering Amethyst, just like that. Heck, I can go and build an Amethyst launcher, from which I start unrelated Amethyst games, all in one application - how cool is that? However, with this RFC, something like that would become impossible, which would be a pity. A usage like that might be an edge case, but you'd cut all edge cases and Amethyst would become a 100% game engine, no other uses supported (without modifying engine code).

That's why I want to throw in another idea: How about providing both, Amethyst as an engine and Amethyst as a framework? The engine would have the framework as a dependency and implement all the nice ideas mentioned here. It basically is the convenience layer anyone creating a game wants and people would pay money for the support. This wrapper, though would only contain the bindings and convenience, no shiny tech. That would be the framework's job. It would contain all the low-level stuff and awesome shiny things we have today, which could be used in exiting ways, which might not be the need of game studios or things anyone here can come up with on the spot. I don't think that only hobbyists would be happy about that, but also professionals who need a modern integrated 3D framework for other jobs than one big game.

Something like the Amethyst framework is missing from the Rust ecosystem, so if Amethyst would go engine-only, who would fill the framework gap? Do you want to send people over to C++ and Ogre3D for that? Or should others create a new project, which basically uses most of Amethysts crates or even duplicates code, just to remove the engine part? Amethyst is shiny, and I'd welcome a focused effort a lot :)

[Xaeroxe]

So to recap after this amethyst will exist as 3 major "sections" of code.

• amethyst crate. This is the only section that exists today and will be unchanged.
• amethyst_bin executable projects. This will consist of, at first, a standalone engine project and an editor project, in the future we might have more, this RFC leaves this open for future enhancement.
• amethyst tools. Major enhancements will be made to our toolset to support easier interfacing with the amethyst_bin projects.

Both amethyst_bin projects and user provided game crates will link to the amethyst crate.

So, all in all this RFC does not remove any existing capabilities for users, only adds a new way to use the amethyst crate.

If we're all in agreement I'll add this to the RFC officially.

[Moxinilian]

Could you explain again what amethyst_bin is?

[Xaeroxe]

It is the executable that the user crate hooks into. It's where fn main() lives for that workflow.

[Moxinilian]

I don't understand why this is still necessary considering what we discussed above?

[Xaeroxe]

Uh ok can you please provide quotes and links to comments that I may have misunderstood? It is my understanding currently that we want to expose both our current setup as a game framework, and our new proposed setup here as a game engine. None of what I've read above indicates we want to nix this RFC, instead we just want to make it supplemental to an already provided interface. amethyst_bin is the crux of this RFC. This RFC accomplishes nothing without it.

[Moxinilian]

Well I expected the framework workflow to be what it currently is, and the engine to build a sample project on top of the framework that is piloted by the editor. How does the external binary fit in that?

[Xaeroxe]

piloted by the editor

And therein is my point, we can't pilot without fn main which is what amethyst_bin provides. amethyst_bin is merely a generalized concept for any receiver of the game crate interface discussed above. amethyst_bin is the editor, and the standalone executable.

[Moxinilian]

But why wouldn't the editor generate the fn main? You don't have to touch any Rust when using the editor, it generates everything.

[Xaeroxe]

after this RFC is implemented the engine would provide fn main() and auto generate the interfaces to fn main(). fn main() is not a part of user code, it's a part of our code.

[AnneKitsune]

From what I understood, the bin would be built over the framework, as a separated crate. That way the user can either get their code call, or call the framework depending on the choosen dependency (bin or lib)

[fu5ha]

I think @Moxinilian 's original idea was basically "amethyst_bin should just be a static codegen scaffolder" much like, say, yeoman from the JS ecosystem if you are familiar with that. In order to keep using things with official Amethyst tools/'Atelier', you'd need to keep much of that scaffolding the same to retain those guarantees.

I actually think the create-react-app model could be fairly effective here. It basically scaffolds and keeps under wraps much of the internal code of the app and only presents user facing things to edit by default, however, you can also 'eject' and work with only the raw framework, but you'd lose support for working with official tools.

In this model, you aren't controlling fn main() by having a completely separate binary, but rather you're having control of it by intrinsically just controlling that part of the source by not exposing it as being editable directly by the user unless they eject from the engine tooling.

[Frizi]

I think that the Prior Art section is not really paying it's respects to the whole history of gamedev. Very many games were created without splitting their codebase into such chunks.

While I agree that it might seem appealing to own the main again, mostly for editor purposes, it also brings a lot of drawbacks. Scripting becomes a necessity, interactions with native code (notably ECS) are suddenly a pain. Modifying the render pipeline will also be unnecessarily complex. And worst of all, any change in the user's part of native game code will require full editor restart. I think we should be smarter than that.

As I see this RFC mostly tries to solve two problems: scripting and editor integration. I think that both can be better solved with other solutions.

Editor could indeed have it's own binary, but that would just be an amethyst_editor_bin. That editor could then hook into the actual amethyst game with some RPC hooks exposed by the engine. That way restarting the game doesn't bring the whole editor down.

Scripts can be treated like assets. Any compilation step can be handled by asset pipeline, reloading them will already be handled by hot reloading. I don't see any reason why amethyst_bin could make that any easier.

[Xaeroxe]

I see the problems you've brought up, and I do agree they're problems I just also am struggling to figure out how to resolve them without a stable Rust ABI. The fact that static linking is the only well supported way for two pieces of Rust code to interact with each other right now really puts a damper on the whole thing. We could use the C ABI, but I think that would result in a severely mangled and disjointed code base that misses out on a lot of the benefits of Rust.

Editor could indeed have it's own binary, but that would just be an amethyst_editor_bin. That editor could then hook into the actual amethyst game with some RPC hooks exposed by the engine. That way restarting the game doesn't bring the whole editor down.

Can you please provide more technical details as to how this would work? What is an "RPC hook" and how do I make one in Rust?

[torkleyy]

I think there are many interesting ideas in this RFC. It seems to me that the main motivation is solving the issue with high compilation times and the resulting inconvenient ECR cycle. As much as I can understand this rationale, I think this comes at a too big cost, as many previous comments have explained in detail.

The most important step in this direction is scripting, followed by other suggestions such as RPC communication.

Thus, I'd like to propose to close this RFC. It would be completely impossible to do this right now, and I'm not sure if we want to do this at a later point. In such a case, we should at least have a working prototype of this feature, and open a new RFC.

[Xaeroxe]

Perhaps now just isn't the right time. I'd recommend re-visiting the idea when Rust stabilizes their ABI.

[Xaeroxe]

So I sought to answer my own question about the "RPC hooks". I found it.

https://docs.rs/libloading/0.5.0/libloading/

This might serve as good inspiration for future RFCs.

[Frizi]

@Xaeroxe My idea about hooks was more like having some points in the engine that could send/receive data over network/ipc pipe in order to cooperate with the editor. DLLs solve different problems. With DLLs, you are loading foreign code into your process (or loading your code into a foreign process, depending on the perspective). What i was referring to is to actually have two processes talking together. That way failures on one can be handled by the other without necessarily bringing the whole setup down.