RFC: Extensions system "Shards"

[White-Oak]

Each Amethyst game project can contain several 'shards', which are essentialy Amethyst fucnctionality packages with code and/or resources.
Examples of such shards can be:

• Renderer
• ECS
• Audio
• Physics
• AI

Shards can be connected together in the core amethyst_engine crate to have additional functionality such as

• ECS & Renderer will provide aditional functionality such as Rendering systems/components.
• ECS & Audio will provide additional systems/components.

Following tasks are considered completed when discussed and implemented. If the team comes to a conclussion on a task, it is marked in a certain way:

• Part 1: Features
  Technically, adding shards to projects will be done through 'features' -
  conditional compilation supported by Rust and Cargo. Thus, adding, for example 'ecs' shard will result in adding 'ecs' feature to Cargo.toml of a project. Feature ecs will enable ecs crate in amethyst_engine.
  Some features, as mentioned above, can be glued together in amethyst_engine crate using combinned features (i.e. cfg(all(ecs, renderer))).
  Some shards may not be included in base amethyst_engine crate and CLI will need to inject an additional dependency, when ading such a shard.

  • CLI (agreed on) (impl PR: subcommands for CLI to add/remove shards)
    Proposed:

  amethyst add <name>  
  amethyst remove <name>
  

  Alternatives:

  amethyst shard add <name>  
  amethyst shard remove <name>
  

  • Shards list
    There may be inconsistence between shard name and its crate name. As well, some shards may not be included in the base crate, so an additonal dependency needs to be injected. A list of such mapping ('exotic_shard' -> 'exotic') and a list of necessary dependencies needs to exist.
    • A format of a list. Those are basically two maps. What format to use?
    • A place to keep it. Options are:
      • inside tools crate
      • in some (existing or separate) repository on GH, being synced when using CLI to add shards.
    • User-specific shards. Users may share their own shards for a specific purpose. How to handle them? These are called third-party libraries or extensions and are not shards.

• Part 2: Shards' resources
  Each shard may bring its own resources, and/or configuration files, and/or modify existing. I have no idea of this part. It should be thouroughfully discussed and, maybe, even separated to its own RFC.

[White-Oak]

Any suggestions on missed tasks or redundant tasks are hihgly welcomed!

That's an extremely early draft of something Amethyst will be built on, so we need to come to conclussions and discover missing parts.

[White-Oak]

Any suggestions on missed tasks or redundant tasks are hihgly welcomed!

That's an extremely early draft of something Amethyst will be built on, so we need to come to conclussions and discover missing parts.

[ebkalderon]

@White-Oak Looks pretty good so far. There are a few points that I'm not quite clear on:

1. Aren't shards supposed to be based on Cargo features, not Rust modules?
2. I didn't really intend for the ECS to be a shard separate from the engine. This is why entities and prefabs are folders included by default in empty game projects. However, I did intend for processors and their on-disk resources to be shards.
3. I'm not completely familiar with Cargo features by any means, but aren't they processed at compile time? If so, how could third-party shards be installed and created without delving into the engine code itself and adding #[cfg(...)] markers directly?

[ebkalderon]

@White-Oak Looks pretty good so far. There are a few points that I'm not quite clear on:

1. Aren't shards supposed to be based on Cargo features, not Rust modules?
2. I didn't really intend for the ECS to be a shard separate from the engine. This is why entities and prefabs are folders included by default in empty game projects. However, I did intend for processors and their on-disk resources to be shards.
3. I'm not completely familiar with Cargo features by any means, but aren't they processed at compile time? If so, how could third-party shards be installed and created without delving into the engine code itself and adding #[cfg(...)] markers directly?

[White-Oak]

1. Yes, you are right. I'm not sure how to reword a title. Packages? Subsystems?
2. What if ECS doesn't suit my needs?
3. I think they can't be called shards anymore, you are right. More like third-party extensions/libraries. i'll edit this out.

[White-Oak]

1. Yes, you are right. I'm not sure how to reword a title. Packages? Subsystems?
2. What if ECS doesn't suit my needs?
3. I think they can't be called shards anymore, you are right. More like third-party extensions/libraries. i'll edit this out.

[ebkalderon]

1. I'd say that these are technically packages or extensions that implement specific functionality in the engine. Amethyst would come supplied with several basic shards by default, and users can tear out those that they don't use. The packaging standard for shards must be detailed enough, however, that it would be possible for people to write their own. All in all, very similar to Cargo subcommands.
2. While I agree that being able to replace the ECS would be great, I think that doing so would complicate things a lot since a large number shards would be reliant on this particular ECS shard. And if someone wrote an alternative ECS shard, how would one make sure that all the other dependent shards were compatible with it?
3. They can still be called "shards." 😄 I think that we could work around this compile-time problem by having all shards, third-party or not follow standard feature names (e.g. all rendering shards would obey the "renderer" feature, all audio-related shards would obey the "audio" feature, and so on). Replacing Amethyst's built-in shards with a third-party one would then involve making your own shard package according to the spec and hooking it up with the appropriate feature name.

[ebkalderon]

1. I'd say that these are technically packages or extensions that implement specific functionality in the engine. Amethyst would come supplied with several basic shards by default, and users can tear out those that they don't use. The packaging standard for shards must be detailed enough, however, that it would be possible for people to write their own. All in all, very similar to Cargo subcommands.
2. While I agree that being able to replace the ECS would be great, I think that doing so would complicate things a lot since a large number shards would be reliant on this particular ECS shard. And if someone wrote an alternative ECS shard, how would one make sure that all the other dependent shards were compatible with it?
3. They can still be called "shards." 😄 I think that we could work around this compile-time problem by having all shards, third-party or not follow standard feature names (e.g. all rendering shards would obey the "renderer" feature, all audio-related shards would obey the "audio" feature, and so on). Replacing Amethyst's built-in shards with a third-party one would then involve making your own shard package according to the spec and hooking it up with the appropriate feature name.

[White-Oak]

2,3. Hm. I had in mind that shards would not be designed to be fullfully interchangeable. I mean if I don't want to use Amethyst's ECS I can either use some other little bit different ECS as a replacement shard or I could just use some other completely different system and it would be my responsibility to glue it with renderer etc.

[White-Oak]

2,3. Hm. I had in mind that shards would not be designed to be fullfully interchangeable. I mean if I don't want to use Amethyst's ECS I can either use some other little bit different ECS as a replacement shard or I could just use some other completely different system and it would be my responsibility to glue it with renderer etc.

[White-Oak]

As being disscussed in Gitter we are still not sure on whether shards should be added via amethyst add/remove <name> or amethyst shard add/remove <name>

Points about the latter form:

• It is longer
• It will allow for some extensions for shards CLI like amethyst shard info <name>
• If somewhere in the future CLI would have some other things to add/remove it won't cause a confusion.

[White-Oak]

As being disscussed in Gitter we are still not sure on whether shards should be added via amethyst add/remove <name> or amethyst shard add/remove <name>

Points about the latter form:

• It is longer
• It will allow for some extensions for shards CLI like amethyst shard info <name>
• If somewhere in the future CLI would have some other things to add/remove it won't cause a confusion.

[Oflor]

As I see it, our ECS should be a separate and independent crate so that it could be used with other projects and game engines, and our Amethyst should depend on it. However, I don't see it as an optional dependency (same as renderer, audio, etc), because all except the most basic games are going to use entities, audio and graphics (but probably not AI). In the end, a game engine's purpose is to provide game developer with the most essential APIs for games, which include graphics and logic/ECS. Because of this, I don't fully understand the necessity for shards. If there are going to be other shards which will be truly optional, please, give an example.

But for now, I'm against that idea, as it introduces additional complexity both for game ("why do I need to explicitly add all these basic subsystems which needed for every game anyway?") and engine ("can't I assume that ECS library is already included?") developers.

[Oflor]

As I see it, our ECS should be a separate and independent crate so that it could be used with other projects and game engines, and our Amethyst should depend on it. However, I don't see it as an optional dependency (same as renderer, audio, etc), because all except the most basic games are going to use entities, audio and graphics (but probably not AI). In the end, a game engine's purpose is to provide game developer with the most essential APIs for games, which include graphics and logic/ECS. Because of this, I don't fully understand the necessity for shards. If there are going to be other shards which will be truly optional, please, give an example.

But for now, I'm against that idea, as it introduces additional complexity both for game ("why do I need to explicitly add all these basic subsystems which needed for every game anyway?") and engine ("can't I assume that ECS library is already included?") developers.

[White-Oak]

All in all, after a discussion in Gitter, I came to a thought, that maybe none of this is needed. ECS, Graphics, Audio are all must-have for every video game, so why bother with 'shards'. It sounds cool, it looks cool, but for the simplicity sake and as common sense tells, it's better just to have monolithic game engine. It will ease interdependencies and all.

[White-Oak]

All in all, after a discussion in Gitter, I came to a thought, that maybe none of this is needed. ECS, Graphics, Audio are all must-have for every video game, so why bother with 'shards'. It sounds cool, it looks cool, but for the simplicity sake and as common sense tells, it's better just to have monolithic game engine. It will ease interdependencies and all.

[thiolliere]

I don't understand what kind of engine you have in mind. Do you want to create a monolithic game engine such as Unity or UnrealEngine ?

My vision about this project wasn't a monolithic game engine.
It was a core engine containing almost nothing but the ECS engine.
and some 'shards':

• Audios
• Physics
• Renders
• Inputs

and the developer had to choose some shards:
basic-audio, 3D-physics ...

all shards can be configurable in config.yml but without interdependency

• audio:
  • default general volume
  • default music volume
  • default sound effect volume
• physic:
  • gravitation
• ...

and they can contains traditional crate feature but also component, system and entities.

• 2D-basic-physic for example have:
  • component
    • dynamic-component
    • static-component
    • kinetic-component
  • system
    • world update
• basic-audio doesn't have any ECS things but just a global access to sound output that would be initiated at the start of the game and then accessed globally like:
  sound.play_effect(effect::boom,x,y,z)

in this vision amethyst would only be

• a core engine that starts global things like window, sound according to config.yml
• lots of 'shards' with
  • ECS implementing the core ECS trait
  • global access
  • functions

[thiolliere]

I don't understand what kind of engine you have in mind. Do you want to create a monolithic game engine such as Unity or UnrealEngine ?

My vision about this project wasn't a monolithic game engine.
It was a core engine containing almost nothing but the ECS engine.
and some 'shards':

• Audios
• Physics
• Renders
• Inputs

and the developer had to choose some shards:
basic-audio, 3D-physics ...

all shards can be configurable in config.yml but without interdependency

• audio:
  • default general volume
  • default music volume
  • default sound effect volume
• physic:
  • gravitation
• ...

and they can contains traditional crate feature but also component, system and entities.

• 2D-basic-physic for example have:
  • component
    • dynamic-component
    • static-component
    • kinetic-component
  • system
    • world update
• basic-audio doesn't have any ECS things but just a global access to sound output that would be initiated at the start of the game and then accessed globally like:
  sound.play_effect(effect::boom,x,y,z)

in this vision amethyst would only be

• a core engine that starts global things like window, sound according to config.yml
• lots of 'shards' with
  • ECS implementing the core ECS trait
  • global access
  • functions

[OvermindDL1]

I would think of it the same way, the 'Engine' should be an ECS/Event or similar driven engine with others 'Shards' that add components and systems, whether Scenegraph (octree systems, portal systems, etc... and related components), OGL/Vulkan/DX Rendering systems (or none for, say, a headless server, heck someone could make a Console rendering system that dumps information to the console or over SSH or so). Just simple addable parts to let people add and use only what they need.

[OvermindDL1]

I would think of it the same way, the 'Engine' should be an ECS/Event or similar driven engine with others 'Shards' that add components and systems, whether Scenegraph (octree systems, portal systems, etc... and related components), OGL/Vulkan/DX Rendering systems (or none for, say, a headless server, heck someone could make a Console rendering system that dumps information to the console or over SSH or so). Just simple addable parts to let people add and use only what they need.

[ebkalderon]

@thiolliere I believe you are mistaking decomposability for monolithic design. Personally, I believe that amethyst_engine should have the ECS as a hard dependency because it affects so many other parts of the engine that it's essentially inseparable. However, if people want to use the engine with something else, they should be able to use the sub-crates individually with their own solution.

I do not believe that something as complex as an ECS, which is essentially a game-making approach, can be hot-plugged without becoming a tangled mess of dependencies, e.g. Piston.

However, I agree with you on all other points. The engine would contain very basic resource management, configuration loading (config.yml), and game loop, with everything else dictated by Cargo features that let people switch functionality on or off at compile-time.

[ebkalderon]

@thiolliere I believe you are mistaking decomposability for monolithic design. Personally, I believe that amethyst_engine should have the ECS as a hard dependency because it affects so many other parts of the engine that it's essentially inseparable. However, if people want to use the engine with something else, they should be able to use the sub-crates individually with their own solution.

I do not believe that something as complex as an ECS, which is essentially a game-making approach, can be hot-plugged without becoming a tangled mess of dependencies, e.g. Piston.

However, I agree with you on all other points. The engine would contain very basic resource management, configuration loading (config.yml), and game loop, with everything else dictated by Cargo features that let people switch functionality on or off at compile-time.

[Xaeroxe]

We're pretty unlikely to implement this at this point. No one is motivated to do so and we have cargo features and modularization if anyone really needs to do something like this. So I'm closing this.

[Xaeroxe]

We're pretty unlikely to implement this at this point. No one is motivated to do so and we have cargo features and modularization if anyone really needs to do something like this. So I'm closing this.