[WIP] nyan specification

[inakoll]

Before I forget, I am committed to write down what @TheJJ explained to me in #228 about nyan.
So here it is, an introduction to nyan (for newbies). At least this is my understanding.
This is a work in progress and I will see if I am able to do more specifications.
In the meantime I'll accept comments, corrections, suggestions..

[TheJJ]

Very nice, it's actually understandable now.

Another thing that might be useful understanding the upcoming deep integration of nyan:

The nyanspec is relevant at compiletime, because it will specify all static types and structs for running and reading nyan files later.

That means: The nyan spec is the single source of format specification. The specification is used to generate code:

• structs and other data formats are created in C++
  • the engine can access all file contents with type safety
  • the compiler can optimize everything better as there are no string-names as there would be with yaml or json
• a parser is generated (by openage/nyan) to create a C++ nyan reader (and later writer) for specified data format
  • basically fills up the structs generated above
  • can read any .nyan file at runtime and make calls to libnyan (see below)

libnyan is written in C++ (at least i think so) and manages all the patching stuff.

• The dynamic data overlaying is not dependent on the data format, but on the generated structs which are simply used then.
• libnyan is queried by the engine for providing all required information ("melding start-deltas"), trigger updates ("runtime deltas"), overlays etc.

And we even push this further:

• openage/convert/gamedata/ currently specifies the format of empires_x1_p1.dat
  • this format allows reading the binary file and storing it into tables
  • these tables and the original format suck, information is redundant and we can heavily improve it
  • buuut: the file format is given (every original game has it.)
  • ➡️ data transformation.
• the original file will be read according to the original format (we can't help it)
• a transformation is specified
  • mapping: original format -> our awesome format
  • redundancy is purged (e.g. no more double huskarl unit (1x castle, 1x barracs))
• export transformation destionation format as nyanspec
• export actual gamedata from the dat file as .nyan files
• create C++ code according to this specification
• ➡️
  • engine gets structs (defined in python openage/convert/gamedata/ -> converted to a nyanspec -> the spec is converted to the structs and parsers by python py/openage/nyan)
  • locally converted data is stored in many .nyan files, read by the generated parser at runtime
  • libnyan is used for aggregating and manipulating all data from loaded .nyan files

(oh gawd we really have to write these plans down, i guess if i get run over by a bus nobody would ever remember) (whee i just wrote them down)

(if you like you can just copy-paste my explanations, but feel free to optimize the understandability 😁)

tl;dr: jj just tried to explain the nyan master plan for storing data without any redundancy

[inakoll]

I updated the documentation and tried to keep it simple. I am not sure I was able to optimize the understandability though.

Oh and please do not die! : )

[andrekupka]

@inakoll Although I appreciate, that you are trying to improve the nyan documentation, it would be nice, if you keep the sections I have written or ask before removing them. The interfaces that you describe are not part of nyan and can't replace the dynamic attributes, because with interfaces you are still limited to a fixed predefined set of types at compile time, while dynamic attributes make it possible to add objects of a type with completely new attributes within nyan data files.

[inakoll]

I've asked on IRC before proposing this here. I'm not trying to impose anything but rather write my understanding of what JJ said :

"the compiler can optimize everything better as there are no string-names as there would be with yaml or json"

The benefits of static types are better performance and better compiler diagnosis. Dynamic attributes reduce the advantages of nyanspec, static types and c++ code generation.

I don't have any clue of what is really needed but I just can see that there is a real development effort here and that nyanspec and dynamic attributes doesn't seem to fit well together. I thought nyan interfaces could be a solution but if I were wrong I have absolutely no problem with putting everything back the way it was. :)

[TheJJ]

@inakoll no problem, at least our ideas no longer exist just in some of these brains 😉.

The ... is actually part of the syntax, it means that this nyan type can have dynamic attributes.
How they are typechecked is unknown to me, but their type is defined in the nyan spec file.

The libnyan will also probably never exist (my misunderstanding) because just defining entry functions that call to generated code is pretty useless. So the whole nyan-framework usable by the game engine is generated from the format specification.

Mods must adhere this format, otherwise they can't load. Static typing ftw!

[inakoll]

If types containing dynamic attributes are partially know by the game engine, type checking is partially irrelevant for types containing one of those "...".

For example an ability defined as

ABILITY {   
    name : string,
    ...
}

would be generated as (something like) :

class Ability {
public:
    std::string name();
    std::map <std::string, util::variant<int, bool, double, std::string, util::set>> dynamic_ attributes();
};

The "dynamic_attributes()" content just can't be statically checked at all or am I missing something? As long as there is no dynamic attributes everywhere or as long as they are not accessed from performance critical code it should be OK.

What is bothering me is that if for example we define the MOVE ability like this :

+ABILITY MOVE {
        description = "Allows units to move.",
        speed = 10.0     # with speed being part of the dynamic attributes of some ABILITY instance
}

at some point in the C++ code we really need to know that there is a MOVE ability with a speed attribute so at some point we would have to write something like :

    if (some_ability->is_move_ability()) { // or some other test
        do_something(some_ability->dynamic_attributes()["speed"]); // virtual function call + map access
    }
} catch (...) // oops is_move_ability implementation is buggy or throw or there is no "speed" attribute in some_ability

This could be done once at startup or maybe somewhere in the game loop and is really error-prone. The real problem is that the MOVE ability type is not known at compile time. One solution could be to provide the code generator with both a nyanspec file and a nyan file containing some base game properties. Thus the MOVE ability defined above would be generated in C++ code as :

class MOVE_ABILITY { // The MOVE ability has its own implicit type!
public:
    std::string name();
    double speed() noexcept; // Great!
};

If everything i wrote here is obvious to you then I've just understood something. Anyway, what do you think about it? As long as the user/modder does not extend from ABILITY but overrides existing abilities, it should be OK.

[TheJJ]

A mod must be able to extend the abilities dynamically.

Imagine: You have a japanese tentacle monster mod, which adds the ability vomit to a villager, something that didn't exist before. You can't regenerate and recompile all engine/nyan code, this attributes must be able to load dynamically. As far as I understood our idea, the ability can only modify known attributes (as these are generated struct members/classes/etc in the engine which can be manipulated). The vomit ability will probably add new animations and sounds, which the engine already supports.

If mods need to go deeper (he-he), then the engine actually needs to be extended, the nyanspec updated and we have to merge the change upstream i'd say. But @Fr3akout will surely know more about that.

[dbrgn]

Can someone explain what differentiates this format from other formats like JSON, YAML, TOML or S-Expressions?

In my eyes, introducing a new format always introduces a greater contributing barrier. New contributors need to learn a new format first. And parsers need to be written and optimized.

Does nyan have a unique property that makes it exceptionally fit for this task? :) (The USP, in marketing speech.)

[TheJJ]

Lemme try:

• nyan is typesafe
  • all allowed members, types and assignments are defined in the nyanspec
  • this specification is then used to generate the nyan parser and C structs (used in the engine)
  • this ensures the C++ engine doesn't need to be recompiled when mods change data
• nyan allows easy modding
  • data packs ship configuration data, all units available, etc.
  • modpacks can update the present information easily, by applying config "patches"
  • these "patches" can be applied when other triggers occur, this is the way technologies will work
• nyan is human readable
  • everything is written whitespace-insensitive (except in strings obv.)
• nyan is minimal
  • no redundancy (e.g. no xml tags)
  • easy to understand special symbols
  • can be machine-generated easily (the nyanspec and the nyan data)
• nyan is invented here™
  • we can change the specification to our needs whenever we want

[mic-e]

Basically what @TheJJ said. In addition, well, nyan tries to be a general language, but it was designed with openage in mind, so it has facilities to easily represent some of the more complex information, especially how a definition from one datapack updates the definition from an other one, and how a technology will modify the attributes of a unit.

[dbrgn]

OK, so you have a special patching syntax? Or inheritance?

[mic-e]

@dbrgn: Unfortunately, the doc seems to lack any larger example code, which I'd find far more useful than theoretical discussions about model-view controllers.

IIRC, the syntax would look something like this:

+UNITTYPE SCOUT {
    name = "scout cavalry"
    hp = 40
    damage = 3
    armor_pierce = 1
    armor_melee = 0
}

+UNITTYPE LIGHTCAVALRY : SCOUT {
    name = "light calvalry"
    hp += 20
    damage += 2
    // inherits all other properties from SCOUT
}

+TECH BLOODLINES {
   description = "through centuries of in-breeding horses and knights, you manage to somehow increase their hp."
   cost_food = 300
   cost_gold = 200

   ^UNITTYPE SCOUT {
       hp *= 1.2                        // also affects LIGHTCAVALRY
   }
}

+TECH LIGHTCAVALRY {
     description = "upgrades your scouts to light cavalry."
     cost_gold = 400
     upgrade_from = SCOUT
     upgrade_to = LIGHTCAVALRY
}

A mod that would increase the hp for light cav and increase the effectiveness of bloodlines to 30% would look like this:

@UNITTYPE LIGHTCAVALRY {
    hp += 5
}

@TECH BLOODLINES {
    cost_gold *= 1.5
    ^UNIT SCOUT {
        hp *= 1.3
    }
}

Regarding the execution order of the changes:

All enabled datapacks are read first; after reading base, SCOUT has 40hp and LIGHTCAVALRY has +=20hp.; after reading the mod, SCOUT has 45 hp and LIGHTCAVALRY has +=20hp.

Once all the datapacks have been read, inheritance is resolved: SCOUT has 45hp, LIGHTCAVALRY has 65hp, and BLOODLINES applies to SCOUT and LIGHTCAVALRY. This is the information (unit types) that will be available at game start.

Now when the player researches bloodlines in-game, that player's SCOUT and LIGHTCAVALRY unit types get modified to have 59hp and 85hp.

If the player researches the light cavalry tech, all the player's units that have unit type SCOUT get changed to unit type LIGHTCAVALRY.

Disclaimer: I did have a large part in designing this, but the final responsible guy/authority is @Fr3akout.

[dbrgn]

@mic-e thanks, looks very interesting :)

[mic-e]

@inakoll: Maybe you could include my example as well?

[inakoll]

@mic-e Done!

I think there is still room for some improvements

[TheJJ]

This ... is driving me nuts, we have to come up with a better alternative. Nobody will understand that, everybody will asume that the example will just go on.

[dbrgn]

Even Python has it :D

>>> ...
Ellipsis
>>> type(...)
<class 'ellipsis'>

[franciscod]

why don't we allow dynamic attributes everywhere?

[mic-e]

Regarding dynamic attributes and modules/abilities: The reason why we might or might not need dynamic attributes is because different units have different abilities.

For example, a Militia doesn't have a ranged attack and thus doesn't need a range attribute. A building can't walk and thus doesn't need a speed attribute.

Now there are several approaches to this issue:

• Simply have every single attribute in every single unit type; the unittype C++ struct will be quite overloaded with all sorts of attributes that this particular unit type will never need, but at least you're not forced to write down all of them in the .nyan file; they all default to some sensible value if they aren't listed. This approach means that if some mod needs a new ability, and that ability needs a new attribute, you'll have to update your nyanspec and re-compile the project (but you won't have to update any of the .nyan files).

  +UNITTYPE archer {
      health=30
      range=5
      attack=3
      speed=3
      abilities=[move, ranged_attack]
  }
  

nyanspec:

    UNITTYPE {
        health: int
        range: int
        attack: int
        speed: int
        abilities: list(str)
    }

• Add some sort of ABILITY member, which has ability-specific attributes. The problem with this is that the nyan parser can't know which attributes a specific ability has, and it's impossible to generate your own C++ struct for every ability, so the ability's will simply have what we call a dynamic attribute table, which is just a map<std::string, T> for every type that you can imagine. The nyan parser cannot verify this at load-time, and it's sort of bad for performance/memory usage/you name it. This is the approach that would require dynamic attributes.

  +UNITTYPE archer {
      health=30
      ABILITY move { speed=5 }
      ABILITY ranged_attack { attack=3, range=5 }
  }
  

nyanspec:

    UNITTYPE {
        health : int
        *ABILITY
    }
    ABILITY {
        ...
    }

• Use the same, or similar, syntax as above, but specify all your abilities in the nyanspec, and generate C++ structs for all of them, thus providing you with load-time type checking and all of nyan's regular type-safety features. In the C++ implementation, the UNITTYPE would have a std::vector<AbstractAbility *>, where all of the defined abilities are stored. This idea was proposed yesterday by @inakoll, and I haven't given it much thought yet. At first, it would require updating the nyanspec and re-compiling the project for every new ability added, but later an optional way to implement abilities purely in python, and thus ship them with a mod, might be added.

  +UNITTYPE archer {
      health=30
      ABILITY[move] { speed=5 }
      ABILITY[ranged_attack] { attack=3, range=5 }
  }
  

nyanspec:

    UNITTYPE {
        health : int
        abilities : list(ABILITY[*])
    }
    ABILITY[move] {
        speed : int
    }
    ABILITY[ranged_attack] {
        range : int
        attack : int
    }

disclaimer: all the nyanspec code was written without having looked at the nyanspec docs for several months.

Note: Whatever syntax we decide to go with, we need to make sure that the corresponding C++ code will be reasonable; for example, it shouldn't have to call into python code every single time a unit needs to move.

[franciscod]

+UNITTYPE archer {
    health=30
    ABILITY move { speed=5}
    ABILITY ranged_attack { attack=3, range=5 }
}

i like this!

for example, putting aside damage weakness/resistance, a militia isn't too far from a cavalry, or a camel, or an elephant: all of them are melee combat units that can move and do melee attacks

[mic-e]

To start an other discussion, I'd actually prefer a braceless syntax:

UNITTYPE archer:
    health=30
    ABILITY move: speed=5
    ABILITY ranged_attack: attack=3, range=5

[dbrgn]

I'd strongly support that.

[franciscod]

#28

here, @Fr3akout said that commas make the parser easier to write. i assume braces are along the same lines. but it'd be awesome to have a pythonish syntax!

...what about translating nyandata into python and then parsing it?

[mic-e]

What's a simple parser compared to an awesome-to-use language... I think making python parse the nyanfiles is a bad idea; there's too much that could go wrong, it's a turing-complete language, etc

[dbrgn]

Yep, I'd create a custom parser.

This shouldn't be too hard. An indentation starts a new scope. De-indentation resets it. A simple state machine should suffice.

Is there already some kind of implementation? Are you going to use a parser generator?

[mic-e]

A parser generator is usually overkill; a lexer generator might make sense.
@Fr3akout has some code in his nyan branch here.

[inakoll]

OK so following the "historical module" approach presented above by @mic-e, I tried to simplify the type system and reinforce the syntax consistency.
Disclaimer:
This proposal is a long shot... With this proposal the barrier between type and objects is kind of dropped. Nevertheless, objects attributes are well-defined and exclusively specified in nyanspec.

## nyanspec - proposal #3 - Object Oriented
# philosophy :
# - base types are exclusively defined in nyanspec
# - types are composable in nyanspec AND in nyan (at runtime)
# - as a consequence each object has its own unique type (a composition of base types)
# - unknown object properties cannot be introduced in the nyan data file
# - type inheritance is handled at runtime ('+' sign syntax)
# - the '+' sign marks the allocation of a new subobject
# - '@' refer to types/objects for static delta and overlays
# - '^' refer to types/objects for runtime delta
# - deltas are contininously applied following the inheritance order
# - runtime deltas are trigger by the game engine and/or a python script provided by a mod

MOVE {
    speed : int
}

WALK : +MOVE {}     
SAIL : +MOVE {}

# the WALK ability inherits from MOVE. Walking units obviously can move.
# WALK is a subtype of MOVE
# the SAIL ability also inherits from MOVE but is a different from the WALK

ATTACK {
    damage : int,
    target : list()      # A list of target objects of any type
}

RANGE_ATTACK : +ATTACK {
    range : int
}

ALIVE {
    health_points : int
}

COST {
    wood : int,
    food : int,
    gold : int,
    stone : int
}

INFANTRY :
    +WALK,
    +ALIVE,
    +ATTACK,
    +COST
{}

ARCHER :
    +WALK,
    +ALIVE,
    +RANGE_ATTACK,
    +COST
{}

BUILD_ABILITY {
    speed : int,
    building : list()  # A list of building objects of any type
}

VILLAGER : 
    +WALK,
    +ALIVE,
    +ATTACK,
    +RANGE_ATTACK,
    +COST,
    +BUILD_ABILITY,
{}

VILLAGER_TECH
{
    ^VILLAGER
}

LOOM :
    +VILLAGER_TECH 
{
    hp_bonus : int
}

## openage nyan datapack

# Set default properties for all villagers :
@VILLAGER {
    @WALK : speed = 10,             # inline syntax
    @ATTACK : {                          # or brace syntax?
        damage = 1,
        targets = [ANIMAL, VILLAGER, INFANTRY, ARCHER]
    }
    @RANGE_ATTACK : damage = 2, target = [ANIMAL],
    @COST : food = 50,
    @BUILD_ABILITY : building = [ ## list all buildings here ## ]
}

@LOOM {
    ^VILLAGER @ALIVE : health_points += hp_bonus
}

@INFANTRY {
    # Infantry units can attack the following nyan objects
    @ATTACK.targets = [PHYS_COORD, ANIMAL, VILLAGER, INFANTRY, ARCHER]
}

@ARCHER {
    @RANGE_ATTACK.targets = [PHYS_COORD, ANIMAL, VILLAGER, INFANTRY, ARCHER]
}

VILLAGER_MALE :
    @VILLAGER 
{
    # VILLAGER_MALE is a static delta/overlay over the VILLAGER object
    # Load male villager graphics here
}

LONGBOWMAN  :
    @ARCHER
{
    @COST : wood = 35, gold = 40

    # LONGBOWMAN is a static delta/overlay over the ARCHER object
    # Load LONGBOWMAN archer graphics and attack properties here
}

## battlefield villager mod nyan datapack example

BATTLEFIELD_VILLAGER :
    @VILLAGER
{
    # BATTLEFIELD_VILLAGER is a static delta of VILLAGER
    @WALK : speed *= 1.1,
    # This unit can range-attack ennemy units
    @RANGE_ATTACK : damage = 2, target += [VILLAGER, INFANTRY, ARCHER]
}

The nyan parser/generator would generate C++ code for leaf types defined in the nyanspec. Other types would simply be a dynamic composition of leaf types (e.g. std::vector<nyantype*>). The C++ should then handle the copy/reference relation between objects. A '+' char in the nyanspec suppose a copy semantic in the C++ code. An '@' char in the nyanspec suppose a reference semantic in the C++ code.

[andrekupka]

@inakoll So basically you want to use traits for composing nyan types?

[inakoll]

@Fr3akout Yes, that sums up everything.

I knew the C++ traits class technique but traits are a more general concept. So, this proposal is about traits for nyan.

[dbrgn]

Are traits something similar to mixins?

[inakoll]

Since nyan mixins define data (or state) they are mixins not traits, good point! :)
A trait would only define methods (runtime deltas) and not attributes.

[inakoll]

To answer @mic-e IRC concerns. The SCOUT and LIGHTCAVALERY units would be defined as follows :

CAVALRY :
    +WALK,
    +ALIVE,
    +ATTACK,
    +ARMOR,
    +COST
{}

SCOUT :
    +CAVALRY   #  scout is a new independent cavalry unit
{
    name = "scout cavalry"
    health_points = 40   # @ALIVE.health_points implicit??
    damage = 3
    armor_pierce = 1
    armor_melee = 0
}

LIGHTCAVALRY : 
    @SCOUT  # Light cavalry is an overlay over SCOUT
{
    name = "light calvalry"
    @ALIVE.health_points += 20
    @ATTACK.damage += 2
    # inherits all other properties from SCOUT
}

[mic-e]

what about

UNITTYPE BOWMAN {
    {+RANGEDATTACK}

    health=50
    range=4
    rangeddamage=3
    rangedfrequency=1

    abilities = [walk, rangedattack]
}

UNITTYPE CROSSBOWMAN : BOWMAN {
    rangeddamage *= 2
    rangedfrequency *= 0.8
}

UNITTYPE SUPERMELEECROSSBOWMAN : CROSSBOWMAN {
    {+MELEEATTACK}

    health += 40
    meleedamage = 10
    meleefrequency = 1
    abilities += [meleeattack]
}

[detrumi]

A few points:

• Try to keep the syntax simple. I like mic-e's example, but you could change {+RANGEDATTACK} to +RANGEDATTACK, for example. Also cut down all the special characters, keywords might be better for clarity.
• Maybe it's a good idea to implement the basic idea first? Hard-code some datatypes for the units first before we implement the codegen, so we can start using them. Inheritance is nice but not needed to get things to work, and those dynamic things aren't really needed until upgrades or civilizations.

[TheJJ]

Current version (nr. 8), non final of course but very interesting ideas in it:
https://pad.stusta.mhn.de/p/openage_nyan_proposal_

[TheJJ]

also very valuable ideas in here, although the previous link has the latest proposal.
https://pad.stusta.mhn.de/p/nyan-format

[franciscod]

Can we start writing some kind of printer/parser/lexer/formalgrammar for the nyan format?

[TheJJ]

No, even the grammar is not fix yet because the languate is far from being specified. See the latest pad for design information, @mic-e and I need to gather some motivation to continue designing this, but it's very complicated :)

[VelorumS]

Ow, I've seen a similar language of one big RTS: don't even think about the hierarchy for the units. Make them out of independent bricks.

Funny observation about the attribute/dynamic attribute system: it's like a straight Qt QObject with its properties and additional dynamic properties that can appear at the runtime.

[TheJJ]

Soooo, we may have a specification:

https://github.com/SFTtech/nyan/blob/master/doc/nyan.md

head over there and improve it if things are missing :)

[TheJJ]

Thanks for all the contributions, I think we can safely close this now.

Once the implementation is more or less working, we need to write the openage nyan library to provide all the NyanObjects the engine can handle. This will turn out totally awesome I hope! 👯