Tokenization and Execution

Tokenization and Execution

A Foreword on Vocabulary

Hello! My name is Mint, and I'm @WispyMouse. I'm sort-of winging it, always. Here, I might not use the words that are correct. Please gently let me know if I do, or if you think there's more accurate words to use. In the meantime, let's proceed with this article as though I have total power over the meanings of words.

What is Tokenization?

Tokenization is the term for taking a string of text and turning it into something the computer program can understand and utilize. The designers of the game write down Token Text that gets parsed into

In Project Space Deck, Token Text is a string of text containing some number of Token Statements. For example:

• [SETTARGET:FOE][DAMAGE:10][APPLYSTATUSEFFECTSTACKS: 1 POISON]

The above text is Token Text containing three Token Statements. These contain references to Scripting Commands as well as arguments for those commands.

Note

• All Token Statements start with [ and end with ].
• There's a catalogue of Scripting Commands. Token Statements contain a reference to exactly one Scripting Command.
  • In the Token Text [SETTARGET:FOE], the Scripting Command being utilized here is SETTARGET.
  • Scripting Commands can be aliases for multiple other Scripting Commands. Scripting Commands usually have one or more strings of text that uniquely identify them, apart from every other Scripting Command in the Database.
• Token Statements can optionally have arguments. [SETTARGET:FOE] has one argument of FOE. [APPLYSTATUSEFFECTSTACKS: 1 POISON] has two arguments, 1 and POISON.
  • Arguments are always separated by spaces . Spaces are only used to separate arguments, and can optionally be used after :. It has no impact on the Parsed Token.
• Without an argument, a Token Statement might look like [FINISH]. With no arguments, there is no : present.

During Tokenization, the Token Text is broken into each Token Statement. These are paired with the appropriate Scripting Commands and supply their arguments. At this stage we now have a set of Parsed Tokens; a combination of Scripting Command and arguments in a list, but without any context. [DAMAGE: COUNTCURRENCY_GOLD] is Token Text that will result in a Parsed Token that knows someone is going to get hurt, but not who or how much. This set of Parsed Tokens is a Parse Tree; they'll be referred to as a ParsedTokenSet to match the planned code.

Cards, status effects, events, etc., will all form their ParsedTokenSet when necessary, usually at the start of the runtime. Cards with these ParsedTokenSet can then be duplicated, modified, and utilized with context provided later.

The utility of Tokenization is to allow designers to use parseable, actual language words to express gameplay concepts. These concepts are then Tokenized into easy to identify paired Scripting Commands, made in C#. This layer of abstraction and interfacing will allow designers to express their intent without getting muddied down in the how. The resulting ParseTokenSet are reusable and can be changed by the effects of other cards, or executed in a delayed manner to allow time for animations to play.

What is Execution?

Consider the sentence: "I have a card I want to play on an enemy now." This is a sentence with Context. Execution is when you supply a ParsedTokenSet with Context. Returning to this example:

• [SETTARGET:FOE][DAMAGE:10][APPLYSTATUSEFFECTSTACKS: 1 POISON]

This requires that you provide a means to identify a Foe. By default, the UX of the game interprets that as "click on the foe to use this on" for the player, but the enemy just targets the player automatically. Now that the foe has been identified, it's able to have a damage event come its way, as well as being poisoned.

Scripting Commands are aware of what arguments they require, and arguments are often aware of what user input is required. A ParsedTokenSet provides a system for providing the needed Context. The result is a ExecutableTokenSet, a set of ExecutableTokens that are instructions ready to fire. Execution is applying this ExecutableToken to the gamestate.

This is the most simple execution of a ParsedTokenSet. It's common for things to branch off in to conditions, or add additional required user input.

![NOTE] TODO!

• What are the names of the classes and functions involved with this process? Relate every concept here to its exact programming counterparts. Link to them in the repository.
• Provide an example of going through the entire flow, in code.
• How does the ParsedTokenSet know what Context it needs?
• Describe the execution engine of a ParsedTokenSet, and give it a name. Describe how a ParsedTokenSet takes in a Context and what the classes involved in that transaction are. Describe what happens when additional context becomes required.
• Name the identifier for a token, or whatever we want to call it.
• The above concept names do not match entirely to what is currently used in the repository. This is a work in progress.
• Describe GameStateDelta, etc., as a way to describe the happenings of a script.