Tokenization and Execution

Tokenization and Execution

A Foreword on Vocabulary

Hello! My name is Mint, and I'm @WispyMouse. I'm sort-of winging it, always. Here, I might not use the words that are correct. Please gently let me know if I do, or if you think there's more accurate words to use. In the meantime, let's proceed with this article as though I have total power over the meanings of words.

What is Tokenization?

Tokenization is the term for taking a string of text and turning it into something the computer program can understand and utilize. The designers of the game write down Token Text that gets parsed into

In Project Space Deck, Token Text is a string of text containing some number of Token Statements. For example:

• [SETTARGET:FOE][DAMAGE:10][APPLYSTATUSEFFECTSTACKS: 1 POISON]

The above text is Token Text containing three Token Statements. These contain references to Scripting Commands as well as arguments for those commands.

Anatomy of a Token Statement

• All Token Statements start with [ and end with ].
• There's a catalogue of Scripting Commands. Token Statements contain a reference to exactly one Scripting Command.
  • In the Token Text [SETTARGET:FOE], the Scripting Command being utilized here is SETTARGET.
  • Scripting Commands can be aliases for multiple other Scripting Commands. Scripting Commands usually have one or more strings of text that uniquely identify them, apart from every other Scripting Command in the Database.
• Token Statements can optionally have arguments. [SETTARGET:FOE] has one argument of FOE. [APPLYSTATUSEFFECTSTACKS: 1 POISON] has two arguments, 1 and POISON.
  • Arguments are always separated by spaces . Spaces are only used to separate arguments, and can optionally be used after :. It has no impact on the Parsed Token.
• Without an argument, a Token Statement might look like [FINISH]. With no arguments, there is no : present.

During Tokenization, the Token Text is broken into each Token Statement. These are paired with the appropriate Scripting Commands and supply their arguments. At this stage we now have a set of Parsed Tokens; a combination of Scripting Command and arguments in a list, but without any context. [DAMAGE: COUNTCURRENCY_GOLD] is Token Text that will result in a Parsed Token that knows someone is going to get hurt, but not who or how much. This set of Parsed Tokens is a Parse Tree; they'll be referred to as a ParsedTokenSet to match the planned code.

Cards, status effects, events, etc., will all form their ParsedTokenSet when necessary, usually at the start of the runtime. Cards with these ParsedTokenSet can then be duplicated, modified, and utilized with context provided later.

The utility of Tokenization is to allow designers to use parseable, actual language words to express gameplay concepts. These concepts are then Tokenized into easy to identify paired Scripting Commands, made in C#. This layer of abstraction and interfacing will allow designers to express their intent without getting muddied down in the how. The resulting ParsedTokenSet are reusable and can be changed by the effects of other cards, or executed in a delayed manner to allow time for animations to play.

What are Conditions?

Token Text may contain Conditions, which indicate when certain effects should be run. Consider this example:

• [REQUIRES: GOLD >= 10][SETTARGET:SELF][HEAL:10]

When a condition is provided, everything that is in the remaining Token Text will require the Condition to be met in order to run. Above the requirement is set at the start, so this entire ability might do nothing.

You can also include Conditions later. [SETTARGET:SELF][HEAL:5][REQUIRES: GOLD >= 10][SETTARGET: FOE][DAMAGE: 10] would always result in an execution that heals the user, but the rest of the effect might not run depending on the amount of gold they have.

Consider when to put Conditions at the very start of an effect, or partway through. If you want the player to be able to play a card, even if none of the effects will happen, consider putting the [SETTARGET:...] text before the Condition so that the UX is consistent.

What are Scopes?

Token Text can contain Scopes. A Scope contains inner Token Text. Scopes are a way of relating sets of Token Statements to each other, grouping them in useful ways. Consider the below example:

• [SETTARGET:FOE]{[REQUIRES: GOLD >= 10][DAMAGE: 10]}{[ELSE][DAMAGE: 1]}

First set the target, using the SetTarget Scripting Command. Then a { indicates that we're beginning a scope. Everything until the next matching closing } will be part of the same scope. Here, we have two scopes defined. The second set relates to the first by using an ELSE Scripting Command, indicating that this Scope's inner logic shouldn't be run unless the previous set wasn't.

Scopes are primarily used to branch outcomes and effects like the above, usually in relation to setting Conditions. The outermost scope is a scope behind the scenes, meaning all Token Text is guaranteed to parse alongside an owning Scope.

When you specify a Condition partway through a scope, everything in that scope going forward will have that requirement. Generally this means that for non-complicated scenarios you can not use the scope system to have effects with straightforward conditions and no extra logic.

What is Execution?

Consider the sentence: "I have a card I want to play on an enemy now." This is a sentence with Context. Execution is when you supply a ParsedTokenSet with Context. Returning to this example:

• [SETTARGET:FOE][DAMAGE:10][APPLYSTATUSEFFECTSTACKS: 1 POISON]

This requires that you provide a means to identify a Foe. By default, the UX of the game interprets that as "click on the foe to use this on" for the player, but the enemy just targets the player automatically. Now that the foe has been identified, it's able to have a damage event come its way, as well as being poisoned.

Scripting Commands are aware of what arguments they require, and arguments are often aware of what user input is required. A ParsedTokenSet provides a system for providing the needed Context. The result is a ExecutableTokenSet, a set of ExecutableTokens that are instructions ready to fire. Execution is applying this ExecutableToken to the gamestate.

This is the most simple execution of a ParsedTokenSet. It's common for things to branch off in to conditions, or add additional required user input.

How is Tokenization done in Space Deck?

Let's assume you're in a context where you're ready to Tokenize Token Text.

The tidy answer is to call SpaceDeck.Tokenization.Processing.ParsedTokenMaker.GetParsedTokensFromTokenText(baseText). This is a static function that will use the ScriptingCommandReference to match Identifiers to Scripting Commands, and supply them with arguments. The result will be a ParsedTokenSet.

Warning

TODO!

• What are the names of the classes and functions involved with this process? Relate every concept here to its exact programming counterparts. Link to them in the repository.
• Provide an example of going through the entire flow, in code.
• How does the ParsedTokenSet know what Context it needs?
• Describe the execution engine of a ParsedTokenSet, and give it a name. Describe how a ParsedTokenSet takes in a Context and what the classes involved in that transaction are. Describe what happens when additional context becomes required.
• Name the identifier for a token, or whatever we want to call it.
• The above concept names do not match entirely to what is currently used in the repository. This is a work in progress.
• Describe GameStateDelta, etc., as a way to describe the happenings of a script.