From 6252b19e4d8ce2912c97ad096b9ae94afb7b59ff Mon Sep 17 00:00:00 2001 From: Jeffrey Shih <34355042+unityjeffrey@users.noreply.github.com> Date: Thu, 12 Nov 2020 14:04:54 -0800 Subject: [PATCH 1/6] Change title --- com.unity.ml-agents.extensions/Documentation~/Match3.md | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-) diff --git a/com.unity.ml-agents.extensions/Documentation~/Match3.md b/com.unity.ml-agents.extensions/Documentation~/Match3.md index 74b1ee9e5b..854f36983e 100644 --- a/com.unity.ml-agents.extensions/Documentation~/Match3.md +++ b/com.unity.ml-agents.extensions/Documentation~/Match3.md @@ -1,7 +1,9 @@ -# Match-3 Game Support +# Example Match-3 with ML-Agents + + +## Overview We provide some utilities to integrate ML-Agents with Match-3 games. - ## AbstractBoard class The `AbstractBoard` is the bridge between ML-Agents and your game. It allows ML-Agents to From 1800a3b1346d1cc96a9120acbac88615a1d2c996 Mon Sep 17 00:00:00 2001 From: Jeffrey Shih <34355042+unityjeffrey@users.noreply.github.com> Date: Thu, 12 Nov 2020 14:32:05 -0800 Subject: [PATCH 2/6] added overvierw, cleaned up structure. --- .../Documentation~/Match3.md | 36 +++++++++++++------ 1 file changed, 25 insertions(+), 11 deletions(-) diff --git a/com.unity.ml-agents.extensions/Documentation~/Match3.md b/com.unity.ml-agents.extensions/Documentation~/Match3.md index 854f36983e..864e77cef3 100644 --- a/com.unity.ml-agents.extensions/Documentation~/Match3.md +++ b/com.unity.ml-agents.extensions/Documentation~/Match3.md @@ -1,11 +1,25 @@ -# Example Match-3 with ML-Agents +# Match-3 with ML-Agents ## Overview -We provide some utilities to integrate ML-Agents with Match-3 games. +One of the main feedback we get is for us on the team to illustrate more real game examples using ML-Agents. We are excited to provide an example implementation of Match-3 using ML-Agents and additional utilities to integrate ML-Agents with Match-3 games. -## AbstractBoard class +Our aim is to enable Match-3 teams to create intelligent player bots using deep learning in order for these player bots to test and play different Match-3 levels. This implementation is intended as a starting point and guide for teams to get started (as there are many nuances with Match-3 for training ML-Agents) and for us to iterate both on the C#, hyperparameters, and trainers to improve Match-3 for ML-Agents. + +This implementation includes: + +* C# implementation catered toward a Match-3 setup including concepts around encoding for moves in [Human Like Playtesting with Deep Learning](https://www.researchgate.net/publication/328307928_Human-Like_Playtesting_with_Deep_Learning) +* An example Match-3 scene with ML-Agents implemented (located under /Project/Assets/ML-Agents/Examples/match3) +* Python training code for training a Match-3 agent + +If you are a Match-3 developer and are trying to leverage ML-Agents to + +## Getting started + +## Technical specifications + +### AbstractBoard class The `AbstractBoard` is the bridge between ML-Agents and your game. It allows ML-Agents to * ask your game what the "color" of a cell is * ask whether the cell is a "special" piece type or not @@ -16,27 +30,27 @@ These are handled by implementing the `GetCellType()`, `IsMoveValid()`, and `Mak The AbstractBoard also tracks the number of rows, columns, and potential piece types that the board can have. -#### `public abstract int GetCellType(int row, int col)` +##### `public abstract int GetCellType(int row, int col)` Returns the "color" of piece at the given row and column. This should be between 0 and NumCellTypes-1 (inclusive). The actual order of the values doesn't matter. -#### `public abstract int GetSpecialType(int row, int col)` +##### `public abstract int GetSpecialType(int row, int col)` Returns the special type of the piece at the given row and column. This should be between 0 and NumSpecialTypes (inclusive). The actual order of the values doesn't matter. -#### `public abstract bool IsMoveValid(Move m)` +##### `public abstract bool IsMoveValid(Move m)` Check whether the particular `Move` is valid for the game. The actual results will depend on the rules of the game, but we provide the `SimpleIsMoveValid()` method that handles basic match3 rules with no special or immovable pieces. -#### `public abstract bool MakeMove(Move m)` +##### `public abstract bool MakeMove(Move m)` Instruct the game to make the given move. Returns true if the move was made. Note that during training, a move that was marked as invalid may occasionally still be requested. If this happens, it is safe to do nothing and request another move. -## Move struct +### Move struct The Move struct encapsulates a swap of two adjacent cells. You can get the number of potential moves for a board of a given size with. `Move.NumPotentialMoves(NumRows, NumColumns)`. There are two helper functions to create a new `Move`: @@ -45,7 +59,7 @@ iterate over all potential moves for the board by looping from 0 to `Move.NumPot * `public static Move FromPositionAndDirection(int row, int col, Direction dir, int maxRows, int maxCols)` creates a `Move` from a row, column, and direction (and board size). -## `Match3Sensor` and `Match3SensorComponent` classes +#### `Match3Sensor` and `Match3SensorComponent` classes The `Match3Sensor` generates observations about the state using the `AbstractBoard` interface. You can choose whether to use vector or "visual" observations; in theory, visual observations should perform better because they are 2-dimensional like the board, but we need to experiment more on this. @@ -53,14 +67,14 @@ better because they are 2-dimensional like the board, but we need to experiment A `Match3SensorComponent` generates a `Match3Sensor` at runtime, and should be added to the same GameObject as your `Agent` implementation. You do not need to write any additional code to use them. -## `Match3Actuator` and `Match3ActuatorComponent` classes +#### `Match3Actuator` and `Match3ActuatorComponent` classes The `Match3Actuator` converts actions from training or inference into a `Move` that is sent to` AbstractBoard.MakeMove()` It also checks `AbstractBoard.IsMoveValid` for each potential move and uses this to set the action mask for Agent. A `Match3ActuatorComponent` generates a `Match3Actuator` at runtime, and should be added to the same GameObject as your `Agent` implementation. You do not need to write any additional code to use them. -# Setting up match-3 simulation +## Setting up Match-3 simulation * Implement the `AbstractBoard` methods to integrate with your game. * Give the `Agent` rewards when it does what you want it to (match multiple pieces in a row, clears pieces of a certain type, etc). From 0c743332c922fea54e5ed3580ab2ef8865021229 Mon Sep 17 00:00:00 2001 From: Jeffrey Shih <34355042+unityjeffrey@users.noreply.github.com> Date: Thu, 12 Nov 2020 15:19:06 -0800 Subject: [PATCH 3/6] Updated with getting started, clean up --- .../Documentation~/Match3.md | 17 +++++++++++------ 1 file changed, 11 insertions(+), 6 deletions(-) diff --git a/com.unity.ml-agents.extensions/Documentation~/Match3.md b/com.unity.ml-agents.extensions/Documentation~/Match3.md index 864e77cef3..b7baa3eeb4 100644 --- a/com.unity.ml-agents.extensions/Documentation~/Match3.md +++ b/com.unity.ml-agents.extensions/Documentation~/Match3.md @@ -3,21 +3,26 @@ ## Overview -One of the main feedback we get is for us on the team to illustrate more real game examples using ML-Agents. We are excited to provide an example implementation of Match-3 using ML-Agents and additional utilities to integrate ML-Agents with Match-3 games. +One of the main feedback we get is to illustrate more real game examples using ML-Agents. We are excited to provide an example implementation of Match-3 using ML-Agents and additional utilities to integrate ML-Agents with Match-3 games. -Our aim is to enable Match-3 teams to create intelligent player bots using deep learning in order for these player bots to test and play different Match-3 levels. This implementation is intended as a starting point and guide for teams to get started (as there are many nuances with Match-3 for training ML-Agents) and for us to iterate both on the C#, hyperparameters, and trainers to improve Match-3 for ML-Agents. +Our aim is to enable Match-3 teams to leverage ML-Agents to create player agents to learn and play different Match-3 levels. This implementation is intended as a starting point and guide for teams to get started (as there are many nuances with Match-3 for training ML-Agents) and for us to iterate both on the C#, hyperparameters, and trainers to improve ML-Agents for Match-3. This implementation includes: * C# implementation catered toward a Match-3 setup including concepts around encoding for moves in [Human Like Playtesting with Deep Learning](https://www.researchgate.net/publication/328307928_Human-Like_Playtesting_with_Deep_Learning) * An example Match-3 scene with ML-Agents implemented (located under /Project/Assets/ML-Agents/Examples/match3) -* Python training code for training a Match-3 agent -If you are a Match-3 developer and are trying to leverage ML-Agents to +If you are a Match-3 developer and are trying to leverage ML-Agents for this scenario, [we want to hear from you](LINK TO GOOGLE FORM). Additionally, we are also looking for interested Match-3 teams to speak with us for 45 minutes. If you are interested, please indicate that in the [form](LINK TO FORM). If selected, we will provide gift cards as a token of appreciation. + +## Interested in more game templates? +Do you have a type of game you are interested for ML-Agents? If so, please post a [forum issue](https://forum.unity.com/forums/ml-agents.453/) with [GAME TEMPLATE] in the title. ## Getting started +The C# code for Match-3 exists inside of the extensions package (com.unity.ml-agents.extensions). A good first step would be to familiarize with the extensions package by reading the document [here](com.unity.ml-agents.extensions.md). The second step would be to take a look at how we have implemented the C# code in the example Match-3 scene (located under /Project/Assets/ML-Agents/Examples/match3). Once you have some familiarity, then the next step would be to implement the C# code for Match-3 from the extensions package. + +Additionally, see below for additional technical specifications on the C# code for Match-3. -## Technical specifications +## Technical specifications for Match-3 with ML-Agents ### AbstractBoard class The `AbstractBoard` is the bridge between ML-Agents and your game. It allows ML-Agents to @@ -74,7 +79,7 @@ It also checks `AbstractBoard.IsMoveValid` for each potential move and uses this A `Match3ActuatorComponent` generates a `Match3Actuator` at runtime, and should be added to the same GameObject as your `Agent` implementation. You do not need to write any additional code to use them. -## Setting up Match-3 simulation +### Setting up Match-3 simulation * Implement the `AbstractBoard` methods to integrate with your game. * Give the `Agent` rewards when it does what you want it to (match multiple pieces in a row, clears pieces of a certain type, etc). From cf84b59f378182f84c4ba77016b815a57f4b9768 Mon Sep 17 00:00:00 2001 From: Jeffrey Shih <34355042+unityjeffrey@users.noreply.github.com> Date: Thu, 12 Nov 2020 16:13:59 -0800 Subject: [PATCH 4/6] added link to form. added note about playable --- com.unity.ml-agents.extensions/Documentation~/Match3.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/com.unity.ml-agents.extensions/Documentation~/Match3.md b/com.unity.ml-agents.extensions/Documentation~/Match3.md index b7baa3eeb4..9fd4ee5b8e 100644 --- a/com.unity.ml-agents.extensions/Documentation~/Match3.md +++ b/com.unity.ml-agents.extensions/Documentation~/Match3.md @@ -12,7 +12,7 @@ This implementation includes: * C# implementation catered toward a Match-3 setup including concepts around encoding for moves in [Human Like Playtesting with Deep Learning](https://www.researchgate.net/publication/328307928_Human-Like_Playtesting_with_Deep_Learning) * An example Match-3 scene with ML-Agents implemented (located under /Project/Assets/ML-Agents/Examples/match3) -If you are a Match-3 developer and are trying to leverage ML-Agents for this scenario, [we want to hear from you](LINK TO GOOGLE FORM). Additionally, we are also looking for interested Match-3 teams to speak with us for 45 minutes. If you are interested, please indicate that in the [form](LINK TO FORM). If selected, we will provide gift cards as a token of appreciation. +If you are a Match-3 developer and are trying to leverage ML-Agents for this scenario, [we want to hear from you](https://forms.gle/TBsB9jc8WshgzViU9). Additionally, we are also looking for interested Match-3 teams to speak with us for 45 minutes. If you are interested, please indicate that in the [form](https://forms.gle/TBsB9jc8WshgzViU9). If selected, we will provide gift cards as a token of appreciation. ## Interested in more game templates? Do you have a type of game you are interested for ML-Agents? If so, please post a [forum issue](https://forum.unity.com/forums/ml-agents.453/) with [GAME TEMPLATE] in the title. @@ -20,7 +20,7 @@ Do you have a type of game you are interested for ML-Agents? If so, please post ## Getting started The C# code for Match-3 exists inside of the extensions package (com.unity.ml-agents.extensions). A good first step would be to familiarize with the extensions package by reading the document [here](com.unity.ml-agents.extensions.md). The second step would be to take a look at how we have implemented the C# code in the example Match-3 scene (located under /Project/Assets/ML-Agents/Examples/match3). Once you have some familiarity, then the next step would be to implement the C# code for Match-3 from the extensions package. -Additionally, see below for additional technical specifications on the C# code for Match-3. +Additionally, see below for additional technical specifications on the C# code for Match-3. Please note the Match-3 game isn't human playable as implemented and can be only played via training. ## Technical specifications for Match-3 with ML-Agents From b50b4de5c4538d8b4b2c3c28dc41f6cbaa693a81 Mon Sep 17 00:00:00 2001 From: Jeffrey Shih <34355042+unityjeffrey@users.noreply.github.com> Date: Thu, 12 Nov 2020 16:14:33 -0800 Subject: [PATCH 5/6] clean up --- com.unity.ml-agents.extensions/Documentation~/Match3.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/com.unity.ml-agents.extensions/Documentation~/Match3.md b/com.unity.ml-agents.extensions/Documentation~/Match3.md index 9fd4ee5b8e..51240222f5 100644 --- a/com.unity.ml-agents.extensions/Documentation~/Match3.md +++ b/com.unity.ml-agents.extensions/Documentation~/Match3.md @@ -9,7 +9,7 @@ Our aim is to enable Match-3 teams to leverage ML-Agents to create player agents This implementation includes: -* C# implementation catered toward a Match-3 setup including concepts around encoding for moves in [Human Like Playtesting with Deep Learning](https://www.researchgate.net/publication/328307928_Human-Like_Playtesting_with_Deep_Learning) +* C# implementation catered toward a Match-3 setup including concepts around encoding for moves based on [Human Like Playtesting with Deep Learning](https://www.researchgate.net/publication/328307928_Human-Like_Playtesting_with_Deep_Learning) * An example Match-3 scene with ML-Agents implemented (located under /Project/Assets/ML-Agents/Examples/match3) If you are a Match-3 developer and are trying to leverage ML-Agents for this scenario, [we want to hear from you](https://forms.gle/TBsB9jc8WshgzViU9). Additionally, we are also looking for interested Match-3 teams to speak with us for 45 minutes. If you are interested, please indicate that in the [form](https://forms.gle/TBsB9jc8WshgzViU9). If selected, we will provide gift cards as a token of appreciation. From 83be436a9c419dd7f3083a601fb9ebbc349daebd Mon Sep 17 00:00:00 2001 From: Jeffrey Shih <34355042+unityjeffrey@users.noreply.github.com> Date: Fri, 13 Nov 2020 09:40:30 -0800 Subject: [PATCH 6/6] Update Match3.md --- com.unity.ml-agents.extensions/Documentation~/Match3.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/com.unity.ml-agents.extensions/Documentation~/Match3.md b/com.unity.ml-agents.extensions/Documentation~/Match3.md index 51240222f5..32c4efe7bc 100644 --- a/com.unity.ml-agents.extensions/Documentation~/Match3.md +++ b/com.unity.ml-agents.extensions/Documentation~/Match3.md @@ -10,7 +10,7 @@ Our aim is to enable Match-3 teams to leverage ML-Agents to create player agents This implementation includes: * C# implementation catered toward a Match-3 setup including concepts around encoding for moves based on [Human Like Playtesting with Deep Learning](https://www.researchgate.net/publication/328307928_Human-Like_Playtesting_with_Deep_Learning) -* An example Match-3 scene with ML-Agents implemented (located under /Project/Assets/ML-Agents/Examples/match3) +* An example Match-3 scene with ML-Agents implemented (located under /Project/Assets/ML-Agents/Examples/Match3) If you are a Match-3 developer and are trying to leverage ML-Agents for this scenario, [we want to hear from you](https://forms.gle/TBsB9jc8WshgzViU9). Additionally, we are also looking for interested Match-3 teams to speak with us for 45 minutes. If you are interested, please indicate that in the [form](https://forms.gle/TBsB9jc8WshgzViU9). If selected, we will provide gift cards as a token of appreciation. @@ -18,7 +18,7 @@ If you are a Match-3 developer and are trying to leverage ML-Agents for this sce Do you have a type of game you are interested for ML-Agents? If so, please post a [forum issue](https://forum.unity.com/forums/ml-agents.453/) with [GAME TEMPLATE] in the title. ## Getting started -The C# code for Match-3 exists inside of the extensions package (com.unity.ml-agents.extensions). A good first step would be to familiarize with the extensions package by reading the document [here](com.unity.ml-agents.extensions.md). The second step would be to take a look at how we have implemented the C# code in the example Match-3 scene (located under /Project/Assets/ML-Agents/Examples/match3). Once you have some familiarity, then the next step would be to implement the C# code for Match-3 from the extensions package. +The C# code for Match-3 exists inside of the extensions package (com.unity.ml-agents.extensions). A good first step would be to familiarize with the extensions package by reading the document [here](com.unity.ml-agents.extensions.md). The second step would be to take a look at how we have implemented the C# code in the example Match-3 scene (located under /Project/Assets/ML-Agents/Examples/match3). Once you have some familiarity, then the next step would be to implement the C# code for Match-3 from the extensions package. Additionally, see below for additional technical specifications on the C# code for Match-3. Please note the Match-3 game isn't human playable as implemented and can be only played via training.