Testing & Balancing
-
Test Schedule
- Schedule
- Which Tests:
- UX Tests
- Functionality Tests
- Surveys / Questionnaires
-
Types of Tests
- Test Tree
- Combinatorial Tests
- Test Flow Diagrams
-
Discussion Turn-Based Combat
- Combat System & Basic Rules
- Damage Calculation
- Weapons, Gear & Items
-
Surveys / Questionnaires (+ as planned per schedule)
- Insert survey evaluation here
We split our testing phases up into three categories of tests: UX tests, surveys / questionnaires and functionality tests. Our UX tests are used to test the most important features, mainly the level design, combat system and later on the full game. Surveys and questionnaires are much more versatile since we can get feedback on virtually any component of the game such as the game’s art style and narrative, GUI and graphic design. We also chose to send out a larger demo of our first level to an online audience to gather more general insights than we could locally. Last but not least, Functionality tests don’t necessarily require an external tester but are important for the game’s overall quality. We hope to cover as much ground with the different testing methods.
19.07. Level Testing
01.08. Kampfsystem
09.08. Graphics,Animations & UI
25.08. All components
*Will likely be postponed.
19.07. Art Style
09.08. Graphics,Animations & GUI
20.08. Level Design
Functionality tests are held once a feature has been implemented. The results are used to further iterate on the corresponding feature.
David Baier
The test tree is used to display the variety of different elements of the game that have to be tested. It's used to get an overview to make sure that tests are developed for all features that require them. For each category questions are formulated that are used to design the tests. Each feature is then assigned to a type of test.
Options
Do the Settings do what they are supposed to do?
Do the settings get saved when closing the game?
User Interface
Do all the buttons on the Interface work?
Are the hitboxes of the buttons correct?
Weapons
Are all the stats of the weapons correct?
Do the weapons deal the right amount of damage?
Do weapons get correctly equipped after looting them?
Gear
Are all the stats of the gear correct?
Do Consumables work correctly?
Interactions
Do all the buttons work?
What happens when multiple buttons are pressed at once?
We use combinatory tests to test how different components of the game interact with each other. These tests will usually be done by developers. These tests will be repeated every time a component that influences the test is changed.
The tester first loads the part of the game described in the preconditions and then repeats the instructions. If the game behaves as intended, no detailed description is required if not the tester describes the behaviour of the game as detailed as necessary. After this the box for the specific test is ticked and the tester proceeds to the next test.
The results of these tests are then discussed by the developers. Issues are ranked based on relevance and forwarded to the responsible developers.
While Combinatorial tests work great for features that react to a single player input, we use Flow diagrams to test features that receive different kinds of input like the combat system. During combat the player constantly makes choices, chooses the type of attack, targets a part of the enemy and uses consumables. The flow charts show all these choices and allow the tester to keep track of all possible paths and make sure that all are tested.
One flow diagram for example is used to show how damage is calculated during combat encounters. The tester can use the diagram to follow along and calculate the damage and compare it to the damage dealt in game. The diagram is also used to test if the correct actions happen in the right order. If, for example, A defending limb only defends the targeted limb and not the core.
David Sinzker
In Assemblia, players build their loadouts by smashing together the different components of other toys to win. We’re dealing with a symmetrical combat system where both the player and the enemy are treated equally. Both combatants can attack their opponent’s body parts (limbs, gear, core) but they can also protect their own body parts. One more important factor is that players don’t get to keep their best loadout. After each battle, they’re required to exchange one of their weapons with one of those of the opponent. We’re hoping to lead players away from meta-gaming that may hurt the puzzle factor of the game’s battles.
1. Opponents take turns.
When the player has chosen their chain of actions, they execute them. Should the opponent still stand after the player’s turn, they will retaliate with their own chain of actions.
2. Actions require AP
Every weapon the player picks up can do a single attack or defend another body part for a certain amount of AP.
3. Players loadout
Players can equip a single weapon to their left / right arms and a piece of supportive gear to their chest.
The combat system requires us to dive into combinatorial testing of the kinds of weapon loadouts the player can use since everything has to be as viable as possible. We have to test enemies against the kinds of loadouts the player can use and decide later on whether an enemy or an underperforming loadout needs to be adjusted. Lots of pushing around values.
Attack on core:
body_health - dmg * (1 - remaining resistance)
Attack on body part:
limb_health - dmg
Attack on guarded body part:
limb_health - dmg * (1 - bodypart_def)
Currently, we’ve set up a table to keep track of the types of enemies and weapons we’ll implement. Everything from health, fighting style, equipment to the internal IDs are saved here. We’re hoping the mammoth task of balancing loadouts against enemies can be assisted by keeping a clear overview on the relations between individual weapons.
David Baier
- Enemy Design
- Graphics & GUI UX Test
- Level Design UX Test (itch.io)
At first we decided which segment of our game should be tested. Here in the graphics presented we are testing our Design both in Art such as Character art or environment art but also Design as in level design.
Our goal was to see if our planned design worked out as a concept but also if we managed to implement it in our art style and created assets.
For that we created surveys which we later handed out to testers. The testers consisted of friends and fellow students who all would be considered as casual gamers if not more than that. For our Art and Design survey we decided to separate it into two sections. World Design and Character Design.
In the World Design section we mainly asked questions about whether or not we managed to implement the setting in our design but also about look and feel and level size.
And in the Character Design section we took a look at our characters in an overall picture comparing them to one another but we also looked at our already finished characters by themselves, their design and animation but also if different designs for different occasions for the same character can be easily connected.
At the bottom of our survey we also decided to ask our participants to give us some textual feedback where we wanted to gather some suggestions for improvement.
For our “questions” we used statements. The testers then were asked to estimate how fitting our statement is according to the graphic displayed.
The participants were able to choose between option 1 to 5 with 1 being the statement does not apply to 5 being the statement does apply and 3 being a neutral answer.
Before we get into the evaluation we describe the process and how we evaluate.
We decided to use a modified version of the SUS-score evaluation. Every user response will be subtracted by one. This scales all values from 0 to 4 with four now being the most positive response. We then add up all the responses for each user and multiply that total by 2,5. This converts the range of possible values from 0 to 100 instead of from 0 to 40. However this would only work for 10 questions in the survey. Our survey consists of 20 questions which is why we divide the end score with 2. We then added all of the scores together and divided them by the amount of participants.
We chose to use a modified version of the SUS score because of it being a reliable and valid measure of perceived usability. On top of that we have a sorta percentile ranking system that lets us set ourselves a goal which rank we want to achieve.
Our goal was to at least achieve an above average score of over 68.
Almost every statement is set to a positive key. That means that when evaluating the question a high score is better than a lower score like 1 or 2. However there is one question with a negative key that reverses the chosen answer during evaluation.
For our evaluation we simply used charts.
Here are the results of our first survey:
And here is the evaluation:
With our first survey we achieved a SUS score of 77 which is above the average. According to the percentile ranking we would have achieved a B so there is still room for improvement.
What we learn from our first survey is that we will have to adjust our design a bit regarding implementing our setting a bit better especially in the environment. Our Slum level is not clearly designed to be identified as such. The bazaar was not recognized. However our character design received only positive scores. The only thing we might have to adjust is to implement our toy setting a bit better.
We will also try to take every fitting textual feedback to heart and try to improve our design with the help of the suggestions. A follow up survey will be focused on the points that have received mainly negative responses.
Marcel Stürzenberger