Test Specification

In examining the testing that we could do on our game, we determined that there are 7 main types of testing that would be beneficial in determining the quality of our product. They are as follows:

• Unit testing
• Automation testing
• Functionality testing
• Compatibility testing
• Soak testing
• Beta testing
• Requirements testing

Below we examine each of these types of testing and document how we will use them.

##Unit testing

We will attempt to use Unity Test Tools to do unit testing on our product. In the case that these tools are not sufficient, we will develop extensions on them to test several scenarios.

A unit test framework will be vital to testing some of the core mechanics of the game. These are the tests for which we will use it:

• Switching gravity also switches the angle at which the player is displayed
• Player death event is triggered after taking the damage necessary to kill the player
• Touching a spike triggers the player damage event
• Picking up health increases the player's health if less than max
• Picking up health only increases player's health to max

##Automated testing
We hope that it will be possible to run Android automated tests against the game, as this will significantly help our testing process. We would use it for 2 main types of testing:

• Unit functionality tests
  We can use automated testing to verify that a puzzle can be solved or that a feature works properly by building a test module to handle the puzzle.

• Integration tests
  We can use the test modules to play through levels, which allows us to do integration testing on all of the pieces.

We would have to make sure that the testing modules work across all devices, so there may be extra work in building solution modules for different versions and devices. We describe our compatibility testing below.

Our team has some familiarity with Android's UIAutomator tool, which allows for automated interactions with the Android device. If this tool proves to be inadequate, we will consider other options, like Selendroid or robotium.

##Functionality testing

We have designed a test level that allows us to test each of the game play requirements. If we can beat the level, we will know that each of these requirements is met (except G1.6). If we find automation to be suitable for Android devices, we will create a script that can play through the test level. This is feasible as our game should be completely deterministic.

As listed below, our requirements can be tied to specific features of our test level:

• G1.1 - User controlled player
  This requirement is inherent in the passing of G1.2.

• G1.2 - Player can move left/right and jump
  There is a simple puzzle that requires jumps to the right and to the left

• G1.3 - Blocks that change gravity in all four directions
  There are puzzles that require the changing of gravity at particular points.

• G1.4 - Blocks that break when shot with gun
  This requirement passes with G1.11.

• G1.5 - Obtainable keys that unlock specific blocks
  There will be a puzzle that requires a key to finish the level.

• G1.6 - Spikes that hurt/kill player
  This will be tested by automation. After creating a script that can finish the level, we will create another script to cause the player to die. After death, we will run the original automation script and verify that the level is completed.

• G1.7 - Blocks that lock screen
  This requirement is tested with G1.8.

• G1.8 - When screen is locked, player jumps from one side of lock to the other We have puzzles that requires that the player wrap around the level in all four directions.

• G1.9 - When player dies, he starts at beginning of level
  This requirement will be tested with G1.6.

• G1.10 - Player can finish level
  This is determined by whether or not the level can be completed.

• G1.11 - Working gun
  There will be blocks that must be broken for completion of the level.

We will also do negative testing on some of these requirements. For example, we will test that the gun doesn't break all blocks, just the breakable ones. We will use automation testing to write a script that shoots unbreakable blocks and tries to go through. If the script still finishes the level, our requirements are met.

See Test and Introduction Level for a view of a preliminary design for the test level.

For milestone 2 game play requirements, we will create a second test level that tests the new requirements:

• G2.1 - Player can manually control vertical gravity
  The level will have no gravity blocks, but will still require switching of gravity to be completed.

• G2.2 - Player can manually control horizontal gravity
  This will pass with G2.1.

• G2.3 - Player can manually lock screen scroll
  The level will require player wrapping around the screen without having locking blocks.

• G2.4 - Mechanic for next region
  At present, this is undecided, so we will plan for how to test it when it is decided.

• G2.5 - Player can progress through levels
  We will verify that finishing the first test level will advance the player to the second.

A link to the design for this new test level will be added here when it is available.

##Compatibility testing We will do compatibility testing to determine whether or not our product works on all Android devices. We will guarantee our device works on all devices after Android 4.0 (Ice Cream Sandwich), and analyze how well it works on previous devices for possible expansion.

We will do this by developing and testing on Android 4.4 and 5.0 physical devices, but since there are so many devices that run Android, it will not be possible to test solely on physical devices. As such, we will use an emulator like Genymotion to allow testing of different types and versions of devices. Given that we will have automated scripts, we will run them against all of the different devices in our analysis.

##Soak testing Soak tests, or tests of running a program for an extended amount of time, are used to catch memory leaks. We plan to use them with automation to test the following scenarios:

• Leave game running for hours on the start screen
• Character in a level not moving
• Character in a level holding a button
• Character in a level continually jumping
• Character in a level continually dying
• Character in a level falling

##Beta testing After implementing the aforementioned testing plans and completing the second milestone (or the levels ported over from the original game), we will do a beta test of the game by allowing a small group of about 100 people install and play the game. We will follow a MapReduce model of these tests to prevent duplicates, as described below:

• Mappers report bugs to their assigned reducer.
• Reducers compare bugs and combine them into a smaller set of bugs that are then reported and tracked on the issue board; reducers assign bugs to specific developers.

We will assign 2 or 3 team members to the reducer role, as bug count requires.

##Requirements testing

The following requirements will be tested using the tools listed above:

• L1.1 - Create instructions/test level
  This will be inherent in game play tests listed above.

• L1.2 - Design puzzles using newly discussed possible mechanics
  Once we decide on which mechanics to implement, we will verify that they work using automation, as described above

• E2.1 - Port initial 10 levels from old version
  We will attempt to use automation to play through each of the original levels and as such verify that they exist in the new version

Some requirements cannot be tested using the automated tools, as they are more of a verification type of testing. For these, we will have a suite of manual test cases. Listed below are the requirements that will require manual testing and verification.

Level Design

• L2.1 - Design for next region using new mechanics, plugging in puzzles previously designed

Storyline

• S1.1 - Create a story line
• S1.2 - Use of existing levels makes sense
• S1.3 - Scaleable - Makes addition of new regions make sense
• S1.4 - Write script for intro and initial 10 levels

• ---

• S2.1 - Write script for next region

Environment

• E2.2 - Decide on mechanic for next region
• E2.3 - Design new "regions" with several levels in each
• E2.4 - Design upgrades that suit environment

Art

• A1.1 - Art for intro screen
• A1.2 - Changes to original sprites for initial 10 levels
• A1.3 - Concept for each of the possible regions
• A1.4 - New basic sprites for new regions

• ---

• A2.1 - In-depth art for next region