Tree planting, inventory GUI, resources, portals

Background and Goals

Having completed the majority of work behind the features of tree planting, resources, inventory GUI and portals we (Team 6) wanted to conduct user testing. Although our features were mostly finished, we were well aware that the game itself was very "broken" in some places. We were therefore realistic about what we were able to get out of this user testing. It would be impractical to try to evaluate how much people enjoy playing the game, for example, because at this stage, neither enemies nor players do not attack properly, removing any challenge or source of entertainment. We decided our goal was to focus on collecting specific feedback relating to any potential bugs in our features, and the usability and visual appeal of our interfaces. We designed multiple informal formative tests to generate ideas for improvement and conducted these tests over the course of a few days.

Test Design

Users

We approached testing candidates who would be the primary users for our game - i.e. people with a strong background in using computers and video games. This lead us to a group of users aged approximately 15 - 25, both males and females who have all worked or played with computers and video games. We did this because we did not want to have to spend too long explaining the game - this group of people already had a fairly good intuition regarding how to play.

Before we asked for any commitment to testing, we explained that each user would be required to interact with our game and answer subsequent questions. We explained that the session would go for about thirty minutes but that we'd probably finish early and that it would be conducted in a comfortable environment, convenient to the user.

We also gained consent for recording the users. This allowed us to review the session and gain the most out of their feedback.

Sessions

Due to availability constraints, we conducted user testing sessions separately for each user. The location of each session was decided by the user and the tester, however, all locations were very informal and relaxed. This conveyed the advantage of an intimate testing environment in which we could really focus on and get the most out of one user, without their opinions being biased by other participants.

Methodology

Sessions were conducted as follows:

1. We briefed each user on what DECO2800 is about - the fact that, as a studio of about 50 people, we were trying to build a game before the end of the semester and that our group of 5 was responsible for creating a couple of features in the game. We described our goals for the session and asked the users that they try to be as verbal as possible throughout the session - talking constantly about what they were thinking and any questions that they had.
2. Each user was read a short description of the game as found in the How to Play Wiki. We then ran the game and let them interact with the game until the users appeared to be running out of things to do (in most cases this was about a few minutes). During this stage we did not offer any advice as to what exactly the user should do, but did prompt the user to explain what they were doing. In some cases, other features broke and we had to restart the game to enable the user to keep exploring. During this phase we observed the user's behaviour and took notes regarding any issues they had/bugs they managed to cause or find.
3. We then gave them specific details about the features we were trying to gain feedback on. If a user had not managed to already work out how one of our features worked, we explained this to them. Naturally, most users had discovered our features and already had some related feedback. We noted this feedback down, asking any follow-up questions along the way. We then gave the users a bit more time to play specifically with our features, trying to get the users to continue discussion throughout.
4. Finally, we had a set of formal questions (see below) that we asked if the users had not already answered these during testing. These questions were designed to be open-ended and to promote discussion in case some of the users had not communicated their ideas on any feature in particular. Some responses required follow-up questions in order to understand what the user was trying to say - again we tried to keep the testing sessions informal to encourage free-flowing, unbiased discussion.
5. Users were thanked for their time and asked if they would like to be considered for the next round of testing. The raw recordings and results of our sessions were recorded on the Results Wiki, and Overall Outcomes were documented.

Formal Questions

General

• Did you find anything that you thought was broken or didn't work as you thought it should?
• Was there anything that you found difficult to understand during gameplay?
• What was the most distracting broken or unintuitive feature?

Tree planting

• Do you think the design of the tree planting menu suits the style of the game?
• Did you find the tree planting menu easy to use?
• Do you have any recommendations for the tree planting menu?

Inventory GUI

• Did you notice the inventory? Was it obvious how it worked initially?
• Do you like how the inventory is displayed for the user? If not would you change anything about it?

Resources

• Did you understand what resources did and why they were useful?
• What do you think about the design of the resources - are there any resource you would modify?
• Do you think the resources fit the context of each world? Would you liked to have seen any other resources?

Portals

• How would you describe the functionality of the portals?
• Does the menu for the portals match the style of the game?
• What would you improve about the portals?