000005

[thetrashy1]

Beta Testing

• There are two active paths you can take to help us beta test this experience.
• We will give you 2 unique codes to test both paths, one will be an even number and one will be an odd number.
  • If you enter in an even number you will be participating in the switch study.
  • If you enter in an odd number you will be participating in the push study.
• Please attempt to answer the following questions to the best of your ability.

Equipment Details

Please select one:

• [ x] Oculus Quest 1
• Oculus Quest 2
• Oculus Qust Rift S
• Oculus OG 🕶️

Scenario Details

Please describe the scenario as you experienced it
I was given instructions on how to interact with various instruments that operate track switching, camera switching, and the ability to stop the train. Then, different scenarios were presented in which manipulating the instruments would allow me to influence if the train killed groups of workers on the tracks or not. I think there were just 2 different choice scenarios presented but I wasn't sure what was going on in each one.

Beta Test Version

This should be 0.1.0-beta

Estimated Time

Please provide the rough time you started the experience and use time zones, e.g. 12:30am EST May 4, 2021
6:30pm EST May 12 2021

Qualtrics Code

Please enter the 6 digit survey code you used here e.g. '123456'
000005

Training Decision on Pulling the Switch

Please select one:

• [x ] Followed Directions: sent the train towards the bridge 🌉
• You Caused Trouble: sent the train towards the curve

Training Decision on Stopping the Train with the Barrel

Please select one:

• [ x] Followed Directions: pushed the barrel to stop the train
• You Caused Trouble: did not push the barrel and the system took over

Moral Control Decision

Please select one:

For EVEN survey code (Switch Study)

• You sent the train away from the people
• You did nothing and the five people died 🤯
• You did nothing and the five people lived 🐛

For ODD survey code (Push Study)

• You stopped the train with a Barrel
• [ x] You did nothing and the five people died 🤯
• You did nothing and the five people lived 🐛

Test Study Decision

Please select one:

For EVEN survey code (Switch Study)

• You pulled the switch, killed 1, saved 5
• You did not pull the switch, killed 5

For ODD survey code (Push Study)

• [x ] You pushed a person, killed 1, saved 5
• You did not push a person, killed 5

System Crash

• Did you Encounter a Crash?

Please explain what happened, do you remember if you were interacting with anything? Do you remember about where you were in the experience?
If you want to be really helpful 🥰 and you think you can recreate the crash and know how to run adb logcat please dump us the file here!
If not no worries, we still 🖤 you.

Additional Comments & Details

Any other feedback? Notes? Suggestions? Little things you noticed? Anything?! Please and thank you!😺

I found the overall experience confusing and difficult to parse. Some of this may have been user error due to my first time using an oculus quest, however throughout I found it difficult to understand what was happening in each scenario due to poor visibility of the general area and understanding the location of the tracks and the workers. I was seated during the test, and found it difficult to naturally reach all the controls. The pull lever was finicky and seemed to have delayed reaction to manipulation. At one point I was unable to switch between camera views (if this is intentional then I was not aware of what made it non-interactive).

It seemed odd that there is no explanation of the most abstract interactive item in the test, the blue position movers. I was able to assume how they worked due to prior experience with VR movement, but to less familiar testers, it might be non-intuitive. I also wonder why the computer monitor in the office room displays the number entry by default, then a voice asks you if you are seated or standing, then the monitor displays a choice of seated or standing. Why have that question asked twice be disassociated sources? Perhaps it would be more streamlined to have the seated or standing question up on the monitor by default? Seems like the user would read what is on the screen, and make that selection pretty simply, without needing to be told they are about to be asked. Probably nit-picky, but it stood out to me as a user flow issue that could easily be tweaked.

I don't quite know what happened when it came to the pushing scenario. The test seemed to suggest I pushed someone onto the tracks, but I don't recall seeing the option to do so or choosing to do so. Not exactly sure what happened there. I think my overall reaction to much of what I was doing was confusion, not so much about the overall concept or what each interactive item did, but generally what was happening outside the control room at any given moment. In this sense, I don't feel like I really made distinct choices about what to do in each situation, and I did not feel connected to the outcomes, which I was only able to infer on the assumption that the yelling audio cue meant someone died.

[JShull]

Thanks so much @thetrashy1 for testing out the application!

I have your raw data cleaned up to remove some unique information and will be building some additional tests from your contribution/submission.

Once we get some of that testing data parsed I am planning on releasing a simple web viewer app that will let you replay your motion and event data. This is probably a few weeks off.

Coming up within by next week we are planning on releasing the raw data back to the users and that will be done here with me closing out the issue. Right now this is looking like May 17th, 2021.

Specific Feedback from the Comments

• I want to thank you for the insight on your experience 💯. I personally think your input regarding the screen decision is well noted, spot on, and I am planning on incorporating those changes before we go out with the full study. I think the user flow here is clunky and it was an addition towards the end of us building the test that I threw in there and it's very obvious looking back at that now 🤦‍♂️

• It never crossed my mind that with providing a lot of up front narration that we totally missed the this is how you move what a simple oversight that I can only sum up as 'duh' and I do think we can compact that information up-front one time without having additional pop-up narrations another point well noted and I'm going to start tailoring those adjustments as you've suggested.

• Lighting is also well noted - this was something we got early feedback on when we showed a sample video a couple months ago at a conference and we are working on adjusting that by increasing the natural light in the environment, increasing the light within the control booth by modifying the textures/materials to be 'brighter' and providing an additional user options menu to increase that via a settings feature.

• Regarding the end when you heard the audio and heard a scream. I looked at your data and it looks like the system registered a collision - do you happen to remember if you were on the portal spot close to the open window where the barrels were? I am going to dive deeper on your data as I can confirm this but it will take me a little bit of time to suss that out. Something like this occured in an earlier test and I have a feeling this is a 🐛 tied to proximity of the user when our character starts to walk on the platform. So if you're looking straight out there's no real indication that our repair guy is walking up and in some previous tests if your hand just happened to be in the right spot of where the character ends up it causes a physics glitch in which the collision registers with a pretty high force value and the system thinks "oh you must have pushed this person". I am about 99% positive this is what's occuring and thank you for letting me know as I have a way to fix this.

• The not connected with the outcome piece, I'm going to let @Kisselphil think about that one 😀 We can look into some of the previous research on the older representations of these studies to get some insight on some options there, but really greatful for the feedback immediately helpful and why I really do ❤️ this process. Thank you so much for this information and I will follow back up with you later on regarding the data we did collect and will let you know when some of those modifications are in - because I'm coming right back to get more feedback!

[JShull]

Created Issue #37 as this needs to be adjusted prior to us going in on the larger study.

[Kisselphil]

This is really helpful feedback @thetrashy1 , thanks! Would you mind if I followed up in person?

[thetrashy1]

Sure, I'd be happy to.

…

On Thu, May 13, 2021 at 9:26 AM Kisselphil ***@***.***> wrote: This is really helpful feedback @thetrashy1 <https://github.com/thetrashy1> , thanks! Would you mind if I followed up in person? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#35 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AUBVOEIRQVFOCPDZEVURCSDTNPHRBANCNFSM44ZPV7UA> .

-- jefflangevin.com

[JShull]

Hey @thetrashy1 thanks for working with us - following up with your data.

You can now access the data that we pulled back from this first event here