HoloLens-ElectricalMuscleStimulation

This repository provides you code to replicate the AR experiences in the paper Adding Force Feedback to Mixed Reality Experiences and Games using Electrical Muscle Stimulation (CHI'18), which you can read here. Also here is the accompanying video.

Structure of this repository

This repository has the two core folders:

1. Assets: pretty much everything lives in here. See the following:

   1.1 Assets/Root/Server/Scenes/Server.unity the Unity3D app that run in the server (a laptop connected to the stimulator) is at

   1.2 Assets/Root/Applications/Scenes/ has four HoloLens applications you see in our video: labyrinth.unity (a marble balancing game to be played with a physical prop), Walkthrough.unity (a room planning application with a couch, lamp, button and thermostat), Escape.unity(and AR escape room!) and Userstudy.unity(if you want to replicate our study conditions).

2. ProjectSettings: Just settings.

Dependencies

Software Dependencies

This will require:

1. Unity 5.5.1f (other versions might work with modifications)
2. Vuforia Plugin for marker tracking (you can read here how to install Vuforia for Unity)
3. A working HoloLens installation with the HoloLens []toolchain](https://developer.microsoft.com/en-us/windows/mixed-reality/install_the_tools) (Windows 10, Visual Studio, etc)
4. A HoloLens Mixed Reality Headset connected to your Wifi router
5. Put the Unity Assets in the right place (AssetStoreTools folder).
   • Holotoolkit 5.5.0
   • Vuforia >6.2.10
   • In Vuforia's DefaultTrackableEventHandler please change OnTrackingFound(),OnTrackingLost() to protected virtual.

Optionally, we recommend:

• using our AR marker generator to quickly create a batch of markers (for hands, objects, etc) instead of manually creating every marker.
• Note: we removed the asset (3D object and scripts) of the catapult you see in our video because this is a commercial asset we purchased (feel free to do the same, it is available as trebuchet on the Unity Asset store!).

Hardware Requirements

This project was built for a particular EMS device (Rehastim V1, which is off the market, sorry!). However, you can easily modify it to run on your custom EMS device (as long as it accepts USB commands) or using an open-source EMS controller such as openEMSstim. If you are modifying the EMS_server to run with another EMS device, read this first.

Since the HoloLens we used does not support a USB connection (for the muscle stimulator) we used an additional laptop running as a server (hence the Assets/Root/Server/Scenes/Server.unity that yoy must run on your laptop). See the paper for how to pack this is a convenient backpack form factor.

1. Running the examples

1. Clone this repository by running git clone https://github.com/PedroLopes/HoloLens-ElectricalMuscleStimulation.git (or you preferred visual tool)
2. Open the project in Visual Studio.
3. Connect your HoloLens, connect also your HoloLens to a Wifi
4. Select to deploy this project to the HoloLens (verify that the app is launched and running)
5. Open the project in your local machine (that will act as EMS_server), connect this machine to the same Wifi.
6. Setup the communication between both HoloLens <> EMS_Server. To do this you have to configure the EMS_server to receive requests from the HoloLens. <give the IP address of the EMS_server to the HoloLens app?>.
7. Deploy and Start both apps (once more double checked the IP addresses and Ports match).
8. Configure your EMS machine, calibrate it to work comfortably and Pain-free (see here for more details).

2. Testing

1. Now, launch the HoloLens app, put your hand out in the pointing gesture (to make sure it is tracked, or attach a Hand_marker to it -- review your code to make sure you supplied the right marker to Vuforia).
2. Reach your hand out and touch a virtual object (e.g., we suggest you try the couch scene first). You should feel EMS (verify that the EMS_Server received the message).
3. Hooray! (or read the FAQ below)

Calibrating your EMS MR force feedback effects

You can't simply copy someone else's EMS calibration since it varies slightly from user to user (refer to openEMSstim for details on how to calibrate or see our paper). Still, you can use these values on the table below to guide your haptic effect design:

Haptic Effect	Muscle	mA	μs	Muscle	mA	μs	Duration
Detents on Dial and Slider	Wrist rotator	15	200	-	-	-	150
Impulse/Impact (i.e., quick force in the opposite direction of motion)
Detents on Lever	Triceps	17	265	-	-	-	150
Marble hits walls	Triceps	18	290	-	-	-	300
Marble drop	Triceps	Right	17	280	Triceps Left	17	280
Spring (i.e., continuous counterforce with spring coefficient)
Button	Shoulder	27	280-350	Wrist extensor	15		100-150
Catapult	Triceps	17	200-275	-	-	-	Proportional
Friction (i.e., continuous counterforce)
Couch Static friction	Shoulder	27	100-420	Wrist extensor	15	100-200	Ramp up
Couch Kinematic Friction	Shoulder	27	300	Wrist extensor	15	120	Continous
Limits (i.e., strong counterforce that stops motion entirely)
Dial maximum position	Wrist rotator	15	265-300	-	-	-	Ramp up
Weight/Gravity (i.e., constant and continuous force)
Marble weight	Triceps Right	17	200	Triceps Left	17	200	Continuous
Marble shift	Triceps Right	17	265	Triceps Left	17	265	Proportional

p.s.: note that they might differ from the current code which is more of a demo-version.

Support

This work was kindly supported by the Hasso Plattner Institute.

Code authors

The code was authored by Sijing You (Hololens Apps, EMS Unity Server) and Pedro Lopes (EMS protocol, Marker Generator). You can contact them via github.

The paper Adding Force Feedback to Mixed Reality Experiences and Games using Electrical Muscle Stimulation (CHI'18), which you can read here was authored by Pedro Lopes, Sijing You, Alexandra Ion and Patrick Baudisch.

FAQ

1. I don't have an EMS device! What to I do?

You can get or build an openEMSstim (a piece of hardware that allows you to easily control an off the shelf muscle stimulator). All the schematics and code are available.

2. I don't feel EMS

1. Look at your EMS_Server, did it get the message from the HoloLens? If not

   1. re-check the IP addresses for both devices.
   2. Make sure they are on the same network
   3. Make sure they can ping each other (in other words, make sure this network is not aggressively blocking ports/messages)

2. If the message arrived at your EMS_Server but you did not feel anything:

   1. Recheck your EMS machine by calibrating it again.

Liability

Please refer to the liability waiver (in documentation/liability_waiver.md).