Using electrical muscle stimulation to create the missing haptics in Augmented Reality
Switch branches/tags
Nothing to show
Clone or download
Fetching latest commit…
Cannot retrieve the latest commit at this time.
Permalink
Failed to load latest commit information.
Assets
ProjectSettings
extra
.gitignore
LICENSE
README.md

README.md

HoloLens-ElectricalMuscleStimulation

This repository provides you code to replicate the AR experiences in the paper Adding Force Feedback to Mixed Reality Experiences and Games using Electrical Muscle Stimulation (CHI'18), which you can read here. Also here is the accompanying video.

Structure of this repository

This repository has the two core folders:

  1. Assets: pretty much everything lives in here. See the following:

    1.1 Assets/Root/Server/Scenes/Server.unity the Unity3D app that run in the server (a laptop connected to the stimulator) is at

    1.2 Assets/Root/Applications/Scenes/ has four HoloLens applications you see in our video: labyrinth.unity (a marble balancing game to be played with a physical prop), Walkthrough.unity (a room planning application with a couch, lamp, button and thermostat), Escape.unity(and AR escape room!) and Userstudy.unity(if you want to replicate our study conditions).

  2. ProjectSettings: Just settings.

Dependencies

Software Dependencies

This will require:

  1. Unity 5.5.1f (other versions might work with modifications)
  2. Vuforia Plugin for marker tracking (you can read here how to install Vuforia for Unity)
  3. A working HoloLens installation with the HoloLens []toolchain](https://developer.microsoft.com/en-us/windows/mixed-reality/install_the_tools) (Windows 10, Visual Studio, etc)
  4. A HoloLens Mixed Reality Headset connected to your Wifi router
  5. Put the Unity Assets in the right place (AssetStoreTools folder).

Optionally, we recommend:

  • using our AR marker generator to quickly create a batch of markers (for hands, objects, etc) instead of manually creating every marker.
  • Note: we removed the asset (3D object and scripts) of the catapult you see in our video because this is a commercial asset we purchased (feel free to do the same, it is available as trebuchet on the Unity Asset store!).

Hardware Requirements

This project was built for a particular EMS device (Rehastim V1, which is off the market, sorry!). However, you can easily modify it to run on your custom EMS device (as long as it accepts USB commands) or using an open-source EMS controller such as openEMSstim. If you are modifying the EMS_server to run with another EMS device, read this first.

Since the HoloLens we used does not support a USB connection (for the muscle stimulator) we used an additional laptop running as a server (hence the Assets/Root/Server/Scenes/Server.unity that yoy must run on your laptop). See the paper for how to pack this is a convenient backpack form factor.

1. Running the examples

  1. Clone this repository by running git clone https://github.com/PedroLopes/HoloLens-ElectricalMuscleStimulation.git (or you preferred visual tool)
  2. Open the project in Visual Studio.
  3. Connect your HoloLens, connect also your HoloLens to a Wifi
  4. Select to deploy this project to the HoloLens (verify that the app is launched and running)
  5. Open the project in your local machine (that will act as EMS_server), connect this machine to the same Wifi.
  6. Setup the communication between both HoloLens <> EMS_Server. To do this you have to configure the EMS_server to receive requests from the HoloLens. <give the IP address of the EMS_server to the HoloLens app?>.
  7. Deploy and Start both apps (once more double checked the IP addresses and Ports match).
  8. Configure your EMS machine, calibrate it to work comfortably and Pain-free (see here for more details).

2. Testing

  1. Now, launch the HoloLens app, put your hand out in the pointing gesture (to make sure it is tracked, or attach a Hand_marker to it -- review your code to make sure you supplied the right marker to Vuforia).
  2. Reach your hand out and touch a virtual object (e.g., we suggest you try the couch scene first). You should feel EMS (verify that the EMS_Server received the message).
  3. Hooray! (or read the FAQ below)

Calibrating your EMS MR force feedback effects

You can't simply copy someone else's EMS calibration since it varies slightly from user to user (refer to openEMSstim for details on how to calibrate or see our paper). Still, you can use these values on the table below to guide your haptic effect design:

Haptic Effect Muscle mA μs Muscle mA μs Duration
Detents on Dial and Slider Wrist rotator 15 200 - - - 150
Impulse/Impact (i.e., quick force in the opposite direction of motion)
Detents on Lever Triceps 17 265 - - - 150
Marble hits walls Triceps 18 290 - - - 300
Marble drop Triceps Right 17 280 Triceps Left 17 280
Spring (i.e., continuous counterforce with spring coefficient)
Button Shoulder 27 280-350 Wrist extensor 15 100-150
Catapult Triceps 17 200-275 - - - Proportional
Friction (i.e., continuous counterforce)
Couch Static friction Shoulder 27 100-420 Wrist extensor 15 100-200 Ramp up
Couch Kinematic Friction Shoulder 27 300 Wrist extensor 15 120 Continous
Limits (i.e., strong counterforce that stops motion entirely)
Dial maximum position Wrist rotator 15 265-300 - - - Ramp up
Weight/Gravity (i.e., constant and continuous force)
Marble weight Triceps Right 17 200 Triceps Left 17 200 Continuous
Marble shift Triceps Right 17 265 Triceps Left 17 265 Proportional

p.s.: note that they might differ from the current code which is more of a demo-version.

Support

This work was kindly supported by the Hasso Plattner Institute.

HPI

Code authors

The code was authored by Sijing You (Hololens Apps, EMS Unity Server) and Pedro Lopes (EMS protocol, Marker Generator). You can contact them via github.

The paper Adding Force Feedback to Mixed Reality Experiences and Games using Electrical Muscle Stimulation (CHI'18), which you can read here was authored by Pedro Lopes, Sijing You, Alexandra Ion and Patrick Baudisch.

FAQ

1. I don't have an EMS device! What to I do?

You can get or build an openEMSstim (a piece of hardware that allows you to easily control an off the shelf muscle stimulator). All the schematics and code are available.

2. I don't feel EMS

  1. Look at your EMS_Server, did it get the message from the HoloLens? If not

    1. re-check the IP addresses for both devices.
    2. Make sure they are on the same network
    3. Make sure they can ping each other (in other words, make sure this network is not aggressively blocking ports/messages)
  2. If the message arrived at your EMS_Server but you did not feel anything:

    1. Recheck your EMS machine by calibrating it again.

Liability

Please refer to the liability waiver (in documentation/liability_waiver.md).