Home
Buttons are a primitive means of interacting with vehicles. Newer iterations of vehicle dashboards have seen a shift away from the button and dial-heavy designs. At the same time, fascinating new technologies involve interactions like gesture recognition and gaze tracking, but have not yet been capitalized in the vehicle industry.
We collaborate with BMW to engineer a seamless interaction between the user and an autonomous vehicle. This is a problem that explores the intersection between new technologies and the driver/passenger experience. Our goal is to create a seamless experience and cultivate a beautiful relationship between the car and the user where the car truly understands the passenger. To do this, we utilize gaze tracking and micro-gesture recognition.
Our specific use case of these technologies explores how the passenger of an autonomous vehicle would interact with the landmarks that pass by. This BMW CES 2020 demo gives a rough idea of how gesture and gaze would be used in the vehicle.
Our use case, however, extends these functionalities to connect the passenger directly to the outside world. Examples of this include but are not limited to: making restaurant reservations, buying movie tickets, and getting information about specific landmarks.
Our gaze tracking components of the project were originally going to make use of the Vive Pro VR headset, which had gaze tracking capabilities built into the headset. The view within the headset would simulate the heads up display, which is just a screen that we would design on Android. We would mirror this Android app to the view on the VR headset (for user gaze interaction) as well as a real-time heads-up display (for everyone else to see).
Our task of micro-gesture recognition uses the Infineon XENSIV 60GHz radar sensor, which is flat just a few inches in its longest dimension. We researched other methods—computer vision, Ultraleap motion controller—but decided to use radar due to its ability to function in an environment most suitable for a passenger in a vehicle. We concluded that it would perform well in low light situations, bypass the need of a camera, and enable the car manufacturer to embed the radar chip below fabric or other non-metallic material in the dashboard.
We have built an Android application from scratch to simulate an autonomous vehicle driving experience on a heads-up display, with menus that appear over a realistic windshield view. The idea is to interact with these menus and landmarks that appear on the left and right using specific gaze directions and micro-gestures.
To integrate gaze tracking and micro-gesture recognition into the driver experience, we use MQTT, a machine-to-machine or "Internet of Things" connectivity protocol. Our MQTT scripts funnel messages from a client (gesture/gaze) to a subscriber (Android application) via a broker. To test our Android app, we can simulate gesture and gaze interactions using keyboard input.
Our project is heavily reliant on hardware and consistent access to it. Therefore, working remotely on this project with respect to both to the hardware and to each team member has proven to be a difficult challenge. While micro-gesture recognition is still a possibility because the radar chip is portable enough for one person to take it and work on it, the gaze tracking hardware portion of our project had to be omitted. It is still integrated/simulated, however, into our application via MQTT keyboard inputs.