Multimodal Emotion Demo

Real-time emotion recognition combining Facial Expression (FER), GSR, and ECG. Designed for a 10-minute live presentation. Runs on Raspberry Pi 4 (or any Linux/Windows) with optional demo mode requiring no hardware.

• Live camera with face box + FER label
• Real-time GSR/ECG charts with HR and HRV
• Simple fusion model for final label + confidence
• Session logging and CSV export
• Offline training scripts to refine models

Repository layout

multimodal-emotion-demo/
  hardware/
    esp32_gsr/skin.ino           # ESP32 GSR streamer (HTTP POST /gsr)
    esp32_gsr/config.example.json
    arduino_ecg/ad8232_ecg.ino   # Arduino AD8232 ECG over Serial 115200
  pi_server/
    server.py                    # FastAPI + Socket.IO server
    requirements.txt
    util_signal.py               # GSR/ECG features
    util_models.py               # FER and fusion models
    train/
      train_visual.py
      train_physio.py
      train_fusion.py
  web_ui/
    index.html app.js styles.css # Socket.IO UI with charts
  demo/
    demo_script.md checklist.md  # Live demo guidance
  presentation/slide.html        # One-slide overview
  tests/test_features.py         # Unit tests for feature functions

Hardware wiring

GSR (ESP32)

• Sensor analog OUT → ESP32 ADC GPIO34 (default)
• Sensor GND → ESP32 GND; VCC → 3.3V
• Edit hardware/esp32_gsr/config.example.json and copy to config.json with your Wi‑Fi and server IP

ECG (AD8232 + Arduino)

• AD8232 OUTPUT → Arduino A0
• AD8232 3.3V → Arduino 3.3V; AD8232 GND → Arduino GND
• Electrode placement (single-lead): RA (right chest), LA (left chest), RL (right abdomen/leg ground)
• Flash hardware/arduino_ecg/ad8232_ecg.ino (Baud 115200)

SparkFun AD8232 Safety warnings:

• Not a medical device; do not use for diagnosis.
• Use on healthy, dry skin only; stop if any discomfort.
• Prefer laptop on battery; isolate from mains.

Setup (Raspberry Pi or PC)

1. Python env

python -m venv .venv
. .venv/bin/activate  # Windows: .venv\\Scripts\\activate
pip install -r pi_server/requirements.txt

2. Run the server

python pi_server/server.py --host 0.0.0.0 --port 8000 --serial COM3

• Replace COM3 with your Arduino port (Linux: /dev/ttyACM0).
• For demo mode (no hardware):

python pi_server/server.py --demo true

3. Open the UI at http://<pi-ip>:8000/.

4. On ESP32, flash hardware/esp32_gsr/skin.ino and upload config.json to SPIFFS with your Wi‑Fi and server IP.

UI controls

• Start Demo: begin processing and logging
• Calibrate Baseline: 3 seconds GSR baseline capture
• Export CSV: download latest session

Training (offline)

• Collect logs in sessions/ by running live demos.
• Train physio: python pi_server/train/train_physio.py --logs sessions --out physio_model.pkl
• Train fusion: python pi_server/train/train_fusion.py --logs sessions --out pi_server/train/fusion_model.pkl
• Restart server to load the new fusion model.

Design

• FER: Haar face detector + heuristic FER; replaceable with DeepFace/TFLite (see util_models.py)
• GSR features: tonic (LPF) and SCR rate (peaks/sec)
• ECG features: HR, RMSSD, SDNN via heartpy fallback
• Fusion: Logistic Regression over visual probs + physio features

Demo timing (~3–5 min)

See demo/demo_script.md for the exact flow and talking points.

Safety

• ECG sensors are not medical devices.
• Keep electrodes away from broken skin. Disconnect if discomfort occurs.

Troubleshooting

• No camera: ensure OpenCV installed; try --demo true.
• No serial: check baud (115200) and correct port; permissions on Linux.
• ESP32 not posting: verify Wi‑Fi and server IP; view server logs.

Camera modes & permissions

This project supports two FER modes:

• Local FER: the server captures the camera and runs FER on the same machine.
• External FER: camera and FER run on another machine; that client POSTs labels to the server /fer, and the server disables its own camera to save CPU.

Run local FER (camera enabled):

python pi_server/server.py --fer local

Run external FER (camera disabled; use pi_server/external_fer_client.py on a laptop):

python pi_server/server.py --fer external
python pi_server/external_fer_client.py --server http://<server-ip>:8000

Windows camera permission: Settings → Privacy & security → Camera → enable access for apps and for "Desktop apps". If another app is using the camera, close it first. On Raspberry Pi OS, if using the legacy stack, enable the camera via raspi-config or use a USB webcam; ensure your user can access /dev/video0.

References

• ddessy/RealTimeArousalDetectionUsingGSR
• TharunSivaprakash/Galvanic-Skin-response-using-ESP32
• ChrisDavi3s/ArduinoAD8232ECG
• justsaumit/opencv-face-recognition-rpi4
• bezareva/emotion_recognition_ECG_GSR

TODOs

• Fill in hardware SSID/password in hardware/esp32_gsr/config.json (copy from example)
• If using different ADC pin or sample rate, update skin.ino
• Optionally implement a stronger FER model or move FER to a laptop if Pi is slow