Magic Home 5.0 -- Edge AI Health Monitoring System

A real-time, multi-modal health monitoring system that fuses mmWave radar and camera-based rPPG for contactless vital sign extraction on edge hardware.

Platform

• Jetson Orin Nano (8GB RAM, ARM64 + GPU)
• JetPack 6, TensorRT, CUDA 12.x
• IMX219 CSI camera + IWR6843ISK mmWave radar

Key Results

Metric	Value
TensorRT acceleration	17x (336ms -> 20ms per inference)
Multi-model pipeline	6.3 FPS (face detection + recognition + rPPG)
rPPG heart rate accuracy	MAE < 4 BPM vs. pulse oximeter ground truth
Radar vital signs	Real-time breathing rate + heart rate extraction
Sensor fusion	Weighted confidence-based radar + rPPG fusion

Architecture

Camera (IMX219, 30fps)
  |
  v
Face Detection (SCRFD, TensorRT)
  |
  v
Face Recognition (ArcFace) + Pose / Expression / Gaze
  |
  v
rPPG Pulse Extraction (POS algorithm + Deep-rPPG)
  |                                              |
  v                                              v
rPPG Heart Rate  ----+                  mmWave Radar (IWR6843)
                     |                           |
                     v                           v
              Fusion Engine  <----  Radar Vital Signs (TLV parsing)
                     |
                     v
              Fused Heart Rate + Breathing Rate + Confidence

Project Structure

magic-home/
  src/
    fusion/          # Multi-sensor data fusion
      engine.py      # Confidence-weighted radar + rPPG fusion
    radar/           # mmWave radar vital sign extraction
      serial_reader.py   # IWR6843ISK serial frame reader (TLV parsing)
      vital_signs.py     # Heart rate & breathing rate extraction
    rppg/            # Camera-based remote photoplethysmography
      camera.py      # Jetson GStreamer capture + POS algorithm
  configs/
    iwr6843_vital_signs.cfg   # Radar chirp configuration (0.3-1.5m range)
  scripts/
    jetson_setup_rppg.sh      # Jetson environment setup script

How It Works

Radar Pipeline (IWR6843)

The IWR6843ISK radar transmits FMCW chirps and receives reflections from the subject's chest. The TI Vital Signs firmware extracts micro-motion caused by breathing and heartbeat. Our serial_reader.py parses the TLV binary protocol over UART, and vital_signs.py applies sliding-window median filtering for stable BPM output.

rPPG Pipeline (Camera)

A CSI camera captures facial video at 30fps. We extract the forehead ROI and compute per-frame RGB channel means. The POS (Plane-Orthogonal-to-Skin) algorithm projects these signals to isolate the blood volume pulse, followed by bandpass filtering (0.7-3.5 Hz) and FFT peak detection to estimate heart rate.

Fusion Engine

When both modalities are available, the fusion engine computes a confidence-weighted average. If the two estimates diverge by more than 20 BPM (indicating one sensor is unreliable), it falls back to the higher-confidence source. Fusion boosts overall confidence by cross-validating independent measurements.

Quick Start

Prerequisites

pip install numpy scipy pyserial opencv-python

Radar Data Capture

python src/radar/serial_reader.py \
  --cli /dev/ttyACM0 --data /dev/ttyACM1 \
  --config configs/iwr6843_vital_signs.cfg

rPPG Heart Rate (Jetson)

python src/rppg/camera.py --device /dev/video0

Jetson Environment Setup

bash scripts/jetson_setup_rppg.sh

Notes

• Model weight files (.pt, .onnx, .engine, .trt) are not included in this repository. See the setup script for instructions on obtaining pretrained models.
• The radar configuration file is tuned for near-range vital sign detection (0.3-1.5m). Adjust vitalSignsCfg parameters for different ranges.
• On Jetson, the camera uses GStreamer + nvarguscamerasrc for hardware-accelerated capture. On desktop, it falls back to standard V4L2.

Background

Built by one person + AI tools. From hardware concept to working demo: 2 weeks.

The goal was to explore whether commodity edge hardware (a $250 Jetson + $50 radar module + $10 camera) could deliver clinically-relevant contactless health monitoring. The answer is a qualified yes -- radar provides robust breathing detection even in darkness, while rPPG adds heart rate precision under good lighting. Fusing both fills each other's blind spots.

License

MIT License. See LICENSE for details.