Multimodal-Network

A multimodal deep learning model combining Bi-LSTM and CNN to predict calorie intake by integrating CGM time-series data, neural signals, and food images. Achieved RMSRE as low as 0.33

Multimodal Calorie Prediction using Bi-LSTM and CNN

This project presents a deep learning approach to accurately predict calorie intake by integrating multiple data modalities, including Continuous Glucose Monitoring (CGM) time-series data, neural features, and meal images. The architecture combines Bi-directional LSTM (Bi-LSTM) for sequential signal processing with Convolutional Neural Networks (CNNs) for image-based feature extraction.

🧠 Motivation

Traditional calorie estimation often lacks precision due to reliance on a single data source. This project enhances prediction accuracy by fusing multimodal inputs, simulating a real-world health monitoring system.

📅 Timeline

Project Duration: September 2024 – December 2024
Model Performance: Achieved Root Mean Square Relative Error (RMSRE) as low as 0.33

🔧 Key Features

• Bi-LSTM for processing CGM and neural time-series data
• CNN for extracting visual features from meal images
• Multimodal fusion for enhanced calorie prediction
• Custom data preprocessing pipelines for synchronizing heterogeneous inputs
• Evaluation metrics: RMSRE, MAE, RMSE

📁 Dataset

This project uses synthetic or anonymized datasets combining:

• CGM time-series readings
• Electrophysiological/neural signal features
• Meal image datasets

Note: Data is not included in the repository due to privacy constraints. Please reach out for access or use your own compatible dataset.