MFM-3DPoseNet: Multi-Frame Monocular 3D Human Pose Estimation

Project Overview

MFM-3DPoseNet is a multi-frame neural network framework for metric-scale 3D human pose estimation from monocular video. This project implements the network architecture described in the paper "MFM-3DPoseNet: Multi-Frame Monocular 3D Human Pose Estimation with Relative Depth Priors and Learnable Scale Recovery".

Paper Submission: This paper has been submitted to The Visual Computer journal.

Key Features

• Input: 9 consecutive RGB frames (resolution 640×640)
• Output: 3D coordinates of 32 joints for the center frame (in meters)
• Core Capabilities:
  • Combines relative depth priors with multi-frame temporal information
  • Recovers absolute depth from relative depth maps
  • Outputs 3D human poses with metric scale

Training Method

Two-Stage Training Strategy

Stage 1: Train relative depth estimation module (20 epochs)

• Only train the relative depth estimation module
• Use sparse depth supervision
• Other modules remain frozen

Stage 2: End-to-end training (80 epochs)

• Freeze the relative depth estimation module
• Train the Depth Scale Recovery Block (DSRB) and Multi-Frame 3D Pose Estimation Module (MF3DPEM)
• Use the complete loss function

Training Commands

Using Synthetic Data for Testing (Recommended for First Run)

python train.py --use_synthetic --stage all --batch_size 4 --num_workers 0

Using Real Data for Training

python train.py --data_dir data/h36m --stage all --batch_size 8 --num_workers 4

Train Stage 1 Only

python train.py --use_synthetic --stage stage1 --batch_size 4

Train Stage 2 Only

python train.py --use_synthetic --stage stage2 --batch_size 4 --resume checkpoints/stage1_best.pth

Training Parameters

• --data_dir: Data directory path
• --batch_size: Batch size (default: 8)
• --num_frames: Number of input frames (default: 9)
• --image_size: Image size (default: 640)
• --lr: Learning rate (default: 5e-5)
• --epochs_stage1: Stage 1 training epochs (default: 20)
• --epochs_stage2: Stage 2 training epochs (default: 80)
• --checkpoint_dir: Checkpoint save directory (default: checkpoints)
• --log_dir: Log save directory (default: logs)
• --resume: Resume training from checkpoint
• --use_synthetic: Use synthetic data
• --stage: Training stage (stage1/stage2/all)
• --num_workers: Number of data loading threads (default: 4)

Monitoring Training

Monitor training with TensorBoard:

tensorboard --logdir logs

Then open http://localhost:6006 in your browser

Project Structure

Codew/
├── models/                 # Model definitions
│   ├── relative_depth.py   # Relative depth estimation module
│   ├── dsrb.py             # Depth scale recovery module
│   ├── mf3dpem.py          # Multi-frame 3D pose estimation module
│   ├── mfm_3dposenet.py    # Main network model
│   └── __init__.py
├── utils/                  # Utility functions
│   ├── losses.py           # Loss functions
│   ├── preprocessing.py    # Data preprocessing
│   └── __init__.py
├── data/                   # Data loading
│   ├── dataset.py          # Dataset class
│   └── __init__.py
├── configs/                # Configuration files
│   ├── config.py           # Configuration class
│   └── __init__.py
├── checkpoints/            # Model checkpoint save directory
├── logs/                   # Training log save directory
├── train.py                # Training script
├── requirements.txt        # Dependency list
└── README.md              # Project documentation

Environment Requirements

• Python >= 3.8
• PyTorch >= 2.0.0
• CUDA >= 11.0 (recommended)

Installation Steps

1. Clone the project locally:

2. Create a virtual environment (recommended):

conda create -n mfm3dpose python=3.10
conda activate mfm3dpose

3. Install dependencies:

pip install -r requirements.txt

Data Preparation

Using Synthetic Data (for Testing)

The project supports using synthetic data for testing without preparing a real dataset.

Using Real Data

If you need to use the Human3.6M dataset for training, organize the data as follows:

data/h36m/
├── S1/
│   ├── WalkingDirection1/
│   │   ├── images/
│   │   │   ├── frame_000001.jpg
│   │   │   ├── frame_000002.jpg
│   │   │   └── ...
│   │   └── annotations.json
│   └── ...
└── ...

The data loader needs to be adjusted according to the actual dataset format.

Model Architecture

1. Relative Depth Estimation Module

• Lightweight version based on DepthAnything v3
• Single-view attention layers: 4
• Cross-view attention layers: 6
• Output: Normalized relative depth map

2. Depth Scale Recovery Block (DSRB)

• Depth stream: 4-layer convolutional network
• Keypoint stream: MLP for processing 2D keypoints
• Fusion mechanism: Gated fusion of depth and keypoint features
• Temporal modeling: ConvGRU
• Output: Per-pixel scale factor map

3. Multi-Frame 3D Pose Estimation Module (MF3DPEM)

• Initial pose generation: Back-projection
• Temporal feature extraction: TCN + BiLSTM
• Residual MLP: Pose refinement
• Output: Final 3D pose

Loss Function

Total loss is a weighted sum:

L = λ1*L_3d + λ2*L_2d + λ3*L_depth + λ4*L_bone + λ5*L_scale

• L_3d: MPJPE (3D pose error)
• L_2d: 2D reprojection error
• L_depth: Depth error
• L_bone: Bone length consistency
• L_scale: Scale factor regularization

Default weights:

• λ1 = 1.0
• λ2 = 0.1
• λ3 = 0.5
• λ4 = 0.05
• λ5 = 0.01

Testing Model

import torch
from models import MFM3DPoseNet

# Load model
model = MFM3DPoseNet(num_frames=9, num_keypoints=135, num_joints=32)
checkpoint = torch.load('checkpoints/final_model.pth')
model.load_state_dict(checkpoint['model_state_dict'])
model.eval()

# Inference
with torch.no_grad():
    images = torch.randn(1, 9, 3, 640, 640)  # (B, T, C, H, W)
    pose_3d, absolute_depth, relative_depth, keypoints_2d = model(images, stage='full')
    print(f"3D Pose shape: {pose_3d.shape}")  # (B, 32, 3)

License

This project is for academic research use only.

Contact

For questions, please contact via Issue or Pull Request.