USWP: Unstable and Stable Walking Pedestrian Prediction Using GEI and Hybrid Deep–Few-Shot Learning Models

Official PyTorch implementation of the research paper: "USWP: Unstable and Stable Walking Pedestrian Prediction Using GEI and Hybrid Deep–Few-Shot Learning Models"

---

Authors

Mahmoud Taha$^{1,4,*}$, Ahmed Fares$^{1,4}$, Hirozumi Yamaguchi$^3$, and Ahmed B. Zaky$^{2,4}$

$^1$ Department of Computer Science and Engineering, Egypt-Japan University of Science and Technology, Alexandria, Egypt
$^2$ Computer Science and Information Technology, Egypt-Japan University of Science and Technology, Alexandria, Egypt
$^3$ Graduate School of Information Science and Technology, Osaka University, Osaka, Japan
$^4$ Faculty of Engineering at Shoubra, Benha University, Cairo, Egypt
$^*$ Correspondence: mahmoud.taha17@feng.bu.edu.eg

---

Abstract

The prediction and recognition of unstable human walking patterns are of critical importance for active video surveillance, smart environments, and assistive healthcare, particularly for fall detection in the elderly. This research investigates the utility of Gait Energy Images (GEIs) combined with deep learning, vision transformers, and few-shot learning architectures to enhance the classification of stable and unstable pedestrian walking patterns.

We evaluate and compare seven methodologies:

1. Support Vector Machine (SVM) as a classical machine learning baseline.
2. MobileNet for efficient, lightweight edge computing.
3. Vision Transformer (ViT) for long-range spatial context dependency modeling.
4. YOLOv8 Classifier (YOLOv8-cls) for global image-level representation.
5. Prototypical Networks for episodic few-shot classification.
6. Matching Networks for attention-based episodic few-shot matching.
7. Relation Networks for learning non-linear similarity metrics in low-data regimes.

To facilitate this evaluation, we introduce the Unstable and Stable Walking Pedestrian (USWP) dataset, constructed by fusing and harmonizing sequences from seven public action recognition databases, containing 3,250 unique GEIs with a subject-independent evaluation protocol to prevent identity-based domain leakage.

Our experiments demonstrate that the YOLOv8-cls model achieves an accuracy of 97.0%, significantly outperforming the conventional SVM baseline (75.4%) and MobileNet (91.0%). Conversely, Relation Networks exhibit lower few-shot performance (70.94%) due to optimization complexities in learning similarity metrics from sparse data.

Computational complexity analysis reveals that MobileNet provides the optimal trade-off for real-time edge deployment (4.2 ms inference latency), while preprocessing ablation studies demonstrate that integrating the Segment Anything Model (SAM) with MediaPipe-derived Regions of Interest (ROIs) yields a 12.5% absolute improvement in accuracy by eliminating noisy silhouettes.

---

Method Overview

The USWP pipeline implements walking pattern classification using Gait Energy Images (GEIs) across four classes:

• Walk-Front (Stable)
• Walk-Side (Stable)
• Drunkenness (Unstable - loss-of-balance anomaly)
• Violent (Unstable - active gait anomaly, e.g., Boxing, Kicking)

graph TD
    A[Input Video Sequence] --> B[MediaPipe Pose Estimation]
    B --> C[Compute Dynamic ROI Bounding Boxes]
    C --> D[Segment Anything Model Segmenter]
    D --> E[Extract Binary Silhouettes]
    E --> F[Cropping, Centering, and Aspect Normalization]
    F --> G[Gait Cycle Averaging]
    G --> H[Gait Energy Image GEI]
    H --> I[Model Zoo Evaluation]
    I --> J1[Linear SVM Baseline]
    I --> J2[MobileNet]
    I --> J3[Vision Transformer]
    I --> J4[YOLOv8 Classifier]
    I --> J5[Episodic Few-Shot Networks]

Loading

1. Preprocessing & Silhouette Extraction

To segment pedestrians automatically and build clean GEIs:

1. Pose Landmark Detection: MediaPipe Pose extracts key landmarks $L = {(x_1, y_1), \dots, (x_n, y_n)}$.
2. ROI / Prompt Generation: Regions of interest (ROIs) are calculated using the bounding box of visible joints: $$ROI_j = {(x, y) \mid (x_{\text{min}}^j \leq x \leq x_{\text{max}}^j) \land (y_{\text{min}}^j \leq y \leq y_{\text{max}}^j)}$$
3. Semantic Segmentation: The Segment Anything Model (SAM) is prompted with the bounding box/landmarks to output a clean binary silhouette $B_t(x, y)$ for frame $t$, minimizing binary cross-entropy loss: $$L_{\text{seg}} = - \frac{1}{N} \sum_{i=1}^{N} \left( y_i \log(\hat{y}_i) + (1 - y_i) \log(1 - \hat{y}_i) \right)$$
4. GEI Computation: Silhouettes are aligned, cropped, resized, and averaged over the gait cycle frames $M$ to produce the Gait Energy Image (GEI): $$GEI(x, y) = \frac{1}{M} \sum_{t=1}^{M} B_t(x, y)$$

---

Model Architectures and Parameters

1. Support Vector Machine (SVM) Baseline

• Extracted GEIs are downsampled to $44 \times 44$ pixels, flattened into a $1,936$-dimensional feature vector, and classified using a Linear SVM with regularization parameter $C=1.0$.

2. MobileNet

• A lightweight, efficient CNN using Depthwise Separable Convolutions to reduce parameters while maintaining local translation invariance.
• Grayscale input adapter ($1$-channel) and 4-class linear classification head.

3. Vision Transformer (ViT)

• Processes images by splitting them into non-overlapping patches, projecting them to latent vectors, appending a learnable [CLS] token, adding positional embeddings, and routing them through Multi-Head Self-Attention (MHSA) encoder blocks.
• Attention equation: $$\text{Attention}(Q, K, V) = \text{softmax}\left(\frac{Q K^T}{\sqrt{d_k}}\right) V$$
• ViT Configuration: Patch size of $8 \times 8$ (yielding 121 patches), embedding dimension of 128, 4 attention heads, 4 encoder blocks, and MLP hidden dimension of 256.

4. YOLOv8 Classifier

• Features a CSP-Darknet backbone containing C2f blocks (CSP Bottleneck with 2 convolutions) and a Spatial Pyramid Pooling Fast (SPPF) block.
• The final classification head utilizes Global Average Pooling (GAP) and a softmax output layer: $$F = \phi(X; \theta)$$ $$z = \frac{1}{N} \sum_{i=1}^{N} F_i$$ $$\hat{y} = \text{softmax}(W z + b)$$ $$\mathcal{L} = - \sum_{c=1}^{C} y_c \log \hat{y}_c$$

5. Few-Shot Learning Models (4-Way 5-Shot)

• Prototypical Networks: Maps support and query sets to a shared embedding space. Calculates class prototypes $c_k$ as the mean embedding of class $k$'s support samples, assigning queries to the nearest class using negative squared Euclidean distance: $$c_k = \frac{1}{|S_k|} \sum_{(x_i, y_i) \in S_k} f_\theta(x_i)$$ $$p(y=k \mid x) = \frac{\exp(-||f_\theta(x) - c_k||^2)}{\sum_{k'} \exp(-||f_\theta(x) - c_{k'}||^2)}$$
• Matching Networks: Leverages an attention mechanism with cosine similarity to map query inputs directly to a weighted combination of support set labels: $$a(x, x_i) = \frac{\exp(\text{cosine}(f(x), g(x_i)) / \tau)}{\sum_{j} \exp(\text{cosine}(f(x), g(x_j)) / \tau)}$$ $$\hat{y} = \sum_{i} a(x, x_i) y_i$$
• Relation Networks: Learns a dynamic similarity metric using a relation neural network module $g_\phi$ that computes similarity scores between concatenated support-query feature maps: $$r_{i, j} = g_\phi(\text{concat}(f_\theta(x_i), f_\theta(x_j)))$$

---

Experimental Results

1. Comparative Classification Accuracy

The overall classification accuracies on the unified USWP dataset are summarized in the table below:

Model Paradigm	Methodology	Accuracy (%)
Support Vector Machine (SVM)	Classical ML (Linear SVM)	75.40%
MobileNet	Standard DL (Lightweight CNN)	91.00%
Vision Transformer (ViT)	Attention-Based DL (MHSA)	95.83%
YOLOv8 Classifier	End-to-End DL (CSP-Darknet)	97.00%
Prototypical Networks	Few-Shot Learning (5-Shot)	82.50%
Matching Networks	Few-Shot Learning (5-Shot)	92.50%
Relation Networks	Few-Shot Learning (5-Shot)	70.94%

2. Computational Complexity & Latency

Evaluated on an embedded device to check real-time feasibility for edge deployment:

Model	Parameter Count (M)	FLOPs (G)	Inference Latency (ms)
SVM Baseline	—	—	1.5 ms
MobileNet	3.2	0.31	4.2 ms
Vision Transformer (ViT)	11.4	1.25	8.5 ms
YOLOv8 Classifier	2.7	0.85	3.8 ms

3. Preprocessing Ablation Study

Omitting accurate silhouettes results in severe background clutter and performance degradation:

Preprocessing Pipeline Configuration	YOLOv8 Accuracy (%)
Raw Silhouettes (No Segmentation / Background Clutter)	84.50%
Simple Thresholding & Background Subtraction (No SAM)	89.20%
SAM Segmentation (Manual Single-Point Prompting)	95.00%
SAM + MediaPipe Automated ROI Bounding Boxes (Proposed)	97.00%

---

Code Repository Structure

├── data_prep/
│   ├── dataset_generator.py      # Synthetic GEI generator for 4 classes
│   └── mediapipe_sam_gei.py      # Automated silhouette extraction pipeline
├── models/
│   ├── few_shot.py               # ProtoNet, MatchingNet, RelationNet architectures
│   ├── mobilenet.py              # Custom MobileNet-based GEI classifier
│   ├── vit.py                    # Vision Transformer (ViT) for GEIs
│   └── yolov8_clf.py             # Custom YOLOv8 classifier backbone
├── train_eval.py                 # Core experimental runner (runs SVM baseline + DL + FS)
├── main.py                       # Orchestration entrypoint
├── requirements.txt              # Project dependencies
└── uswp_pipeline.ipynb           # Step-by-step Jupyter Notebook walkthrough

---

Installation & Getting Started

1. Install Dependencies

pip install -r requirements.txt

2. Run the Benchmarks

To generate the dataset split, train the baseline linear SVM, train the standard classifiers, train the episodic few-shot networks, and evaluate the comparative reports:

python main.py --epochs 50 --episodes 500

3. Interactive Walkthrough

Open the interactive Jupyter Notebook to step through dataset generation, visualize GEIs, train individual models, and test predictions on single query samples:

jupyter notebook uswp_pipeline.ipynb

---

Citation

If you find this repository or our research helpful in your work, please cite our paper:

@article{taha2026uswp,
  title={USWP: Unstable and Stable Walking Pedestrian Prediction Using GEI and Hybrid Deep--Few-Shot Learning Models},
  author={Taha, Mahmoud and Fares, Ahmed and Yamaguchi, Hirozumi and Zaky, Ahmed B.},
  journal={arXiv preprint arXiv:XXXX.XXXXX},
  year={2026}
}