🚜 Equipment Utilization & Activity Classification Prototype

A real-time video analytics pipeline for construction equipment monitoring that detects, tracks, and classifies excavator activities using computer vision and streaming architecture.

📋 Overview

This system is designed as a modular, real-time video analytics pipeline for construction equipment monitoring. It integrates computer vision techniques with a scalable streaming architecture to enable continuous analysis of machine activity.

Problem Statement

Build a real-time system that analyzes videos of construction equipment to:

• Detect and track machines automatically
• Determine if equipment is ACTIVE (working/moving) or INACTIVE (not moving)
• Classify specific activities: Digging, Swinging/Loading, Dumping, and Waiting

🏗️ Solution Architecture

Video Input → YOLO Detection → DeepSORT Tracking → Motion Analysis → Activity Classification → Output (Video + JSON + Kafka Stream)

🔧 Technical Steps

Step 1: Object Detection with YOLO

• Dataset: Construction Equipment Dataset from Roboflow
• Model: Custom trained YOLO classification model
• Performance: mAP50 = 0.95
• Model weights: best.pt
• Training Notebook: Google Colab Link

Step 2: Object Tracking with DeepSORT

To maintain consistent equipment IDs across frames:

• Tracker: DeepSORT with max_age=30
• Behavior: Preserves identity for 30 frames after detection loss before assigning new ID

Step 3: Active/Inactive State Classification

The main challenge: distinguishing active vs idle states when the excavator body is stationary but the arm moves.

Solution: Split bounding box into two vertical regions and compute frame-to-frame motion independently:

Upper Zone (top 50%)	Lower Zone (bottom 50%)	Status	Type
motion > 8	motion < 5	ACTIVE	arm_only
any	motion > 8	ACTIVE	full_body
motion < 8	motion < 8	INACTIVE	none

Step 4: Activity Classification

Track centroid (cx, cy) over a rolling window of 10 frames. The displacement vector determines the activity:

Activity	Condition
DIGGING	dy > 15
DUMPING	dy < -15
SWINGING/LOADING	|dx| > |dy|
WAITING	|dx| + |dy| < 10

Note: We attempted to train another YOLO model for motion classification but couldn't find a suitable dataset. We also tried Qwen VLM but encountered notebook crashes.

🚀 Live Demo

Try the application on Hugging Face: 👉 Gaglevision: Equipment Activity Tracker

The app accepts video input and outputs:

• Processed video with annotations
• JSON file with detailed analysis

📁 Project Structure

Equipment-Utilization-Prototype/ ├── app.py # Main application with Gradio UI ├── best.pt # Trained YOLO model weights ├── requirements.txt # Python dependencies ├── Dockerfile # Container configuration ├── docker-compose.yml # Multi-service setup (app + Kafka) ├── README.md # This file └── .gitignore

🔌 Kafka Integration

The system streams activity data to Apache Kafka for downstream processing:

• Topic: equipment_activity
• Server: localhost:9092 (configurable via KAFKA_SERVER env var)

Kafka Message Schema

{
  "frame_id": 123,
  "equipment_id": "EX-1",
  "timestamp": "5.12s",
  "utilization": {
    "current_state": "ACTIVE",
    "current_activity": "DIGGING",
    "motion_source": "arm_only"
  },
  "time_analytics": {
    "total_tracked_seconds": 10.5,
    "total_active_seconds": 8.2,
    "utilization_percent": 78.1
  }
}

## 🐳 Docker Setup

### Prerequisites
- Docker
- Docker Compose

### Run with Docker Compose (App + Kafka)

```bash
docker-compose up --build

Services:

• Gradio App: http://localhost:7860
• Kafka: localhost:9092

Run App Only

docker build -t equipment-tracker .
docker run -p 7860:7860 equipment-tracker

📦 Installation (Local)

# Clone the repository
git clone https://github.com/yourusername/equipment-utilization-prototype.git
cd equipment-utilization-prototype

# Install dependencies
pip install -r requirements.txt

# Run the application
python app.py

📊 Output Examples

Processed Video

• Bounding boxes with equipment IDs
• Color-coded status (Green = Active, Red = Inactive)
• Activity labels displayed above each detection

JSON Report

Complete frame-by-frame analysis including:

• Activity transitions
• Utilization percentages
• Time-based analytics

🧪 Experimentation Notes

We experimented with various thresholds for motion detection and selected the optimal values:

• Upper zone threshold: pixel difference > 8 for arm/bucket motion
• Lower zone threshold: pixel difference > 8 for track/body motion

These thresholds produced the best balance between sensitivity and false positives.

🔮 Future Improvements

• Collect/annotate dataset for direct activity classification with YOLO
• Implement VLM-based activity recognition
• Add support for multiple equipment types (bulldozers, loaders, trucks)
• Real-time dashboard with Grafana + Kafka consumer
• Edge deployment (Jetson Nano, Raspberry Pi)

📝 Requirements

See requirements.txt for full list. Key dependencies:

• ultralytics - YOLO model
• deep-sort-realtime - Object tracking
• gradio - UI interface
• opencv-python - Video processing
• kafka-python - Streaming integration
• numpy - Numerical computations

👥 Contributors

• [Your Name] - Initial work

📄 License

This project is licensed under the MIT License - see the LICENSE file for details.

🙏 Acknowledgments

• Roboflow Universe for the construction equipment dataset
• Ultralytics for YOLO implementation
• Deep SORT for tracking algorithm