RoboFailRing: Retrieval-Augmented and Language Grounding Failure Detection for VLM-enabled Robotic Manipulation

RoboFailRing is a retrieval-augmented failure detection framework for robotic manipulation. It constructs a Spatio-Temporal Memory Bank from historical failure and utilizes Neuro-Symbolic Grounding to empower Vision-Language Models (VLMs) with precise, grounded reasoning capabilities.

This repository implements the core pipeline:

1. Memory Construction: Building a dual-frame (Start+End) vector index using CLIP.
2. Retrieval: Task-constrained similarity search to identify analogous historical failures.
3. Reasoning: Generating structured failure reports and actionable repair plans using Gemini 2.5 Flash.

---

📂 Project Structure

robofailring/
├── config.py                 # Global configuration (paths, models, devices)
├── build_index.py            # Script to build FAISS index from failure dataset
├── retriever.py              # Core class for retrieval and search logic
├── grounding.py              # Neuro-symbolic bridge (Symbol -> Natural Language)
├── run_reasoning_demo.py     # VLM reasoning demo
├── evaluate_metrics.py       # Evaluation script for Detection Rate (Recall/FPR)
└── utils.py                  # Helper functions for data handling

🛠️ Installation

1. Environment Setup

We recommend using Conda to manage the environment.

conda create -n robofailring python=3.10
conda activate robofailring

2. Install Dependencies

pip install -r requirements.txt

Note: This project requires faiss-gpu for efficient indexing. If you are running on a machine without NVIDIA GPUs, please switch to faiss-cpu in requirements.txt.

3. API Key Setup (For VLM Reasoning)

To use the VLM reasoning module (run_reasoning_demo.py), you need a Google Gemini API Key.

export GEMINI_API_KEY="your_api_key_here"

---

📊 Data Preparation

The system expects the dataset to be organized in a hierarchical structure. Please configure the DATASET_ROOT in config.py to point to your data directory.

Directory Structure:

data/
└── data_0/                     # Source Domain (For Memory Bank Construction)
    ├── task_name_A/
    │   ├── failure_case_1/     # Must contain image sequence (0.png ... N.png)
    │   └── ...
    └── ...

data_success/
└── data_1/                     # Target Domain (For Evaluation/Testing)
    ├── task_name_A/
    │   ├── success_case/       # Folder name matches task name (Exact match)
    │   └── failure_case_X/     # Folder name contains failure suffix
    └── ...

---

🚀 Usage

1. Build the Memory Bank

First, encode the historical failure cases from the source dataset (e.g. data_0) into a FAISS vector index.

python failure_memory/build_index.py

Output: A .faiss index file and metadata pickle will be saved in failure_memory/memory_bank/.

2. Run Reasoning Demo

Detects a specific failure case using the retrieval-augmented VLM pipeline. This script retrieves a similar past failure, generates a structured failure report, and queries Gemini-2.5-flash.

python failure_memory/run_reasoning_demo.py

3. Evaluation

We provide script to quantitatively evaluate the system's performance.

Detection Robustness (Recall & FPR) Evaluate the system's ability to distinguish between success and failure episodes across different similarity thresholds.

python failure_memory/evaluate_metrics.py

---

⚙️ Configuration

You can customize the system behavior in failure_memory/config.py:

• CLIP_MODEL_NAME: Visual backbone (Default: ViT-B/32).
• DEVICE: Execution device (cuda or cpu).

---

🙏 Acknowledgments

We thank the following projects that parts of our code and data are derived from:

• AHA

• maniskill-failgen