XMAS FAF - Architecture Change Detection Challenge

Team 12 - BOOMBAYAH

Marry Chrisler everyone!

Here is the README for the challenge with our solution proposal and a short overview on how to setup and run the model.

First of all, the challenge details are from the following doc: https://docs.google.com/document/d/1BMrEhTndkOj2VFITPza83NkTTeBMMzRTqKhTi9iCrzk/edit?usp=drivesdk

Second of all, next parts of the README will contain some solution proposal and details about running the model.

For a detailed technical report on the model architecture, methodology, and implementation, see TECHNICAL_REPORT.md.

---

Solution Proposal

Overview

Our solution addresses the challenge of detecting which architectural design comments have been implemented in modified drawings, localizing where changes occurred, and . We employ a vision-language model approach that combines CLIP (Contrastive Language-Image Pre-training) with ResNet backbone to understand both visual changes and textual comments.

Architecture

• Base model: Fine-tuned OpenCLIP with ResNet backbone
• Approach: Vision-Language Multimodal Learning
• Key components:
  • Image encoder: ResNet-based feature extractor for architectural drawings
  • Text encoder: CLIP text encoder for processing design comments
  • Change detection module: Compares original and modified architecture images
  • Bounding box regression: Localizes detected changes in the modified image

Methodology

1. Feature extraction:

   • Extract visual features from both original and changed architectural drawings
   • Encode textual comments using the CLIP text encoder

2. Change detection:

   • Compare feature representations between original and modified images
   • Match comment semantics with visual changes detected in the images
   • Generate binary predictions for each comment (implemented/not implemented)

3. Localization:

   • Predict bounding boxes for regions where changes corresponding to each comment are detected
   • Use regression head to output coordinates [x1, y1, x2, y2] for change locations

Training Strategy

• Fine-tuned OpenCLIP model on synthetic architectural change dataset
• Trained to learn domain-specific representations of architectural drawings
• Optimized for both comment implementation classification and bounding box regression
• Model checkpoint stored at: openclip_ft_out/final.pt

Key Features

• Multimodal understanding: Leverages both visual and textual information
• Generalization: Trained on synthetic data but designed to work on real-world architectural sketches
• Dual output: Provides both implementation status and spatial localization
• Robustness: Handles varying image quality, resolution, and drawing conventions

---

Setup and Running the Model

Prerequisites

• Python 3.8 or higher
• CUDA-capable GPU (recommended for faster inference)
• Required Python packages (see installation below)

Installation

1. Clone or navigate to the repository:

   cd boombayah

2. Create a virtual environment (recommended):

   python -m venv venv
   
   # On Windows:
   venv\Scripts\activate
   
   # On Linux/Mac:
   source venv/bin/activate

3. Install dependencies:

   pip install torch torchvision
   pip install pillow
   pip install open-clip-torch
   pip install numpy

4. Download the trained model:

   • Download the model checkpoint from: https://drive.google.com/file/d/12GUisYHgBz4iq3AuMPDcBh4mgxCYQpq1/view?usp=drive_link
   • Place it in the openclip_ft_out/ directory as final.pt
   • Ensure the directory structure: openclip_ft_out/final.pt

Running the Model

1. Testing on Dataset

To test the model on a CSV dataset:

python test.py dataset.csv

This will:

• Load images and comments from the CSV file
• Run predictions for comment implementation
• Generate bounding box predictions
• Calculate accuracy and IoU metrics
• Save visualization images with bounding boxes in pred_bboxes/ directory

CSV Format Expected:

original_file,changed_file,comments,bounding_boxes,comments_impl
"arch_001_original.png","arch_001_changed.png","Comment 1;Comment 2;Comment 3","[[x1,y1,x2,y2],...]","1,0,1"

2. Using the Model Programmatically

from PIL import Image
from model import predict, predict_bboxes

# Load images
original_img = Image.open("path/to/original.png")
changed_img = Image.open("path/to/changed.png")

# Define comments
comments = [
    "Add safety valve to pipe 3",
    "Remove pressure gauge",
    "Relocate pump to north wall"
]

# Predict comment implementation
predictions = predict(original_img, changed_img, comments)
# Returns: [1, 0, 1] (binary list indicating which comments were implemented)

# Predict bounding boxes
bboxes = predict_bboxes(original_img, changed_img, comments)
# Returns: [[x1, y1, x2, y2], ...] (list of bounding boxes for each comment)

3. Detailed Inference

For more detailed results including confidence scores:

from model import predict_detailed

result = predict_detailed(original_img, changed_img, comments)
# Returns dictionary with:
# - pred: binary predictions
# - scores: confidence scores
# - boxes: all detected change regions
# - boxes_for_comment: best matching box for each comment
# - features: detailed feature information
# - meta: metadata about detection

Notes

• Ensure images are in PNG format
• The model expects PIL Image objects as input
• Bounding boxes are returned in format [x1, y1, x2, y2] where (x1, y1) is top-left and (x2, y2) is bottom-right

Troubleshooting

• Import errors: Ensure model2 directory is in the Python path (handled automatically in model.py)
• Model not found: Verify openclip_ft_out/final.pt exists and is the correct checkpoint
• CUDA errors: If GPU is not available, the model will fall back to CPU (slower)

---

Model Checkpoint

Trained model location: openclip_ft_out/final.pt

Download link: https://drive.google.com/file/d/12GUisYHgBz4iq3AuMPDcBh4mgxCYQpq1/view?usp=drive_link

Make sure to download and place the model file in the correct directory before running inference.