Conceptual Similarity

A similarity metric based on concepts learnt with Sparse AutoEncoders (SAE).

Installation

conda create -n conceptual_similarity python=3.11.*
conda activate conceptual_similarity
git clone https://github.com/facebookresearch/dinov3.git
cd dinov3
pip install .
cd ..
pip install overcomplete
cd IQA-PyTorch
pip install -e .
cd ..
pip install clean-fid
pip install jupyter

Evaluation

Robustness Experiments

Evaluate how different image transformations affect similarity metrics. Multiple evaluation options are available:

python src/ConceptSim/eval.py --help

1. Augmentation-based evaluation - Compute metrics between ground-truth images and their augmented versions:

python src/ConceptSim/eval.py eval-augment \
    --img-dir "path/to/images" \
    --augment-type "rotation" \
    --batch-size 32 \
    --num-workers 4

Supported augmentation types:

• Positive (similarity-preserving): rotation, shift, flip, color, padding, jpeg-pos, none
• Negative (dissimilarity-inducing): jpeg-neg, grayscale, blur, invert, crop
• Extreme: white, black, noise

2. Folder comparison - Compute metrics between two aligned image folders:

python src/ConceptSim/eval.py eval-folders \
    --img-dir1 "path/to/folder1" \
    --img-dir2 "path/to/folder2"

3. FID metrics - Compute FID, CLIP-FID, and KID between two image folders:

python src/ConceptSim/eval.py eval-fid \
    --img-dir1 "path/to/folder1" \
    --img-dir2 "path/to/folder2"

Human Similarity Steering

Run experiments to evaluate how human-guided concept weighting affects similarity metrics. This experiment:

• Compares pairs of visual concepts (e.g., round neck vs v-neck shirts)
• Tests "Same" and "Difference" steering modes to emphasize or de-emphasize specific concepts
• Runs multiple shuffled trials for statistical robustness
• Outputs detailed CSV results for analysis

Before running, update the paths in the script:

• img_dir: Path to your dataset images
• results_dir: Path to save output CSV files

python src/ConceptSim/human_similarity_steering_experiment.py

Output files:

• concept_eval_detailed_runs.csv - Per-run metrics (long format)
• concept_eval_summary_wide.csv - Summary statistics with mean ± std (wide format)