CLIP Few-Shot Adaptation with Prompt Learning

Deep Learning Course Project | University of Trento | 2025

This project, developed as part of the Deep Learning course at the University of Trento, reproduces state-of-the-art prompt-learning methods (CoOp, CoCoOp) and extends them with novel attribute-based approaches to improve robustness and few-shot generalization in vision–language models.

🎯 Project Overview

This project tackles the challenge of few-shot adaptation in vision-language models, specifically addressing the trade-off between adapting to new tasks with limited data (base classes) while preserving zero-shot generalization capabilities (novel classes).

We implement and extend three seminal works:

• CoOp: Context Optimization with learnable continuous prompts
• CoCoOp: Conditional prompts that adapt based on input images
• LoRA CLIP: Parameter-efficient adaptation of text encoder

Methodological Development and Technical Implementation

Initial Baseline: Context Optimization (CoOp)

We initiated our investigation by implementing Context Optimization (CoOp), a method that replaces manually engineered text prompts (e.g., "a photo of a [CLASS]") with learnable continuous context vectors. While CoOp demonstrated strong performance on base classes (95.19% accuracy), we observed significant performance degradation on novel classes (68.09% accuracy), indicating substantial overfitting to the training distribution.

Investigation of Existing Approaches

Subsequently, we evaluated Conditional Context Optimization (CoCoOp), which introduces input-conditional prompt generation via a lightweight Meta-Network architecture. This approach yielded improved novel class generalization (72.47% accuracy) and achieved the highest harmonic mean performance (81.25%). We additionally integrated Low-Rank Adaptation (LoRA) to enable parameter-efficient fine-tuning while maintaining competitive performance metrics.

Problem Identification and Hypothesis Formation

Through systematic analysis of the generalization failure modes, we identified a fundamental limitation in the Oxford Flowers dataset: the semantic gap between class labels and visual features. Specifically, taxonomic class names (e.g., "Canna lily", "Fritillary", "Cyclamen") provide minimal information about discriminative visual characteristics. Analysis of reference descriptions revealed an emphasis on botanical metadata (chromosomal information, geographic distribution) rather than visually salient features such as petal coloration, inflorescence morphology, or foliar texture.

This observation motivated our central hypothesis: embedding explicit visual attribute concepts within the prompt structure could bridge the semantic-visual gap.

Proposed Method: General Attribute Context Optimization (GaCoOp)

We developed General Attribute Context Optimization (GaCoOp), which incorporates fixed visual attribute tokens as structural anchors within learnable context vectors.

Architectural Design:

Traditional prompt structure:

[V₁...Vₘ] [class_name]

GaCoOp prompt structure:

[V¹] color [V²] shape [V³] texture [V⁴] pattern [V⁵] [class_name]

Where:

• V^j ∈ ℝ^(d×L_j) represents learnable context token blocks
• Attribute tokens ("color", "shape", "texture", "pattern") serve as fixed semantic anchors
• These attributes provide explicit visual grounding absent from taxonomic nomenclature

Extended Architecture: GaCoCoOp

We further developed GaCoCoOp, which integrates our attribute-structured prompting with input-conditional context generation, enabling prompts that are simultaneously attribute-grounded and instance-adaptive.

Theoretical Foundation

1. Visual Semantic Grounding: Fixed attribute tokens inject domain-relevant visual concepts directly into the embedding space, compensating for information-poor class labels
2. Cross-Class Knowledge Transfer: Shared attribute context vectors facilitate feature learning across all C=102 classes, improving data efficiency
3. Label-Efficient Learning: Requires only class names and four generic attribute descriptors, without additional annotation overhead

Formal Implementation

For each attribute j, we define learnable context embeddings V^(j). The complete prompt representation for class c_i is constructed as:

t_i = [V^(1), "color", V^(2), "shape", V^(3), "texture", V^(4), "pattern", V^(5), c_i]

Optimization Protocol:

• Parameters: Context vectors {V^(j)}_{j=1}^5 only
• Constraints: CLIP text and image encoders remain frozen
• Objective: Standard cross-entropy loss over classification logits, consistent with CoOp/CoCoOp baseline formulations

This architecture maintains computational efficiency while introducing structured inductive biases that align with human visual perception and category organization.

📊 Results Summary

Evaluated on Oxford Flowers dataset with 10-shot learning

Method	Base Acc (%)	Novel Acc (%)	Harmonic Mean (%)
Few-Shot CLIP	71.25	78.24	74.58
CoOp	95.19	68.09	79.39
CoCoOp	92.44	72.47	81.25
LoRA - CLIP	93.49	70.54	80.41
GaCoOp (Ours)	93.25	67.82	78.53
GaCoCoOp (Ours)	87.14	74.40	80.27

📚 References

Key papers implemented and extended:

• CLIP - Radford et al., 2021
• CoOp - Zhou et al., 2022
• CoCoOp - Zhou et al., 2022
• LoRA - Hu et al., 2022