Continual & Lifelong Unsupervised Domain Adaptation via Prototypical Learning

1. Overview

This project implements a continual, lifelong unsupervised domain adaptation framework evaluated on 20 sequentially arriving CIFAR-10 data distributions[cite: 1]:

• Task 1 ($D_1 \to D_{10}$): Homogeneous input distribution ($p(x)$ remains invariant) where $D_1$ is labeled and $D_2 \dots D_{10}$ are unlabeled[cite: 1].
• Task 2 ($D_{11} \to D_{20}$): Heterogeneous input distributions with sequential distribution shifts where all datasets are unlabeled[cite: 1].

---

2. Pipeline & Methodology

2.1 Feature Extraction

• Backbone: Vision Transformer pre-trained with Self-Supervised Learning (vit_base_patch14_dinov2 via timm).
• Input Resolution: $518 \times 518$ (Task 1), $224 \times 224$ (Task 2).
• Embedding Dimension: $D = 768$.

2.2 Task 1: Pseudo-Labeled Learning with Prototypes (PLwP)

1. Supervised Initialization ($D_1$):
   • Class prototype vector: $$\mathbf{c}k = \frac{1}{N_k} \sum{i: y_i = k} \mathbf{x}_i$$
   • Pooled within-class covariance matrix $\mathbf{\Sigma}$: $$\mathbf{\Sigma} = \frac{1}{N - K} \sum_{k=1}^{K} \sum_{i: y_i = k} (\mathbf{x}_i - \mathbf{c}_k)(\mathbf{x}_i - \mathbf{c}_k)^T + \lambda \mathbf{I}$$
2. Pseudo-Labeling ($D_2 \dots D_{10}$):
   • Mahalanobis distance metric: $$d_M(\mathbf{x}, \mathbf{c}_k) = (\mathbf{x} - \mathbf{c}_k)^T \mathbf{\Sigma}^{-1} (\mathbf{x} - \mathbf{c}_k)$$
   • Label prediction: $$\hat{y} = \arg\min_{k} d_M(\mathbf{x}, \mathbf{c}_k)$$
3. EMA Prototype Updates: $$\mathbf{c}_k^{(t)} = \alpha \mathbf{c}_k^{(t-1)} + (1 - \alpha) \mathbf{c}_k^{\text{new}}$$

---

2.3 Task 2: Robust Lifelong Adaptation Pipeline

1. Domain Shift Centroid Correction:
   • Global translation alignment prior to pseudo-labeling using confident anchor samples: $$\mathbf{\delta} = \frac{1}{|S_{\text{conf}}|} \sum_{i \in S_{\text{conf}}} \mathbf{x}i - \frac{1}{K} \sum{k=1}^K \mathbf{c}_k$$ $$\mathbf{c}_k^{\text{shifted}} = \mathbf{c}_k + \mathbf{\delta}$$
2. Two-Stage Pseudo-Labeling (T2PL):
   • Stage 1 (Centroid Refinement): Compute confidence weights $w_i = \frac{1}{d_{\min}(\mathbf{x}_i) + \epsilon}$ to form refined centroids $\mathbf{\tilde{c}}_k$.
   • Stage 2 ($k$-NN Anchor Voting): Assign labels using cosine similarity top anchors and majority voting across $k=5$ nearest neighbors.
   • Quantile Thresholding: Dynamic filtering retaining the top $60% - 75%$ most confident samples per batch.
3. Memory Replay Buffering (LDAuCID-inspired):
   • Stores $M$ nearest exemplars per class to prototype centroids into a fixed-size replay buffer $\mathcal{M}$: $$k_{\text{per_class}} = \min(M_{\text{base}} + \lfloor(1 - \gamma) \cdot 60\rfloor, 150)$$
4. Adaptive EMA Update: $$\alpha = \text{clip}\left(\alpha_{\text{base}} + (1 - \gamma) \cdot 0.4, \alpha_{\text{base}}, 0.7\right)$$

---

3. Evaluation Metrics

• Classification Accuracy ($A_{i, j}$): Accuracy of model snapshot $f_i$ evaluated on held-out dataset $\hat{D}_j$[cite: 1]: $$\text{Accuracy}(f_i, \hat{D}j) = \frac{1}{N_j} \sum{m=1}^{N_j} \mathbb{I}\left(f_i(\mathbf{x}_m) = y_m\right)$$
• Catastrophic Forgetting Rate ($CF_k$): $$CF_k = \max_{t \in {k, \dots, T-1}} A_{t, k} - A_{T, k}$$

---

4. Empirical Results

Task 1: In-Domain Continual Learning ($10 \times 10$ Matrix)

Accuracies of model snapshots $f_1 \dots f_{10}$ across held-out sets $\hat{D}1 \dots \hat{D}{10}$[cite: 1]:

Model	$\hat{D}_1$	$\hat{D}_2$	$\hat{D}_3$	$\hat{D}_4$	$\hat{D}_5$	$\hat{D}_6$	$\hat{D}_7$	$\hat{D}_8$	$\hat{D}_9$	$\hat{D}_{10}$
$f_1$	0.9872	—	—	—	—	—	—	—	—	—
$f_2$	0.9792	0.9744	—	—	—	—	—	—	—	—
$f_3$	0.9784	0.9732	0.9756	—	—	—	—	—	—	—
$f_4$	0.9764	0.9700	0.9720	0.9696	—	—	—	—	—	—
$f_5$	0.9772	0.9696	0.9716	0.9680	0.9732	—	—	—	—	—
$f_6$	0.9760	0.9692	0.9728	0.9688	0.9740	0.9688	—	—	—	—
$f_7$	0.9732	0.9700	0.9724	0.9684	0.9752	0.9688	0.9684	—	—	—
$f_8$	0.9740	0.9680	0.9716	0.9652	0.9720	0.9688	0.9684	0.9644	—	—
$f_9$	0.9732	0.9664	0.9700	0.9652	0.9708	0.9668	0.9672	0.9644	0.9664	—
$f_{10}$	0.9732	0.9684	0.9712	0.9668	0.9704	0.9648	0.9676	0.9664	0.9644	0.9696

---

Task 2: Cross-Domain Lifelong Adaptation ($10 \times 20$ Matrix)

Accuracies of model snapshots $f_{11} \dots f_{20}$ across Task 1 ($\hat{D}1 \dots \hat{D}{10}$) and Task 2 ($\hat{D}{11} \dots \hat{D}{20}$) held-out domains[cite: 1]:

Model	$\hat{D}_1$	$\hat{D}_2$	$\hat{D}_3$	$\hat{D}_4$	$\hat{D}_5$	$\hat{D}_6$	$\hat{D}_7$	$\hat{D}_8$	$\hat{D}_9$	$\hat{D}_{10}$	$\hat{D}_{11}$	$\hat{D}_{12}$	$\hat{D}_{13}$	$\hat{D}_{14}$	$\hat{D}_{15}$	$\hat{D}_{16}$	$\hat{D}_{17}$	$\hat{D}_{18}$	$\hat{D}_{19}$	$\hat{D}_{20}$
$f_{11}$	0.9692	0.9656	0.9664	0.9700	0.9676	0.9700	0.9688	0.9596	0.9624	0.9684	0.8724	—	—	—	—	—	—	—	—	—
$f_{12}$	0.9660	0.9644	0.9612	0.9684	0.9664	0.9684	0.9652	0.9604	0.9640	0.9660	0.8744	0.6084	—	—	—	—	—	—	—	—
$f_{13}$	0.9696	0.9668	0.9672	0.9724	0.9676	0.9676	0.9652	0.9640	0.9672	0.9680	0.8748	0.6200	0.8860	—	—	—	—	—	—	—
$f_{14}$	0.9700	0.9672	0.9680	0.9728	0.9700	0.9688	0.9664	0.9624	0.9668	0.9692	0.8724	0.6244	0.8808	0.9592	—	—	—	—	—	—
$f_{15}$	0.9688	0.9652	0.9688	0.9720	0.9712	0.9708	0.9672	0.9628	0.9672	0.9688	0.8692	0.6228	0.8796	0.9588	0.9600	—	—	—	—	—
$f_{16}$	0.9688	0.9660	0.9676	0.9728	0.9704	0.9700	0.9676	0.9608	0.9656	0.9676	0.8708	0.6112	0.8796	0.9600	0.9592	0.8748	—	—	—	—
$f_{17}$	0.9688	0.9648	0.9652	0.9696	0.9684	0.9668	0.9668	0.9608	0.9620	0.9668	0.8604	0.6008	0.8704	0.9572	0.9576	0.8652	0.8992	—	—	—
$f_{18}$	0.9684	0.9632	0.9612	0.9656	0.9656	0.9616	0.9640	0.9560	0.9600	0.9636	0.8536	0.5836	0.8640	0.9548	0.9568	0.8556	0.8932	0.8476	—	—
$f_{19}$	0.9640	0.9608	0.9572	0.9600	0.9604	0.9584	0.9592	0.9580	0.9588	0.9616	0.8528	0.5792	0.8660	0.9492	0.9516	0.8512	0.8860	0.8464	0.6860	—
$f_{20}$	0.9692	0.9624	0.9624	0.9636	0.9676	0.9644	0.9636	0.9572	0.9608	0.9656	0.8560	0.5824	0.8680	0.9536	0.9568	0.8540	0.8944	0.8484	0.6856	0.9448