What has been done till now

[Knight-Fury1102]

Machine Unlearning

Okay, let us begin developing this Machine Unlearning Project. This document serves as the internal architectural context and state-tracker.

Infrastructure Setup (Completed)

• DVC and Git are configured.
• Data (CIFAR-10) is mirrored on remote and pulled locally.
• Docker environment is locked.

Core Pipeline Implementation (Completed)

The foundational pipeline has been engineered with strict separation of concerns, deterministic execution, and a YAML-driven configuration cascade.

1. Data Preparation (src/data/)

• Built lazy-loading ImageDataset with deterministic class_to_idx mappings.
• Implemented memory-efficient generate_splits supporting both class and random unlearning paradigms. Guaranteed strict mutual exclusivity.
• Data splits are saved as isolated tensors (.pt) to prevent redundant processing, and the dataset object is passed directly back to the orchestrator.

2. Model Architecture & Registry (src/models/)

• Implemented a centralized Model Registry (get_model) to route configurations to specific architectures.
• Built a dataset-aware ResNet18 class. It detects target datasets (e.g., CIFAR) and dynamically patches the computational graph (swapping 7x7 conv for 3x3, bypassing ImageNet maxpool) to preserve 32x32 spatial dimensions natively without upstream image resizing.
• Support for structural **kwargs injection (e.g., dropout routing).

3. Training Engine (src/models/)

• Created an Optimizer/Loss Factory (utils.py / setup_optimizers.py) to completely decouple hyperparameter logic from the training loop.
• Built the baseline train_model loop with aggressive memory management (set_to_none=True, non_blocking=True for GPU transfers).
• Integrated tqdm for real-time loss, accuracy, and learning rate monitoring.
• Implemented Threshold Early Stopping. The loop auto-terminates the moment training accuracy hits the 90%+ memorization target to prevent wasted compute.
• Integrated CosineAnnealingLR scheduling for stable convergence. Triggered via a factory method, same as Optimizer.
• Added robust YAML parsing to forcefully cast scientific notation strings (e.g., 1e-3) to floats, preventing scheduler crash loops.

4. Unlearning Engine & Config Cascade (src/unlearning/)

• Implemented exact_retrain as the mathematical ground-truth ($M^*$). It automatically inherits the updated train_model logic (CosineAnnealing, early stopping) for a perfectly identical hyperparameter environment.
• Built a smart cascading configuration wrapper (unlearn.py). Hyperparameters explicitly defined in [unlearning] override the baseline; otherwise, they strictly inherit from the [training] block, falling back to centralized defaults.
• Enforced strict "fresh architecture" initialization during exact retraining. The engine actively rejects the trained baseline model to mathematically guarantee zero weight leakage from the forget set.

5. Membership Inference Attack & Privacy Evaluation (src/evaluation/) (Completed)

• Implemented automated Shadow Model generation (evaluation/shadow_models.py) mapped to random overlapping data subsets.
• Built a dynamic data extraction pipeline (evaluation/extract_attack_data.py) that executes target models to harvest dynamic sizes of features (sorted probabilities, one-hot encoded true labels, and the dynamic individual training loss).
• Engineered AttackMLP, a 2-layer binary classifier, trained using BCEWithLogitsLoss and ReduceLROnPlateau scheduling.
• Advanced MIA Logic: Upgraded MIA evaluation to run two 1:1 balanced tests using a test set: Forget vs. Test (Privacy Check) and Retain vs. Test (Utility Check).
• Subset Balancing: Implemented dynamic 1:1 sub-sampling via random permutation within the evaluation loop to eliminate class-imbalance exploitation by the attacker, establishing a true 50% random-guess baseline.
• Edge Case Identification: Identified and documented the "Class Forgetting MIA Anomaly" where MIA AUC drops near 0.05 during full-class deletion. This occurs due to structural loss distribution shifts (the model becomes "perfectly wrong" on the alien class), proving unlearning success despite the inverted metric.

6. Central Orchestrator (run.py)

• Tied the ecosystem together into a seamless execution runner: Data Prep -> Model Init -> Baseline Train -> Unlearn -> Utility Evaluation -> MIA Attack.
• Built strict state-checking. The runner checks for existing .pt artifacts and safely skips phases unless the override: True flag is explicitly set in the YAML.
• Runner successfully passes the test_dataset into the MIA evaluation loop for standard privacy auditing against unseen data.

Current Status & Next Immediate Steps (End of Session Checkpoint)

Status: The benchmark and evaluation infrastructure (P6/P7 scope) is 100% complete and verified with robust pytests. Pre-flight checks (DVC, Pytests, Config validation) are done, and the PR for the MIA evaluation suite is ready to be merged into main. The baseline model ($M$) and exact retrain model ($M^*$) have been successfully evaluated, providing a verified privacy yardstick.

Next Session Goals (Team Handoff):

1. Gradient Ascent (M2): P3 to branch out and implement gradient_ascent.py. Integrate dual-loop ascent/descent with gradient clipping.
2. Fisher Forgetting (M3): P4 to branch out and implement fisher_forgetting.py. Implement diagonal FIM computation and calibrated noise injection.
3. Report Documentation: P8 to document the "Class Forgetting MIA Anomaly" in the final report, explaining the statistical mechanics of why the AUC inverts to 0.05 during full-class unlearning.

Mode of development:

• Make a new branch for each feature/issue.
• Write tests for that issue using mocked datasets (MagicMock) to bypass heavy I/O and keep CI fast.
• Merge into main strictly via Pull Request once CI is green.
• Development is milestone-based: Issue creation -> Test integration (Red state) -> Logic implementation (Green state) -> Merge.

[Knight-Fury1102]

Can use the above, to know how much is done and start from there