🎼 PaRSP: Harmonizing the Past, Present, and Future

A Null-Space Constrained Region-Specific Method for Continual Learning in LLMs

A bio-inspired, rehearsal-free continual learning paradigm achieving State-of-the-Art (SOTA) by explicitly balancing stability, plasticity, and long-term sustainability.Read the Paper  |  Documentation  | Citation

---

📣 News

• [Apr 2026] 🎉 PaRSP has been officially accepted to the Main Conference of ACL 2026!

---

💡 About PaRSP

Continual Learning (CL) in Large Language Models (LLMs) faces a fundamental trilemma. Traditional paradigms inevitably sacrifice one objective for another: Full Fine-Tuning causes catastrophic forgetting; Regularization restricts adaptation; and Architecture-based methods incur unbounded parameter growth.

Inspired by the human brain's memory engrams and functional partitioning, we introduce PaRSP. By distinguishing between specialized neurons (prioritizing plasticity) and generalization neurons (requiring stability), PaRSP elegantly harmonizes three critical capabilities:

1. 🏛️ Stability (Preserving the Past): Effectively mitigating catastrophic forgetting without requiring privacy-invasive data replay.
2. ⚡ Plasticity (Learning the Present): Rapidly adapting to new streaming tasks by identifying the optimal parameter subspace.
3. 🌱 Sustainability (Reserving for the Future): Strategically sequestering the vast majority of the network to accommodate long-term, lifelong learning.

Figure 1: Comparison of CL paradigms across the Past, Present, and Future dimensions.

---

⚙️ How It Works: A Synergistic Dual-Stage Design

PaRSP operates via a dual-stage mechanism that combines topological isolation with algebraic constraints. This synergy resolves the failure modes that arise when either technique is used in isolation.

Figure 2: Overview of the PaRSP framework, illustrating the interplay between Task-Specific Activation (Masking) and Null-Space Orthogonal Updates (Projection).

Stage 1: Task-Specific Parameter Region Activation

When facing a new task, PaRSP employs a static neuron-level attribution method to pinpoint the salient "functional core." By topologically isolating these specialized neuron clusters, we maximize present Plasticity while sequestering massive capacity for future Sustainability.

Stage 2: Orthogonal Update via Null Space Projection

Within the active regions, we mathematically rectify the gradient updates $\Delta \mathbf{W}$. By constraining updates to lie strictly within the null space of prior activations ($\Delta \mathbf{W} \perp X_{old}$), we ensure representation invariance, thereby rigorously safeguarding past Stability.

🚀 Production-Ready Scalability: The Dual SVD Trick

To make null-space projection tractable for massive LLMs (where $d > 4096$), PaRSP introduces an optimized Dual SVD strategy and an implicit projection mechanism.

By shifting computational complexity from a prohibitive $\mathcal{O}(d^3)$ to a linear scale $\mathcal{O}(dN^2)$, the pre-computation overhead for a massive 70B model is reduced to mere milliseconds (~0.71s).

---

🏆 Main Results

We comprehensively evaluate PaRSP on multiple rigorous benchmarks using Llama-3.1-8B and Qwen-2.5-7B as backbones.

• SOTA Performance: Establishes new state-of-the-art results on the Standard CL (SC) and Long Sequence (LS) benchmarks.
• Protection of General Intelligence: In complex reasoning tasks (e.g., GSM8K, BBH), while baseline methods suffer catastrophic cognitive collapse, PaRSP successfully preserves the intrinsic general intelligence of the foundation models at near zero-shot levels.

---

🚀 Quick Start

1. Installation

We recommend using Conda for environment management.

# Clone the repository
git clone https://github.com/JinhuiBot/PaRSP.git
cd PaRSP

# Create conda environment
conda create -n parsp python=3.10
conda activate parsp

# Install dependencies (adjust torch version for your CUDA)
pip install torch torchvision --index-url https://download.pytorch.org/whl/cu121
pip install -r requirements.txt

2. Data Preparation

Organize your data root as described below. The Long Sequence Benchmark (LS) is included in this repository. TRACE and MMLU (used for Cognitive Anchoring) should be downloaded separately.

Click to expand Recommended Directory Structure

data_root/
├── LS_Benchmark/               # (Included) Standard JSON format
│   ├── dbpedia/
│   │   ├── train.json
│   │   └── test.json
│   └── ... (15 tasks in total)
├── TRACE/                      # (External)
│   ├── ScienceQA/
│   └── ...
└── mmlu/                       # (External) 
    ├── global_facts/
    └── ...

3. Training & Evaluation (Llama-3.1-8B Example)

We provide ready-to-use shell scripts in the shells/ directory. To train the model using PaRSP on a specific task (e.g., DBpedia):

cd shells
bash train_llama.sh

Note: Key hyperparameters such as --layer_ratio (Macro-Sparsity), --neuron_ratio (Micro-Sparsity), and --history_cov_datasets (Null-Space Anchors) can be directly configured within the shell script.

To evaluate the model's continual learning metrics (Average Performance, Final Performance, Forgetting Rate), run:

bash eval_llama.sh

---

📝 Citation

If you find our methodology or code useful for your research, please consider citing our ACL 2026 paper:

@inproceedings{chen2026parsp,
  title={Harmonizing the Past, Present, and Future: A Null-Space Constrained Region-Specific Method for Continual Learning in LLMs},
  author={Chen, Jinhui and He, Shizhu and Yang, Xingchang and Liao, Huanxuan and Wang, Yequan and Liao, Xiangwen and Teng, Wenhao and Liu, Kang and Zhao, Jun},
  booktitle={Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (ACL 2026)},
  year={2026},
  publisher={Association for Computational Linguistics}
}

---

🛡️ License & Contact

This project is licensed under the MIT License. For any technical issues or inquiries, please open an issue in this repository.