Additional training of Llama 2 with scientific articles

About

• Source codes for arxiv preprint.
• Teaching Specific Scientific Knowledge into Large Language Models through Additional Training

Setup

• Clone this repo.

  git clone https://github.com/KanHatakeyama/Additional-training-Llama2.git
  

• Download dataset from HuggingFace

  git lfs clone https://huggingface.co/datasets/kanhatakeyama/nature-family-CC-papers
  mv nature-family-CC-papers/database/ .
  mv nature-family-CC-papers/smallDB/ .
  

• Create env (conda)

  conda env create -f environment.yml
  

Codes

• Demo code and model for additional training with scientific texts
  • Here!
• Fictional datasets
  • Result analysis
    • Full-parameter
    • LoRA
  • Training
    • Full-parameter
      • Change number of target texts
      • Change number of irrelevant texts
    • LoRA
      • 7b model
        • Random search
        • Optimization
      • 7,13,70b models
        • Selected adapter layers
          • Change number of target texts
          • Change number of irrelevant texts
        • Full adapter layers
          • Change number of target texts
          • Change number of irrelevant texts
• Scientific papers
  • Result analysis
  • Training

Author

• Kan Hatakeyama
• Tokyo Tech., Japan