Extend LLM Context

ANN Final Project

1. PEI(Positional Embedding Interpolation)

1. Download gpt2 from hugging-face.

2. Extend and finetune gpt2:

cd PEI
python extend_gpt2_ctx.py
python finetuned_gpt2.py

3. Up to now, you have extended and finetuned gpt2, let's chat with it!

python test_gpt2_finetuned.py

4. You also can use long context to test it like this:

python test_long_context.py

2. LongRoPE

See README in LongRoPE/

3. FoPE

See README in FoPE/Fourier-Position-Embedding-main/