Perplexity Script and Notebook

[Stillerman]

#53

Here is a notebook and script that can be used to calculate perplexity. The notebook runs on colab, but I cannot test the script locally so that will need to be tested. Should output perplexity results toperplexity-results.jsonl.

Should be able to run

python3 calculate_perplexity.py --model Multi-Domain-Expert-Layers/expert-arxiv --dataset Multi-Domain-Expert-Layers/arxiv --split validation_domain

and test any model/dataset/split we want.

[Stillerman]

Perplexity script produced the following results

Model	arxiv	freelaw	github
expert-arxiv	6.588	6.077	5.675
expert-freelaw	6.744	6.038	5.673
expert-github	6.827	6.145	5.680
pythia-1b-deduped	6.560	6.013	5.604

All makes sense except github expert does worse on github dataset. Is this possible @mrcabbage972

[Stillerman]

We are still getting unexpected results but we believe it is not due to a bug in the perplexity script so undrafted. Details of weird perplexities in the issue #53