Where are performance results of experiments stored

[stefanhgm]

Hi,

thank you very much for sharing your code!

I ran the example from the readme and the parts of the few-shot-pretrained-3b-100k.sh script. However, the dev_scores.json for the readme example only contains the line:

{"accuracy": 0.6101083032490975, "score_gt": 0.3983679488032303, "score_cand": 0.6958685107394676}

And for t03b_copa_seed42_ia3_pretrained100k (the first experiment of few-shot-pretrained-3b-100k.sh):

{"accuracy": 0.85, "score_gt": 0.06061243396921782, "score_cand": 0.4640417302213609}

Those are just the results of the "Validation sanity check" right at the beginning, so I wondered where the validation results after each epoch are stored or am I missing something here?

Thanks!

[cherry979988]

Hi Stefan, I came across a similar problem and I hope my solution is helpful.

I noticed that in 11b model experiments there is a flag "eval_epoch_interval=50"

t-few/bin/few-shot-pretrained-11b-100k.sh

Line 5 in 217cfa3

python -m src.pl_train -c t011b.json+ia3.json+${dataset}.json -k load_weight="pretrained_checkpoints/t011b_ia3_finish.pt" exp_name=t011b_pretrained100k_${dataset}_seed${seed}_ia3 few_shot_random_seed=${seed} seed=${seed} eval_epoch_interval=50 batch_size=1 eval_batch_size=2 grad_accum_factor=8

So I added it to the few-shot-pretrained-3b-100k.sh as well. With this, the dev_scores.json file will have multiple lines, and the get_result_table.py will only read the last row in the dev_scores.json.

[stefanhgm]

Hi Qinyuan, sorry for coming back to you so late. That was exactly what I was looking for. Thank you very much!