How to evaluate geobench

[jyshee]

How can I reproduce the results of geobench in paper？Any eval code or prompt uaed?

[davendw49]

We share the original evaluation code in folder evaluation, and we will release end to end version in the near future, along with Geo-Eval.
p.s. The evaluation methods can be various, we will also take others into consideration, like ppl and exactly match.

[jyshee]

Thanks for the code.
It seems the data file ../data/geobench/npee.json not exists?

[davendw49]

It's only the name of the benchmark split, you can find it in folder data or hf repo.