Version of a task should be configurable.

[PhilipMay]

The LightevalTask class has a VERSION. This is not configurable and fixed to 0. See:

lighteval/src/lighteval/tasks/lighteval_task.py

Line 166 in 11b4833

self.VERSION = 0

I think in the results table the version number is also displayed. See:

Task	Version	Metric	Value		Stderr
all		acc	0.8737	±	0.0094
		acc_norm	0.8737	±	0.0094
community:german_rag_eval:_average:0		acc	0.8737	±	0.0094
		acc_norm	0.8737	±	0.0094
community:german_rag_eval:choose_context_by_question:0	0	acc	0.7290	±	0.0141
		acc_norm	0.7290	±	0.0141
community:german_rag_eval:choose_question_by_context:0	0	acc	0.8490	±	0.0113
		acc_norm	0.8490	±	0.0113
community:german_rag_eval:context_question_match:0	0	acc	0.9770	±	0.0047
		acc_norm	0.9770	±	0.0047
community:german_rag_eval:question_answer_match:0	0	acc	0.9400	±	0.0075
		acc_norm	0.9400	±	0.0075

If I slightly change the task or the prompt (e.g. #171 ) - it would be nice to increase the version number. Should this be changed / added?

[jphme]

Would also appreciate this feature (and obviously someone thought of this already, given the existence of VERSION). 👍

[clefourrier]

Hi! Yep, we'll need to add the mechanism again - we had removed it to simplify the code, and because we did not want to fix any task version while still being in alpha.

[PhilipMay]

@clefourrier do you want me to suggest a PR or do you want to implement it?

[clefourrier]

If you have the time to open a PR I'd be grateful!
You'll need to add a default version argument to the TaskConfig, and edit the task_table.jsonl to set "version" to 0 there for all tasks, and edit your own to pass it to 1.

[PhilipMay]

If you have the time to open a PR I'd be grateful! You'll need to add a default version argument to the TaskConfig, and edit the task_table.jsonl to set "version" to 0 there for all tasks, and edit your own to pass it to 1.

I started here: #181

[PhilipMay]

This is done with PR #181. Closing...