Evaluate: Qwen/Qwen1.5-XX-Chat

[ggbetz]

For {XX} in [0.5B, 1.8B, 4B, 7B, 14B, 32B and 72B]:

Check upon issue creation:

• The model has not been evaluated yet and doesn't show up on the CoT Leaderboard.
• There is no evaluation request issue for the model in the repo.
• The parameters below have been adapted and shall be used.

Parameters:

NEXT_MODEL_PATH=Qwen/Qwen1.5-XX-Chat
NEXT_MODEL_REVISION=main
NEXT_MODEL_PRECISION=bfloat16
MAX_LENGTH=2048 
GPU_MEMORY_UTILIZATION=0.7
VLLM_SWAP_SPACE=8

ToDos:

• Run cot-eval pipeline
• Merge pull requests for cot-eval results datats (> @ggbetz)
• Create eval request record to update metadata on leaderboard (> @ggbetz)

[ggbetz]

Qwen models fail to generate reasoning traces.
https://github.com/logikon-ai/cot-eval/blob/f9bfe8f757edbed49324df680214a24fbde37213/src/cot_eval/__main__.py#L139C1-L146C53