Feat: Retry individual eval runs

[jmnmv12]

Summary

When a single evaluation run fails from a validator or platform fault, the whole evaluation is thrown away and restarted from scratch.

This PR lets the platform grant a retry for just the failed run, in-session, while the validator still holds the evaluation.

When a validator reports a run as error, POST /validator/update-evaluation-run checks retry policy and, if it passes, responds with retry=true, the new attempt number, and a fresh artifact upload URL.

A new evaluation_run_attempts table records each attempt's status, error, cost, and lifecycle timestamps. The evaluation_runs row is kept as a mirror of the latest attempt, so every existing query, endpoint, and scoring path reads the same shape it always has. Granting a retry inserts attempt N+1 and resets the evaluation run to pending in a single transaction. The evaluation_run_logs table gains attempt_number in its primary key so each attempt keeps its own logs instead of overwriting the previous ones.

Existing tests were updated and new ones were added to make sure the new behaviour is valid.

Deployment notes

Configure the environment variables:

• MAX_ATTEMPTS_PER_EVALUATION_RUN

[jmnmv12]

I updated the inference gateway to track the cost per (eval run id, attempt number), but we are not using the inference gateway anymore or am I missing something? 🤔 @ibraheem-abe

[ibraheem-abe]

I updated the inference gateway to track the cost per (eval run id, attempt number), but we are not using the inference gateway anymore or am I missing something? 🤔 @ibraheem-abe

No no. Its deprecated now. We use sandbox proxy and directly call Openrouter

[jmnmv12]

I updated the inference gateway to track the cost per (eval run id, attempt number), but we are not using the inference gateway anymore or am I missing something? 🤔 @ibraheem-abe

No no. Its deprecated now. We use sandbox proxy and directly call Openrouter

Then I'll remove the updates I did there, thanks 👌