Evaluation validation & restart

[czaloom]

Issue Description

Evaluations dataset/model validation is occurring within the computation causing silent failure.

Example

{"method": "compute_detection_metrics", "event": "Valor Exception: Evaluation '136'", "level": "error", "timestamp": "2024-04-19T21:24:08.293418Z", "exception": [{"exc_type": "RuntimeError", "exc_value": "Model '2fK6nXvtq9JfMnZntWiI7kLMH94_detect' does not meet filter requirements.", "syntax_error": null, "is_cause": false, "frames": [{"filename": "/src/valor_api/backend/metrics/metric_utils.py", "lineno": 292, "name": "wrapper", "line": "", "locals": {"args": "()", "kwargs": "\"{'db': <sqlalchemy.orm.session.Session object at 0x7fc75681a1d0>, 'evaluation_id\"+7", "db": "<sqlalchemy.orm.session.Session object at 0x7fc75681a1d0>", "evaluation_id": "136", "e": "'RuntimeError(\"Model \\'2fK6nXvtq9JfMnZntWiI7kLMH94_detect\\' does not meet filter re'+13", "fn": "<function compute_detection_metrics at 0x7fc759deec20>"}}, {"filename": "/src/valor_api/backend/metrics/detection.py", "lineno": 941, "name": "compute_detection_metrics", "line": "", "locals": {"db": "<sqlalchemy.orm.session.Session object at 0x7fc75681a1d0>", "evaluation_id": "136", "_": "()", "evaluation": "<valor_api.backend.models.Evaluation object at 0x7fc756818fa0>", "groundtruth_filter": "\"Filter(dataset_names=['2V2Z9CNQHCjuYu0R0XLCzFDlfD1_Object_Detection'], dataset_m\"+374", "prediction_filter": "\"Filter(dataset_names=['2V2Z9CNQHCjuYu0R0XLCzFDlfD1_Object_Detection'], dataset_m\"+408", "parameters": "\"EvaluationParameters(task_type=<TaskType.OBJECT_DETECTION: 'object-detection'>, \"+251", "datasets": "\"[(34, '2V2Z9CNQHCjuYu0R0XLCzFDlfD1_Object_Detection', {'task': 'Object Detection\"+376", "model": "None"}}]}]}

Expected Behavior

If a dataset or model has no data that conforms to the evaluation filter, then the job should return as Done with no metrics.

Other Additions

• Evaluation jobs with Failed statuses will restart if queried.

[rsbowman-striveworks]

The change to mark evaluations with no data as DONE makes sense to me. I'm open to the other changes but less sure they're the right thing to do.

waiting for thoughts on Sean's comments