[Bug]: Runs may be resubmitted indefinitely

[r4victor]

dstack version

master

Python version

3.11

Host OS

macOS 13.4

Host Arch

ARM

What happened?

Currently, _run_job() only handles BackendError from the backends:

dstack/src/dstack/_internal/server/background/tasks/process_submitted_jobs.py

Line 247 in 23662eb

except BackendError as e:

If the backend raises a different exception, _process_submitted_job() fails and the run is processed indefinitely. This should be fixed by catching the unexpected exceptions as well and logging them with logger.exception().

CLI logs

No response

Server logs

No response

Runner logs

No response

Additional Information

No response