fix: better exception handling in detached mode

[azhou-determined]

Ticket

Description

in _LogShipper and _Heartbeat, atexit handlers never get called, since _core_v2 registers an atexit on start, and on interpreter exit the latter atexit gets triggered and calls close() on _LogShipper and _Heartbeat which deregisters the handlers.

Test Plan

run a detached mode experiment that throws an error:

import random
import tempfile
from determined.experimental import core_v2


def main():
    checkpoint_storage = tempfile.mkdtemp()
    core_v2.init(
        defaults=core_v2.DefaultConfig(
            name="unmanaged-1-error",
        ),
        checkpoint_storage=checkpoint_storage,
    )

    for i in range(100):
        core_v2.train.report_training_metrics(steps_completed=i, metrics={"loss": random.random()})
        if i == 15:
            raise ValueError("oops")
        if (i + 1) % 10 == 0:
            core_v2.train.report_validation_metrics(
                steps_completed=i, metrics={"loss": random.random()}
            )
    core_v2.close()


if __name__ == "__main__":
    main()

ensure that the trial does not hang, logs show the value error, and trial/exp is terminated in an ERROR state.

Checklist

• Changes have been manually QA'd
• User-facing API changes need the "User-facing API Change" label.
• Release notes should be added as a separate file under docs/release-notes/.
  See Release Note for details.
• Licenses should be included for new code which was copied and/or modified from any external code.

[netlify]

✅ Deploy Preview for determined-ui canceled.

Name	Link
🔨 Latest commit	3d25260
🔍 Latest deploy log	https://app.netlify.com/sites/determined-ui/deploys/662024b65aa1d60008dee30e

[codecov]

Codecov Report

Attention: Patch coverage is 0% with 7 lines in your changes are missing coverage. Please review.

Project coverage is 45.47%. Comparing base (1037d83) to head (3d25260).
Report is 2 commits behind head on main.

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #9183      +/-   ##
==========================================
+ Coverage   45.44%   45.47%   +0.02%     
==========================================
  Files        1197     1197              
  Lines      147558   147549       -9     
  Branches     2437     2438       +1     
==========================================
+ Hits        67051    67091      +40     
+ Misses      80275    80226      -49     
  Partials      232      232              

Flag	Coverage Δ
backend	43.69% <ø> (-0.09%)	⬇️
harness	64.02% <0.00%> (+0.20%)	⬆️
web	35.41% <ø> (ø)

Flags with carried forward coverage won't be shown. Click here to find out more.

Files	Coverage Δ
harness/determined/core/_log_shipper.py	35.24% <ø> (-0.19%)	⬇️
harness/determined/core/_heartbeat.py	38.37% <0.00%> (-0.52%)	⬇️

... and 8 files with indirect coverage changes