fix: await_first_trial errors when Experiment is terminal

[wes-turner]

Pre-patch, when running await_first_trial on an Experiment the method would never exit if the experiment that backed it was in a terminal state and no trials started. It now checks that the underlying experiment is not terminal, and raises an exception if it becomes such (believing no trial will ever be started).

Side effect: Like wait, await_first_trial now reloads the underlying Experiment object.

MD-335

Description

Test Plan

Find or create an experiment that either fails or is canceled before any trials start. Then, in the SDK:

from determined.experimental import client
client.login(<login>)
exp = client.get_experiment(<id>)
exp.await_first_trial()

Also try awaiting a trial on experiments that actually do have trials. Or any weird cases you can think of.

Commentary (optional)

Checklist

• Changes have been manually QA'd
• User-facing API changes need the "User-facing API Change" label.
• Release notes should be added as a separate file under docs/release-notes/.
  See Release Note for details.
• Licenses should be included for new code which was copied and/or modified from any external code.

Ticket

[netlify]

✅ Deploy Preview for determined-ui canceled.

Name	Link
🔨 Latest commit	587d6cd
🔍 Latest deploy log	https://app.netlify.com/sites/determined-ui/deploys/65fa0324375d15000817cd42

[codecov]

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 47.80%. Comparing base (a603f4c) to head (587d6cd).

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #9022      +/-   ##
==========================================
+ Coverage   47.78%   47.80%   +0.01%     
==========================================
  Files        1161     1161              
  Lines      143441   143471      +30     
  Branches     2372     2372              
==========================================
+ Hits        68547    68580      +33     
+ Misses      74741    74738       -3     
  Partials      153      153              

Flag	Coverage Δ
backend	42.96% <ø> (+0.01%)	⬆️
harness	63.84% <100.00%> (+0.01%)	⬆️
web	40.86% <ø> (ø)

Flags with carried forward coverage won't be shown. Click here to find out more.

Files	Coverage Δ
...rness/determined/common/experimental/experiment.py	71.95% <100.00%> (+0.46%)	⬆️
.../determined/common/experimental/test_experiment.py	98.94% <100.00%> (+0.16%)	⬆️

... and 4 files with indirect coverage changes

[gt2345]

LGTM, thanks for the improvement

[azhou-determined]

lgtm