-
Notifications
You must be signed in to change notification settings - Fork 0
Experiment Page Training Status
Fionna Chan edited this page Jun 24, 2020
·
2 revisions
These are applicable to non-draft trainings.
| RunStatus Java Constant | Condition | On Experiment Page |
|---|---|---|
| NotStarted | Training not started | Loading icon |
| Starting | Starting training | Loading icon |
| Running | Training in progress | Loading icon |
| Completed | Training completed | Checkmark icon |
| Error | Errors in Runs | Exclamation mark error icon |
| Killed | Stopped manually | Grey stopped icon |
| Killed (Mapped to Error) | AWS killed the nodetraining_error_id column in Run table is filled |
Exclamation mark error icon |
| Restarting | Training restarting | Loading icon |
| Stopping | Training stopping | Grey stopped icon |
For details of "Killed" UI status, see #1122
@ejunprung: this is how it works rigth now in web app side (actually, as you'll see, not only web app):
- there is updater. Updater is a process that run each X seconds (60s now, I think) and check for each run if there is info about the run in S3. i.e. S3 is the way training comunicates with Updater
- if there is info about the run in S3, updater gets the info, process it, save in database, generate a message and put in SQS
- webapp has a listener for SQS messages. When it receives one, it will update the UI if needed.
- whenever the user goes to a experiment view, the info for the run is read from the db. so, afaik, we don't read process_output-XX.log in updater. Actually, I don't even know if this file is saved in S3. If, however, this file is processed by some entity and generate the S3 files that we actually read, pardon me :) I don't really know too much about training phase.