You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
In theory, if you run train_ec2.sh and exit before training completes, and then restart the job, it should pick up where it left off. But this doesn't actually work because on the second run, TF emits an out of memory error. We should isolate the exact conditions when this occurs and file an issue in the repo for TF Object Detection. We should also check to see if there's an issue already there.
The text was updated successfully, but these errors were encountered:
In theory, if you run
train_ec2.sh
and exit before training completes, and then restart the job, it should pick up where it left off. But this doesn't actually work because on the second run, TF emits an out of memory error. We should isolate the exact conditions when this occurs and file an issue in the repo for TF Object Detection. We should also check to see if there's an issue already there.The text was updated successfully, but these errors were encountered: