Add new condition here for localization that hit quota issue; infinite retries

[julianhess]

canine/canine/orchestrator.py

Line 103 in 01ce0a7

echo -n $LOCALIZER_JOB_RC > $CANINE_JOB_ROOT/.localizer_exit_code

[julianhess]

Would need to catch exit code (5) from gcloud_exp_backoff here:

canine/canine/localization/base.py

Line 1289 in 01ce0a7

"gcloud compute instances attach-disk ${CANINE_NODE_NAME} --zone ${CANINE_NODE_ZONE} --disk ${CANINE_RODISK} --device-name ${CANINE_RODISK} --mode ro &>> $DIAG_FILE || true",

[julianhess]

Here too:

canine/canine/localization/base.py

Line 1435 in 01ce0a7

' gcloud compute instances detach-disk $CANINE_NODE_NAME --zone $CANINE_NODE_ZONE --disk $CANINE_RODISK || echo "Error detaching disk ${CANINE_RODISK}" >&2',

[julianhess]

Would want to make gcloud_exp_backoff retry less (so time spent retrying is lower); specify with variable maximum backoff time