Add metric reporting for job step timeouts.

Summary:
Job step timeouts are one of the significant causes of build failure that
aren't test failure, they can indicate problems, and they (as far as I can tell)
aren't currently easy to track.
This adds a very simple counter for job steps that have timed out so we can
monitor that.

Test Plan: Unit

Reviewers: ar

Reviewed By: ar

Subscribers: wwu

Differential Revision: https://tails.corp.dropbox.com/D89455

changes/jobs/sync_job_step.py

@@ -6,7 +6,7 @@
 from sqlalchemy.sql import func
 
 from changes.constants import Status, Result
-from changes.config import db
+from changes.config import db, statsreporter
 from changes.db.utils import try_create
 from changes.models import (
     ItemOption, JobPhase, JobStep, JobPlan, TestCase, ItemStat,
@@ -159,6 +159,7 @@ def sync_job_step(step_id):
             })
 
             db.session.flush()
+            statsreporter.stats().incr('job_step_timed_out')
         if step.status != Status.in_progress:
             retry_after = QUEUED_RETRY_DELAY
         else: