Release 2329
Trello card
Context
The yabeda-delayed_job gem adds a plugin to Delayed::Job that sends metric information via Prometheus. A problem encountered with it is that the job payload is included under the worker label. As some of our job payloads have PII it results in personal data ending up in our metrics store.
Changes proposed in this pull request
- Switch to fork of yabeda-delayed_job
Prevents PII appearing in the metric labels. You can find the fork here and respective commit here.
Guidance to review
I've forked the gem we use to fix it rather than monkey patch it (which was my first avenue but felt too brittle for something like this). If we're happy with this I'll see if I can improve the change to work better for when active job is not being used and upstream the change to the gem.
Example metric after change:
delayed_job_jobs_enqueued_total{app="review-school-experience-2329",app_instance="0",organisation="dfe",space="get-into-teaching",queue="default",worker="Candidates::Registrations::PlacementRequestJob"} 1.0
Initially enables metrics only in staging so I can be sure there's no PII leaking elsewhere.