[CON-457] add retry and exponential backoff to ensurePrimaryMiddlware

[jonaylor89]

Description

This PR adds retry logic to ensurePrimaryMiddleware with an exponential backoff to reduce strain on discovery nodes

Tests

No extra tests needed since each individual piece seems to be tested well enough

Monitoring - How will this change be monitored? Are there sufficient logs / alerts?

This change can be monitored with content node logs to make sure it actually retries as well as discovery provider grafana metrics more generally to see if there is in fact less strain.

[SidSethi]

hahah this will make things much worse - sorry the card was not very clear, but getReplicaSetSpIdsByUserId internally makes retries with a fixed delay (check out that code)
the scope of this is to actually replace that retry logic with asyncRetry

[jonaylor89]

hahah this will make things much worse - sorry the card was not very clear, but getReplicaSetSpIdsByUserId internally makes retries with a fixed delay (check out that code) the scope of this is to actually replace that retry logic with asyncRetry

Oh god it does have its own retry log RIP - reverting and taking another crack at it

[SidSethi]

need to change values, otherwise lgtm
this is critical path change (every user write) - good to ensure things seem ok locally and on staging