feat: Only run one DB expiry check per cluster at a time

[DRK3]

Added a mechanism to the expiry service to allow it to coordinate with other Orb servers within a cluster so that they can ensure only one DB expiry check at a time.

Signed-off-by: Derek Trider Derek.Trider@securekey.com

closes #810

[codecov]

Codecov Report

Merging #824 (50b3fbf) into main (2addba7) will increase coverage by 0.05%.
The diff coverage is 89.47%.

@@            Coverage Diff             @@
##             main     #824      +/-   ##
==========================================
+ Coverage   89.38%   89.44%   +0.05%     
==========================================
  Files         144      144              
  Lines       12638    12700      +62     
==========================================
+ Hits        11297    11359      +62     
+ Misses        828      825       -3     
- Partials      513      516       +3     

Impacted Files	Coverage Δ
cmd/orb-server/startcmd/params.go	79.92% <81.81%> (+0.04%)	⬆️
pkg/store/expiry/expiry.go	89.83% <90.62%> (-1.22%)	⬇️
cmd/orb-server/startcmd/start.go	73.80% <100.00%> (ø)
pkg/observer/pubsub.go	94.44% <0.00%> (ø)
...pub/service/inbox/httpsubscriber/httpsubscriber.go	100.00% <0.00%> (+9.87%)	⬆️

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 2addba7...50b3fbf. Read the comment docs.

[DRK3]

@bstasyszyn I've updated this PR per our conversation earlier. Take a look and let me know. If it looks okay, I can improve the code coverage. Some of the cases to improve code coverage are going to be time-consuming to cover, so if I need to make any major changes I should do that first before writing unit tests for them.