feat: rudder-server modification to use suppression-backup service.

[saurav-malani]

Description

Modified rudder-server suppression component to make use of suppression-backup-service to get latest & complete user suppression list instead of syncing with data-regulation-service. This modification would result is drastically reducing the suppression sync time at gateway.

Here, we make sure that gateway starts once we have the latest users suppressions. Also, we start getting the full suppression list asynchronously in a separate badgerdb repo, since it might take sometime. Once, we have the full list in badgerdb. We simply swap the old badgerdb (with latest users) with the new badgerdb (with all users).

Notion Ticket

https://www.notion.so/rudderstacks/Speed-up-resync-of-suppress-regulations-b29f4df9d10047c7ae3ae7d80781b9bb?pvs=4

[codecov]

Codecov Report

Patch coverage: 61.46% and project coverage change: +0.11 🎉

Comparison is base (d7ac3a8) 52.28% compared to head (8bf9f2e) 52.40%.

Additional details and impacted files

@@            Coverage Diff             @@
##           master    #3116      +/-   ##
==========================================
+ Coverage   52.28%   52.40%   +0.11%     
==========================================
  Files         321      322       +1     
  Lines       52789    52944     +155     
==========================================
+ Hits        27602    27745     +143     
+ Misses      23527    23526       -1     
- Partials     1660     1673      +13     

Impacted Files	Coverage Δ
enterprise/suppress-user/repo_switch.go	28.57% <28.57%> (ø)
runner/runner.go	69.88% <42.10%> (ø)
...rprise/suppress-user/internal/badgerdb/badgerdb.go	78.53% <58.82%> (-1.47%)	⬇️
enterprise/suppress-user/factory.go	69.00% <69.12%> (+59.57%)	⬆️
regulation-worker/cmd/main.go	71.95% <100.00%> (ø)

... and 6 files with indirect coverage changes

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

☔ View full report in Codecov by Sentry.
📢 Do you have feedback about the report comment? Let us know in this issue.

[fracasula]

It feels like it would be better to log the error in the anonymous function instead of logging it here. This way we can add some context to the error like:

m.retryIndefinitely(ctx, func() error {
	if _, err = os.Create(filepath.Join(fullSuppressionPath, model.SyncDoneMarker)); err != nil {
		return fmt.Errorf("could not create sync marker: %v", err)
	}
	return nil
}, 1*time.Second)

Also, I'm not sure trying these ops forever without anybody looking at the logs is a good idea.
For example, let's say that it fails forever at the 2nd step (i.e. fullSyncer.Sync(ctx)). How are we going to notice?

cc @atzoum

[atzoum]

Alerting for sync failures is something we are missing for the whole suppression feature (not just for this scenario). Let's treat alerting in another task