feat: rudder-server modification to use suppression-backup service.

[saurav-malani]

Description

Modified rudder-server suppression component to make use of suppression-backup-service to get latest & complete user suppression list instead of syncing with data-regulation-service. This modification would result is drastically reducing the suppression sync time at gateway.

Here, we make sure that gateway starts once we have the latest users suppressions. Also, we start getting the full suppression list asynchronously in a separate badgerdb repo, since it might take sometime. Once, we have the full list in badgerdb. We simply swap the old badgerdb (with latest users) with the new badgerdb (with all users).

Notion Ticket

https://www.notion.so/rudderstacks/Speed-up-resync-of-suppress-regulations-b29f4df9d10047c7ae3ae7d80781b9bb?pvs=4

[codecov]

Codecov Report

Attention: Patch coverage is 62.37113% with 73 lines in your changes missing coverage. Please review.

Project coverage is 52.40%. Comparing base (d7ac3a8) to head (8bf9f2e).
Report is 1207 commits behind head on master.

Files with missing lines	Patch %	Lines
enterprise/suppress-user/factory.go	69.12%	32 Missing and 14 partials ⚠️
enterprise/suppress-user/repo_switch.go	28.57%	20 Missing ⚠️
...rprise/suppress-user/internal/badgerdb/badgerdb.go	58.82%	5 Missing and 2 partials ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##           master    #3116      +/-   ##
==========================================
+ Coverage   52.28%   52.40%   +0.11%     
==========================================
  Files         321      322       +1     
  Lines       52789    52944     +155     
==========================================
+ Hits        27602    27745     +143     
+ Misses      23527    23526       -1     
- Partials     1660     1673      +13     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[fracasula]

It feels like it would be better to log the error in the anonymous function instead of logging it here. This way we can add some context to the error like:

m.retryIndefinitely(ctx, func() error {
	if _, err = os.Create(filepath.Join(fullSuppressionPath, model.SyncDoneMarker)); err != nil {
		return fmt.Errorf("could not create sync marker: %v", err)
	}
	return nil
}, 1*time.Second)

Also, I'm not sure trying these ops forever without anybody looking at the logs is a good idea.
For example, let's say that it fails forever at the 2nd step (i.e. fullSyncer.Sync(ctx)). How are we going to notice?

cc @atzoum

[atzoum]

Alerting for sync failures is something we are missing for the whole suppression feature (not just for this scenario). Let's treat alerting in another task