scheduled jobs: avoid "top-of-the-hour" service level degradation

[mwang1026]

When creating a schedule it's common to use simple cron syntax of '@daily' or '@hourly'. The downside of that is if every schedule is created with that syntax, cron runs at at the top of the hour / day. And if every service across all of your systems are using similar scheduling syntax then you get times where your services are hosed.

What we're looking for here is something where you can create schedules to avoid the above issues. "Random" or some sort of random jitter is probably a good enough heuristic to spread schedules out to avoid the problem. The ideal is to have a solution that can identify the best times to run to avoid "top of the hour" degradation.

For certain scheduled operations, you also want them to be on a consistent cadence but on staggered start times (e.g. every hour on the 23rd minute) -- namely Backups so that you can target an RPO.

• Deliverables *

• A way to specify randomness of the start time within the recurring window
• That randomness is applied on the initial schedule and the specified cadence determines future schedule times

• Other alternatives *

• Load based determination of when a schedule should run
• Randomness that's always applied on every run

Epic CRDB-7909

Jira issue: CRDB-3742

[blathers-crl]

cc @cockroachdb/disaster-recovery

[dt]

I don't think we expect to implement this in the jobs system any time soon; users who are used to @houly in their cron system of choice may well expect it to be a shorthand for exactly 0 * * * *, as it is in most cron implementations, and those who want a random minute of the hour can already do that, just be using something like (random()*60)::int::string || ' * * * *' when they create their schedule instead of @hourly.

I think we could probably close this as unplanned.

[rafiss]

Also noting that the Schema Telemetry Job does some of its own randomization before choosing @weekly or @hourly:

cockroach/pkg/sql/catalog/schematelemetry/schematelemetrycontroller/controller.go

Lines 188 to 214 in 9c510f9

	// MaybeRewriteCronExpr is used to rewrite the interval-oriented cron exprs
	// into an equivalent frequency interval but with an offset derived from the
	// uuid. For a given pair of inputs, the output of this function will always
	// be the same. If the input cronExpr is not a special form as denoted by
	// the keys of cronExprRewrites, it will be returned unmodified. This rewrite
	// occurs in order to uniformly distribute the production of telemetry logs
	// over the intended time interval to avoid bursts.
	func MaybeRewriteCronExpr(id uuid.UUID, cronExpr string) string {
	if f, ok := cronExprRewrites[cronExpr]; ok {
	hash := fnv.New64a() // arbitrary hash function
	_, _ = hash.Write(id.GetBytes())
	return f(rand.New(rand.NewSource(int64(hash.Sum64()))))
	}
	return cronExpr
	}
	var cronExprRewrites = map[string]func(r *rand.Rand) string{
	cronWeekly: func(r *rand.Rand) string {
	return fmt.Sprintf("%d %d * * %d", r.Intn(60), r.Intn(23), r.Intn(7))
	},
	cronDaily: func(r *rand.Rand) string {
	return fmt.Sprintf("%d %d * * *", r.Intn(60), r.Intn(23))
	},
	cronHourly: func(r *rand.Rand) string {
	return fmt.Sprintf("%d * * * *", r.Intn(60))
	},
	}

Other internal jobs could do something like this too.