Autodiscover sharding for large number of repos

[ChrisSteinbach]

Tell us more.

TL;DR Introduce a deterministic sharding mechanism for autodiscovered repositories so self-hosted Renovate can scale across large numbers of repos, either via parallel jobs or split serial runs—reducing pressure from CI time windows, access token lifetimes, and shared infra limits.

Motivation

• Large self-hosted setups often need to process hundreds or thousands of repos.
• Running everything in one job can hit:
  • Access token validity windows
  • Shared test/build server resource caps
• Splitting the workload across deterministic shards lets us:
  • Run N parallel jobs, each handling a distinct subset
• Or run N serial jobs in sequence (or on a schedule) to keep within limits
  • Avoid overlap/duplication without having to maintain static repo lists

Proposal overview Add global config to shard the autodiscovered repository list:

• autodiscoverShardCount: number of shards (integer > 0)
• autodiscoverShardSelector: which shard(s) to run (string syntax below)
• autodiscoverShardSalt: optional salt used in hashing, to control the partitioning

Behavior

• For each repo full name, compute a sha256 hash (optionally salted), map to [0, shardCount-1], and only include repos whose shard index matches the selector.
• Deterministic across runs when salt and shard count are stable.
• Sharding applies only when autodiscover is enabled and both shardCount and selector are set.

Selector syntax

• Single shard index: "0"
• Range: "0-3" (inclusive)
• Modular class: "*/k+o"
  • Example: "*/4+1" selects shard indices where i % 4 == 1
• Comma-separated combinations: "0,3,*/5"
  • Tokens that don’t parse are ignored

See #38614

[jamietanna]

Hey @ChrisSteinbach, thanks for sharing this feedback!

Unfortunately this isn't something that we'll be adding into the Renovate CLI, for a few reasons:

• adding complex functionality like this into the CLI codebase will increase complexity, and may require increased work to ensure that this is well-tested and regressions are not introduced
• renovate (the CLI) is intended to be stateless, and reduce complexity when running it
• managing this scheduling outside of the CLI is a more scalable option

We (Mend) have found that folks that do want this sort of functionality can do so using the Mend Renovate Community Server (commercial-but-free) or Mend Renovate Enterprise Server (commercial and paid).

An alternative option (used by a number of organisations/companies/enterprises - including ones I've worked at in the past) is to extract this logic out, into an "orchestration" layer over Renovate, which will perform this logic themselves, and then i.e. schedule an Argo Workflow, or an Elastic Container Service (ECS) task to process the job.

This leaves the Renovate CLI's logic fairly lean, and allows you to do more interesting things when you control that logic yourself.

With this in mind, we're going to close this as "resolved" (as there's no "won't do" in a Discussion) and mark #38614 as a "no".

I appreciate that this will probably be a disappointing response, sorry about that - but I hope you understand.