What is the preferred dependency shape for (very) large multi-project setups?

[jmvandebron]

At one of my clients we're reaching the limits of what sbt can handle (or so it seems). We have a monorepo that contains 1 big sbt project with 150+ sub-projects/modules, besides several other non-sbt projects (frontend stuff).
sbt has 175.000+ settings, and takes about 3 to 5 minutes to reload when ppl are switching branches and/or updating the build.sbt (updating dependsOn or updating libraryDependencies).

There are very many sub-projects, each with dependsOn to 5 to 10 other sub projects (with a lot of overlap - most of them depend on the same sub-set of projects). Each sub-project can add to its libraryDependencies, but always (99% of the cases :| ) to the same version of external libraries.

We do not do cross-version builds.

What is the preferred way to structure such a large sbt project (ours currently is both wide AND deep):

• wide - lots of projects, but only 1 or 2 layers of dependsOn
• deep - lots of projects, with many layers of dependsOn
• small - fewer projects that each contain more
• other?

What are the ballpark limits on number of sub-projects and depth of dependsOn to keep an sbt project manageable on macbook pro 16Gb.

I'd appreciate pointers to resources that explain appropriate ways to (re)structure big sbt projects.

[adpi2]

That's an interesting question and I don't have a definitive answer for you.

I would not suspect the deepness of dependsOn to have any impact on the reload time, but maybe I am wrong. IMO It is rather a question of numbers of settings, the dependencies between those settings, and the computation time of each setting.

I know a project that contains 200+ projects, and 200.000+ settings, lots of projects with about 1 or 2 layers of dependsOn, and it takes about 30 seconds to load on my laptop. That project is softwaremill/sttp.

It would be useful to know what is taking so long to load in your project using a CPU profiler such as async-profiler and Flame Graph visualization.

Also do you use a recent version of sbt?

[jmvandebron]

We're on sbt 1.8.2.
I'll look into ways to track the computation times of settings/tasks to see if there are bottlenecks there.
I've also cloned sttp to see how that project performs wrt update and reload. It takes about 25s to reload, and 40s to update, so it might serve as an example of how to structure larger sbt multi-projects.

[eed3si9n]

Back in 2018 Johannes (@jrudolph) did some analysis - #4012, and maybe it's something we should review for sbt 1.x if not sbt 2.x to see something we could revive, for example using fork/join instead of thread pool for some concurrency.

[mzuehlke]

There is another issue:

• update gets slower with the number of modules in project #5508

with the workaround to enhance the speed of sbt update by specifying COURSIER_PROGRESS=false

[jmvandebron]

@adpi2 , you wrote

... IMO It is rather a question of numbers of settings, the dependencies between those settings, and the computation time of each setting....

What can I do to get insights into these 'dependencies between settings' and 'computation time of each setting'?

[dragos]

Sbt is measuring the time it takes to load and resolve projects, and even settings evaluation, see calles to timed around here. However, I haven't been able to make Sbt log those since the very early debug level is hard coded to Info.