Skip to content

Conversation

@bric3
Copy link
Contributor

@bric3 bric3 commented Jan 26, 2026

Fix OutOfMemoryError with Eclipse-based formatters on large projects

Fixes #2788, slow and memory hungry configuration phase when using Eclipse based formatters.

Problem

After upgrading to Spotless 8.x, in large multi-project builds (e.g., DataDog/dd-trace-java with now ~630 subprojects)
experience OutOfMemoryError when using GrEclipse, an Eclipse-based formatter, (all Equo/P2 based formatters
are affected: Eclipse JDT, GrEclipse, Eclipse CDT).

The P2 repository queries and nested JAR extraction were executed for every Spotless task during Gradle configuration.
While Maven dependencies benefit from DedupingProvisioner caching, P2 provisioning did not, causing:

  • Repeated expensive P2 repository queries
  • Multiple nested JAR extractions from P2 bundles
  • Massive memory usage scaling with the number of subprojects

Proposed change

Introduced a new P2Provisioner interface (parallel to the existing Provisioner) to enable build-tool-specific
caching strategies for Eclipse P2 dependencies.

  1. P2Provisioner interface (lib-extra)
  2. DedupingP2Provisioner (plugin-gradle) - Gradle-specific implementation that caches P2 resolution results
    across all tasks, similar to DedupingProvisioner for Maven dependencies
  3. P2 supported with spotlessPredeclare

Backward Compatibility

  • Maven plugin behavior unchanged (But passes the provisioner implementation)
  • Gradle plugin gains automatic caching without configuration changes
  • Existing test suites pass without modification

Testing

  • Added new unit tests, especially for
    • the GradleProvisioner
    • the spotlessPredeclare

Performance Impact

Expected improvements for large multi-project builds using Eclipse formatters:

  • Memory usage: P2 resolution happens once per provisioner instance instead of once per task
  • Configuration time: Cached P2 queries eliminate repeated repository access
  • Predeclare mode: Enforces all P2 dependencies are resolved upfront during spotlessPredeclare task

On dd-trace-java, the peak usage appear to be ~2GiB.


About changes, I intentionally didn't added any changes to the maven-plugin since there are no user-visible changes I think.

@bric3
Copy link
Contributor Author

bric3 commented Jan 26, 2026

Notice the different shape of the profiles, they are no more dominated by NestedJars.extractAllNestedJars (Compared to #2788 (comment)).

Allocations

8 14 3-allocation-raw-flames

Now the main CPU from this plugin is related to the way FileSignature$Cache.sign works, it appears in ~5% of the samples, which correlates with monitor profile.

image
CPU

8 14 3-cpu-raw-flames

Before, there was significant contention due to Jar extraction, with that change that shifted to FileSignature$Cache.sign (this PR didn't explore avenue to improve on that front).

Monitor Blocked

8 14 3-monitor-blocked-simplified-flames

@bric3
Copy link
Contributor Author

bric3 commented Jan 26, 2026

FYI, one can build the spotless project locally, install it in the local maven repository, then use it in the project. Adapt as you see fit:

$ ./gradlew spotlessApply --init-script spotless-override.init.gradle.kts -PspotlessVersion=8.2.1-SNAPSHOT

=> ~21s on the second run, first run is ~50s.

image
spotless-override.init.gradle.kts
logger.info("=== Spotless Override Init Script ===")

val processedSettings = mutableSetOf<String>()

beforeSettings {
  val settingsId = settings.rootDir.absolutePath
  val isSubproject = settings.rootDir.name == "buildSrc" ||
    settings.rootDir.absolutePath.contains("/buildSrc/")

  logger.info("  Processing settings for: ${settings.rootProject.name}")
  logger.info("    Root dir: ${settings.rootDir}")
  logger.info("    Is subproject: $isSubproject")

  // Only apply plugin substitution to the main root, not to buildSrc subprojects
  if (!isSubproject && settingsId !in processedSettings) {
    // Version must be provided via -PspotlessVersion=x.y.z
    val spotlessVersion = gradle.startParameter.projectProperties["spotlessVersion"]
      ?: throw GradleException("Property 'spotlessVersion' is required. Use: -PspotlessVersion=x.y.z")

    logger.info("  Target Spotless version: $spotlessVersion")
    processedSettings.add(settingsId)

    pluginManagement {
      repositories {
        mavenLocal()
        gradlePluginPortal()
        mavenCentral()
      }

      resolutionStrategy {
        eachPlugin {
          if (requested.id.id == "com.diffplug.spotless") {
            logger.info("  -> Substituting Spotless plugin to $spotlessVersion (requested: ${requested.version})")
            useVersion(spotlessVersion)
          }
        }
      }
    }
  } else if (settingsId !in processedSettings) {
    processedSettings.add(settingsId)
    logger.info("  -> Skipping plugin substitution for subproject to avoid conflicts")

    // Just ensure repositories are configured
    pluginManagement {
      repositories {
        mavenLocal()
        gradlePluginPortal()
        mavenCentral()
      }
    }
  }
}

While Spotless 8.2.0, on the second run (the first run is 2m at the very least):

./gradlew spotlessApply

=> ~1m

image

@nedtwigg
Copy link
Member

Great fix, thanks!

@nedtwigg nedtwigg merged commit 4864bc8 into diffplug:main Jan 27, 2026
20 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Spotless failed with OutOfMemory on large project (https://github.com/DataDog/dd-trace-java)

2 participants