Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Too excessive component scanning for repository implementation fragments causing slow bootstrap [DATACMNS-1172] #1614

Closed
spring-projects-issues opened this issue Sep 23, 2017 · 8 comments
Assignees
Milestone

Comments

@spring-projects-issues
Copy link

@spring-projects-issues spring-projects-issues commented Sep 23, 2017

Réda Housni Alaoui opened DATACMNS-1172 and commented

We are using Spring Data JPA 1.10.11.RELEASE, therefore using Spring Data Commons 1.12.11.RELEASE. Integration test boot is particularly slow on our biggest application. After some digging, we found out that Spring Data ignition via CustomRepositoryImplementationDetector.detectCustomImplementation() takes 45 seconds.

I didn't find any issue opened on Spring Framework or Spring Data about this. I didn't notice a massive change of this part of the source code in the incoming Spring Data 2.0. This discourages our collaborators from writing integration test


Affects: 1.12.11 (Hopper SR11), 2.0 RC3 (Kay), 1.13.7 (Ingalls SR7)

Attachments:

Referenced from: pull request #248

Backported to: 1.13.8 (Ingalls SR8), 1.12.12 (Hopper SR12)

2 votes, 5 watchers

@spring-projects-issues
Copy link
Author

@spring-projects-issues spring-projects-issues commented Sep 25, 2017

Oliver Drotbohm commented

Do you have more information you can add as otherwise it's really hard to act on what you describe. Looks like the actual bottleneck is a class in Spring Framework. I'd be interested in how many repository interfaces your project contains, and whether they're located in packages with many sub-packages

@spring-projects-issues
Copy link
Author

@spring-projects-issues spring-projects-issues commented Sep 25, 2017

Réda Housni Alaoui commented

Hello Oliver,

The app has:

  • 261 repositories
  • a depth of 5 below the root package provided to Spring
  • 365 jars in the final WAR

The scanned root package is present in a dozen of jars

@spring-projects-issues
Copy link
Author

@spring-projects-issues spring-projects-issues commented Sep 25, 2017

Réda Housni Alaoui commented

My laptop is a 2017 Razer Blade with nvme disk, 16 GB of RAM and Core i7 Quad Core 6700HQ running Ubuntu 17.
We have the issue on all dev machines

@spring-projects-issues
Copy link
Author

@spring-projects-issues spring-projects-issues commented Sep 25, 2017

Anthony Donnefort commented

According to the documentation:

If you use namespace configuration, the repository infrastructure tries to autodetect custom implementations by scanning for classes below the package we found a repository in.

This is not the case. CustomRepositoryImplementationDetector looks for the implementation anywhere in the base package.
We have a large codebase, looking for a single implementation that way takes hundreds of milliseconds. Multiply that by our 260 repositories, and you get startup times close to a minute.

Limiting the package search as documented would probably fix the problem

@spring-projects-issues
Copy link
Author

@spring-projects-issues spring-projects-issues commented Sep 27, 2017

Mark Paluch commented

I pushed a branch with a proposal that aligns scanning to how it's documented, we scan only for custom implementations in the package where the actual repository interface/fragment interface resides.

That's a change in comparison to how scanning works today. We need to extend our @Enable…Repositories annotations to allow configuration of the behavior:

@EnableJpaRepositories(limitImplementationBasePackages = false)
@Configuration
class AppConfiguration {
  //
}

@spring-projects-issues
Copy link
Author

@spring-projects-issues spring-projects-issues commented Sep 27, 2017

Mark Paluch commented

We deployed snapshot versions for Hopper and Ingalls that limit implementation scanning to the repository package. Care to give the change a spin?

Hopper:

<dependency>
  <groupId>org.springframework.data</groupId>
  <artifactId>spring-data-commons</artifactId>
  <version>1.12.12.DATACMNS-1172-SNAPSHOT</version>
</dependency>

Ingalls:

<dependency>
  <groupId>org.springframework.data</groupId>
  <artifactId>spring-data-commons</artifactId>
  <version>1.13.8.DATACMNS-1172-SNAPSHOT</version>
</dependency>

You can obtain the artifacts from our snapshot repository:

<repositories>
  <repository>
    <id>spring-libs-snapshot</id>
    <url>https://repo.spring.io/libs-snapshot</url>
  </repository>
</repositories>

@spring-projects-issues
Copy link
Author

@spring-projects-issues spring-projects-issues commented Sep 27, 2017

Anthony Donnefort commented

On two different usecases init time for spring data with this snapshot went from 30s to 5s, and 1m30 to 20s.
That's awesome !

@spring-projects-issues
Copy link
Author

@spring-projects-issues spring-projects-issues commented Sep 27, 2017

Oliver Drotbohm commented

That's now merged into Ingalls and Hopper bugfix branches as well as master for Kay. I didn't merge the addition of a config option top enable exhaustive scanning again, to see whether users were really relying on that

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

None yet
2 participants