Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

BLOCKED "reactor-http-nio-*" threads under load [SPR-15874] #20429

Closed
spring-projects-issues opened this issue Aug 18, 2017 · 3 comments
Closed

BLOCKED "reactor-http-nio-*" threads under load [SPR-15874] #20429

spring-projects-issues opened this issue Aug 18, 2017 · 3 comments
Assignees
Labels
in: web status: duplicate

Comments

@spring-projects-issues
Copy link
Collaborator

@spring-projects-issues spring-projects-issues commented Aug 18, 2017

Eduard Dautov opened SPR-15874 and commented

I created POC of webflux stub server and faced with performance problems - my application can't handle more than 1000 users simultaneously.
!slow.png|thumbnail!

I made some investigations and found the root cause - all "reactor-HTTP" threads spent a lot of time in BLOCKED state.
!slowVM.png|thumbnail!

"reactor-http-nio-2" #15 daemon prio=5 os_prio=31 tid=0x00007fc8589f6800 nid=0x5a03 waiting for monitor entry [0x00007000017d9000]
   java.lang.Thread.State: BLOCKED (on object monitor)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:404)
	- waiting to lock <0x00000006c0478b28> (a java.lang.Object)
	at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
	at org.springframework.util.ClassUtils.forName(ClassUtils.java:255)
	at org.springframework.http.converter.json.Jackson2ObjectMapperBuilder.registerWellKnownModulesIfAvailable(Jackson2ObjectMapperBuilder.java:737)
	at org.springframework.http.converter.json.Jackson2ObjectMapperBuilder.configure(Jackson2ObjectMapperBuilder.java:619)
	at org.springframework.http.converter.json.Jackson2ObjectMapperBuilder.build(Jackson2ObjectMapperBuilder.java:602)
	at org.springframework.http.codec.json.Jackson2JsonEncoder.<init>(Jackson2JsonEncoder.java:51)
	at org.springframework.http.codec.AbstractCodecConfigurer$AbstractDefaultCodecs.jackson2JsonEncoder(AbstractCodecConfigurer.java:171)
	at org.springframework.http.codec.DefaultServerCodecConfigurer$ServerDefaultCodecsImpl.getSseEncoder(DefaultServerCodecConfigurer.java:98)
	at org.springframework.http.codec.DefaultServerCodecConfigurer$ServerDefaultCodecsImpl.getObjectWriters(DefaultServerCodecConfigurer.java:90)
	at org.springframework.http.codec.AbstractCodecConfigurer.getWriters(AbstractCodecConfigurer.java:115)
	at org.springframework.web.reactive.function.server.support.ServerResponseResultHandler$1.messageWriters(ServerResponseResultHandler.java:100)
	at org.springframework.web.reactive.function.server.DefaultServerResponseBuilder$BodyInserterServerResponse$1.messageWriters(DefaultServerResponseBuilder.java:313)
	at org.springframework.web.reactive.function.BodyInserters.lambda$bodyInserterFor$17(BodyInserters.java:298)
	at org.springframework.web.reactive.function.BodyInserters$$Lambda$292/1708834600.insert(Unknown Source)
	at org.springframework.web.reactive.function.server.DefaultServerResponseBuilder$BodyInserterServerResponse.writeTo(DefaultServerResponseBuilder.java:310)
	at org.springframework.web.reactive.function.server.support.ServerResponseResultHandler.handleResult(ServerResponseResultHandler.java:96)
	at org.springframework.web.reactive.DispatcherHandler.handleResult(DispatcherHandler.java:144)
	at org.springframework.web.reactive.DispatcherHandler.lambda$handle$2(DispatcherHandler.java:131)
	at org.springframework.web.reactive.DispatcherHandler$$Lambda$255/801252068.apply(Unknown Source)
	at reactor.core.publisher.MonoFlatMap$FlatMapMain.onNext(MonoFlatMap.java:118)

This happens because of org.springframework.http.codec.AbstractCodecConfigurer.AbstractDefaultCodecs#jackson2JsonEncoder() - if you don't specify Jackson2JsonEncoder, Configurer will create new Encoder.

protected Jackson2JsonEncoder jackson2JsonEncoder() {
			return (this.jackson2JsonEncoder != null ? this.jackson2JsonEncoder : new Jackson2JsonEncoder());
		}

In my application, I avoid this using WebFluxConfigurer

@Override
    public void configureHttpMessageCodecs(ServerCodecConfigurer configurer) {
        boolean fastMode = environment.getProperty("turnOnFastMode", Boolean.class, false);
        if (fastMode) {
            configurer.defaultCodecs().jackson2JsonEncoder(new Jackson2JsonEncoder());
            configurer.defaultCodecs().jackson2JsonDecoder(new Jackson2JsonDecoder());
        }
    }

Results:
!fast.png|thumbnail!
!fastVM.png|thumbnail!

Source code of my application with Gatling tests https://github.com/Zajs/webflux-test
This bug can be related to #20143.


Affects: 5.0 RC3

Attachments:

Issue Links:

  • #20371 AbstractCodecConfigurer creates Jackson Encoders/Decoders each time when getters are called ("duplicates")
@spring-projects-issues
Copy link
Collaborator Author

@spring-projects-issues spring-projects-issues commented Aug 18, 2017

Rossen Stoyanchev commented

The issue with the Jackson encoder/decoder being re-created was fixed recently #20371. Can you try with the latest 5.0.0.BUILD-SNAPSHOT?

@spring-projects-issues
Copy link
Collaborator Author

@spring-projects-issues spring-projects-issues commented Aug 18, 2017

Eduard Dautov commented

This fix works for me. Thank you.

@spring-projects-issues
Copy link
Collaborator Author

@spring-projects-issues spring-projects-issues commented Aug 18, 2017

Rossen Stoyanchev commented

Thanks for confirming.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
in: web status: duplicate
Projects
None yet
Development

No branches or pull requests

2 participants