BLOCKED "reactor-http-nio-*" threads under load [SPR-15874]

[spring-projects-issues]

Eduard Dautov opened SPR-15874 and commented

I created POC of webflux stub server and faced with performance problems - my application can't handle more than 1000 users simultaneously.
!slow.png|thumbnail!

I made some investigations and found the root cause - all "reactor-HTTP" threads spent a lot of time in BLOCKED state.
!slowVM.png|thumbnail!

"reactor-http-nio-2" #15 daemon prio=5 os_prio=31 tid=0x00007fc8589f6800 nid=0x5a03 waiting for monitor entry [0x00007000017d9000]
   java.lang.Thread.State: BLOCKED (on object monitor)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:404)
	- waiting to lock <0x00000006c0478b28> (a java.lang.Object)
	at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
	at org.springframework.util.ClassUtils.forName(ClassUtils.java:255)
	at org.springframework.http.converter.json.Jackson2ObjectMapperBuilder.registerWellKnownModulesIfAvailable(Jackson2ObjectMapperBuilder.java:737)
	at org.springframework.http.converter.json.Jackson2ObjectMapperBuilder.configure(Jackson2ObjectMapperBuilder.java:619)
	at org.springframework.http.converter.json.Jackson2ObjectMapperBuilder.build(Jackson2ObjectMapperBuilder.java:602)
	at org.springframework.http.codec.json.Jackson2JsonEncoder.<init>(Jackson2JsonEncoder.java:51)
	at org.springframework.http.codec.AbstractCodecConfigurer$AbstractDefaultCodecs.jackson2JsonEncoder(AbstractCodecConfigurer.java:171)
	at org.springframework.http.codec.DefaultServerCodecConfigurer$ServerDefaultCodecsImpl.getSseEncoder(DefaultServerCodecConfigurer.java:98)
	at org.springframework.http.codec.DefaultServerCodecConfigurer$ServerDefaultCodecsImpl.getObjectWriters(DefaultServerCodecConfigurer.java:90)
	at org.springframework.http.codec.AbstractCodecConfigurer.getWriters(AbstractCodecConfigurer.java:115)
	at org.springframework.web.reactive.function.server.support.ServerResponseResultHandler$1.messageWriters(ServerResponseResultHandler.java:100)
	at org.springframework.web.reactive.function.server.DefaultServerResponseBuilder$BodyInserterServerResponse$1.messageWriters(DefaultServerResponseBuilder.java:313)
	at org.springframework.web.reactive.function.BodyInserters.lambda$bodyInserterFor$17(BodyInserters.java:298)
	at org.springframework.web.reactive.function.BodyInserters$$Lambda$292/1708834600.insert(Unknown Source)
	at org.springframework.web.reactive.function.server.DefaultServerResponseBuilder$BodyInserterServerResponse.writeTo(DefaultServerResponseBuilder.java:310)
	at org.springframework.web.reactive.function.server.support.ServerResponseResultHandler.handleResult(ServerResponseResultHandler.java:96)
	at org.springframework.web.reactive.DispatcherHandler.handleResult(DispatcherHandler.java:144)
	at org.springframework.web.reactive.DispatcherHandler.lambda$handle$2(DispatcherHandler.java:131)
	at org.springframework.web.reactive.DispatcherHandler$$Lambda$255/801252068.apply(Unknown Source)
	at reactor.core.publisher.MonoFlatMap$FlatMapMain.onNext(MonoFlatMap.java:118)

This happens because of org.springframework.http.codec.AbstractCodecConfigurer.AbstractDefaultCodecs#jackson2JsonEncoder() - if you don't specify Jackson2JsonEncoder, Configurer will create new Encoder.

protected Jackson2JsonEncoder jackson2JsonEncoder() {
			return (this.jackson2JsonEncoder != null ? this.jackson2JsonEncoder : new Jackson2JsonEncoder());
		}

In my application, I avoid this using WebFluxConfigurer

@Override
    public void configureHttpMessageCodecs(ServerCodecConfigurer configurer) {
        boolean fastMode = environment.getProperty("turnOnFastMode", Boolean.class, false);
        if (fastMode) {
            configurer.defaultCodecs().jackson2JsonEncoder(new Jackson2JsonEncoder());
            configurer.defaultCodecs().jackson2JsonDecoder(new Jackson2JsonDecoder());
        }
    }

Results:
!fast.png|thumbnail!
!fastVM.png|thumbnail!

Source code of my application with Gatling tests https://github.com/Zajs/webflux-test
This bug can be related to #20143.

---

Affects: 5.0 RC3

Attachments:

• fast.png (64.62 kB)
• fastVM.png (126.80 kB)
• slow.png (78.15 kB)
• slowVM.png (148.72 kB)

Issue Links:

• #20371 AbstractCodecConfigurer creates Jackson Encoders/Decoders each time when getters are called ("duplicates")

[spring-projects-issues]

Rossen Stoyanchev commented

The issue with the Jackson encoder/decoder being re-created was fixed recently #20371. Can you try with the latest 5.0.0.BUILD-SNAPSHOT?

[spring-projects-issues]

Eduard Dautov commented

This fix works for me. Thank you.

[spring-projects-issues]

Rossen Stoyanchev commented

Thanks for confirming.