There is an unknown problem with transaction support for reactive projects

[ht-coder-wu]

Description

A simple demo project, using spring boot+webflux+spring data-r2dbc, stress testing, and some requesters hang for a long time until timeout.
I'm not sure which link caused the suspension，but i did some experiments at https://github.com/ht-coder-wu/demo.
I think :

1. there must be some problems with connection pool or transactions
2. request is suspended for some unknown reasons in the concurrent scenario for a long time

Version

• spring-boot-starter-parent 2.7.8
• r2dbc-mysql 0.8.2.RELEASE
• r2dbc-pool 0.9.2.RELEASE
• jdk 8

Recurrence（occasional）

application.properties like this:
spring.application.name=demo
spring.r2dbc.url=r2dbcs:mysql://localhost:3306/test
spring.r2dbc.username=root
spring.r2dbc.password=root
spring.r2dbc.pool.enabled=true
server.port=8080
logging.level.org.springframework.r2dbc=info
spring.r2dbc.pool.initial-size=100
spring.r2dbc.pool.max-size=500

I did four sets of tests:

Their difference is :
testA Declarative transaction+request db
testB Declarative transaction+do not request db
testC no transaction+do not request db
testD no transaction+request db

I use scripts for concurrency testing:
iwtbafpdeMacBook-Pro:~ iwtbafp$ ab -n 100000 -c 100 -s 120 http://localhost:8080/testA

there are 4 requests timeout(cause i set timeout 100s in webfilter and do response status 400),If I don't restart the service, continue to pressure test testD

request is suspended for some unknown reasons in the concurrent scenario for a long time

iwtbafpdeMacBook-Pro:~ iwtbafp$ ab -n 100000 -c 100 -s 120 http://localhost:8080/testD

It looks like there's no problem but there are some tx hang in db

If i restart the service after pressure test testA and test testD directly
iwtbafpdeMacBook-Pro:~ iwtbafp$ ab -n 100000 -c 100 -s 120 http://localhost:8080/testD

request is suspended for some unknown reasons in the concurrent scenario for a long time

I think there must be some problems with connection pool or transactions,It looks like testA causes some connections in the thread pool to no longer be submitted automatically,then testD which not in transaction hang in db tx.

there must be some problems with connection pool or transactions

Restart the service and pressure test testB directly
iwtbafpdeMacBook-Pro:~ iwtbafp$ ab -n 100000 -c 100 -s 120 http://localhost:8080/testB

request is suspended for some unknown reasons in the concurrent scenario for a long time

Restart the service and pressure test testC directly
iwtbafpdeMacBook-Pro:~ iwtbafp$ ab -n 100000 -c 100 -s 120 http://localhost:8080/testC

I have tried this scene several times ,It seems to be all right (no transaction,no db).

Summarize

• testA(transaction&db)+testD ，we see tx hang in db till i restart service, there must be some problems with connection pool or transaction(Declarative & Programmatic).

• testC(no transaction&no db), it seems to be all right.

• testB(transaction&no db),request hang till time out.

• testD(no transaction& db ),request hang till time out (Occasional appearance).

I'm not sure what the problem is, but it does exist，hope to get helps,thanks.

[QuantumXiecao]

@ht-coder-wu
Please check as follows: When your ab hung there, you open another terminal and just use curl http://localhost:8080/test*(ABCD is all OK) for test. I think the curling result should be ok. One possible problem is that your sysctl config for port range is not enough, which would cause ab to hang there(You could cat /proc/sys/net/ipv4/ip_local_port_range to check and ajust it to 1024~65536 and do your pressure test again).

[ht-coder-wu]

@ht-coder-wu Please check as follows: When your ab hung there, you open another terminal and just use curl http://localhost:8080/test*(ABCD is all OK) for test. I think the curling result should be ok. One possible problem is that your sysctl config for port range is not enough, which would cause ab to hang there(You could cat /proc/sys/net/ipv4/ip_local_port_range to check and ajust it to 1024~65536 and do your pressure test again).

@QuantumXiecao
Thank you for your suggestion,I ignored the possibility of port range ,but when i adjusted it to 1024~65536 ,my ab hung there as before.
indeed, when my ab hung there,,i can curl for test,the result is ok ,i think the problem is occasional.

you can see the change:

and the ab results like this:

4 requests hung (100s) till time out as before ,then I didn't restart services and ab testD, problems existed as before.

[QuantumXiecao]

@ht-coder-wu Please try ab -n 50000(modify from 100000 to 50000) -c 100 -s 120 http://localhost:8080/testC and show us the results. Many thx!

[ht-coder-wu]

@ht-coder-wu Please try ab -n 50000(modify from 100000 to 50000) -c 100 -s 120 http://localhost:8080/testC and show us the results. Many thx!

@QuantumXiecao
request 50000's result:

request 100000's result:

I ignored the influence of port range factors at first,so I ab 100000 times testC again .

Compare these results，increase or decrease in quantity of request will affect the final result,connect will waste more time ,what's your opinion?

Then I pressure (50000 requests) testA and testD without restart service, things have no change.

vs

by the way,It's hard for us to control production enviroment visits,if the key of the problem are hardware resource,we want it to be more visible instead of hanging .....

[snicoll]

@ht-coder-wu I am trying to get to the bottom of this. Do you confirm the problem goes away if that custom filter is not registered anymore?

[spring-projects-issues]

If you would like us to look at this issue, please provide the requested information. If the information is not provided within the next 7 days this issue will be closed.

[ht-coder-wu]

@ht-coder-wu I am trying to get to the bottom of this. Do you confirm the problem goes away if that custom filter is not registered anymore?

If you are referring to ContextFilter, I confirm that the problem still exists,because initially, this issue occurred in the production environment.

[snicoll]

@ht-coder-wu thanks for following-up but that wasn't my question. I am asking if the problem does not occur if ContextFilter is not registered.