-
Notifications
You must be signed in to change notification settings - Fork 343
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Spring Boot 3.1.6 health endpoint latency is very high when other query is running. #1898
Comments
please provide a repository link where issue can be reproduced |
It is on private project, I will try to create public project and add there. |
In heavy IO mode this happens, hard to upload whole project. I don't have 24hour live mock server. If you guys have mock server and open api spec, I will generate client, I need at least two mock servers(micro services). This problem happens when all the reactive threads are busy and the time acts like blocking. My management port is different 9999, does it share same thread pool with port 8080? Something like this should solve the problem:
But I am not sure how to configure this globally singleton way. I used k6 to generate the traffic:
health.ts
|
Screen.Recording.2023-12-05.at.11.29.57.PM.movFor security reason I can't show whole screen, Left side: calling /health endpoint single user(health in cloud usually have single call). Right side: actual query with 500 users. query throughput:
health endpoint throughput:
health check timeout could be increased but not ideally more than 5 sec. |
Spring Boot Webflux defaults to reactor Netty threadpool which in turns defaults to You have few options
class MySpringGraphQLServer(
requestParser: SpringGraphQLRequestParser,
contextFactory: SpringGraphQLContextFactory,
requestHandler: GraphQLRequestHandler
) : SpringGraphQLServer(requestParser, contextFactory, requestHandler) {
override suspend fun execute(request: ServerRequest): GraphQLServerResponse? = withContext(Dispatchers.Default) {
super.execute(request)
}
} *we may end up pushing this change to the repo as well |
@dariuszkuc Something like this doesn't work.
This worked for me, thanks lot.
And config:
|
I'm guessing you should be increasing the workers instead of IO selector thread, per Netty docs it should be That being said.... since Also note that the switch there in the |
Agreed: spring-cloud/spring-cloud-gateway#1844 I see here. It works with those. |
Will create a PR to execute operations out of the reactor http epoll |
Library Version
7.0.2
Describe the bug
Health endpoint is very slow when other continuous queries are running to fetch the data.
With Other Query
✓ checks.........................: 100.00% ✓ 3 ✗ 0
data_received..................: 2.4 kB 94 B/s
data_sent......................: 344 B 14 B/s
http_req_blocked...............: avg=564.33µs min=3µs med=4µs max=1.68ms p(90)=1.34ms p(95)=1.51ms
http_req_connecting............: avg=55.33µs min=0s med=0s max=166µs p(90)=132.8µs p(95)=149.4µs
http_req_duration..............: avg=7.02s min=6s med=6.06s max=8.99s p(90)=8.41s p(95)=8.7s
{ expected_response:true }...: avg=7.02s min=6s med=6.06s max=8.99s p(90)=8.41s p(95)=8.7s //slow
http_req_failed................: 0.00% ✓ 0 ✗ 3
http_req_receiving.............: avg=84.66µs min=45µs med=48µs max=161µs p(90)=138.4µs p(95)=149.69µs
http_req_sending...............: avg=35.66µs min=19µs med=24µs max=64µs p(90)=56µs p(95)=60µs
http_req_tls_handshaking.......: avg=0s min=0s med=0s max=0s p(90)=0s p(95)=0s
http_req_waiting...............: avg=7.02s min=6s med=6.06s max=8.99s p(90)=8.41s p(95)=8.7s
http_reqs......................: 3 0.118047/s
iteration_duration.............: avg=7.02s min=6s med=6.06s max=8.99s p(90)=8.41s p(95)=8.7s
iterations.....................: 3 0.118047/s
vus............................: 1 min=1 max=1
vus_max........................: 1 min=1 max=1
query:
✓ account query is 200
✓ checks.........................: 100.00% ✓ 754 ✗ 0
data_received..................: 143 kB 4.6 kB/s
data_sent......................: 507 kB 16 kB/s
http_req_blocked...............: avg=94.44µs min=1µs med=3µs max=2.82ms p(90)=352.4µs p(95)=451.35µs
http_req_connecting............: avg=47.11µs min=0s med=0s max=557µs p(90)=298µs p(95)=377.4µs
http_req_duration..............: avg=3.78s min=534.59ms med=3.43s max=13.88s p(90)=6.61s p(95)=7.79s
{ expected_response:true }...: avg=3.78s min=534.59ms med=3.43s max=13.88s p(90)=6.61s p(95)=7.79s
http_req_failed................: 0.00% ✓ 0 ✗ 754
http_req_receiving.............: avg=54.57µs min=19µs med=50µs max=372µs p(90)=77.7µs p(95)=94.4µs
http_req_sending...............: avg=33.74µs min=11µs med=24µs max=407µs p(90)=50µs p(95)=66.35µs
http_req_tls_handshaking.......: avg=0s min=0s med=0s max=0s p(90)=0s p(95)=0s
http_req_waiting...............: avg=3.78s min=534.15ms med=3.43s max=13.88s p(90)=6.61s p(95)=7.79s
http_reqs......................: 754 24.427578/s
iteration_duration.............: avg=3.81s min=538.34ms med=3.44s max=13.93s p(90)=6.61s p(95)=7.83s
iterations.....................: 754 24.427578/s
vus............................: 100 min=100 max=100
vus_max........................: 100 min=100 max=100
Without Query
✓ health query is 200
✓ checks.........................: 100.00% ✓ 317 ✗ 0
data_received..................: 253 kB 68 kB/s
data_sent......................: 27 kB 7.4 kB/s
http_req_blocked...............: avg=8.18µs min=1µs med=3µs max=1.74ms p(90)=4µs p(95)=4µs
http_req_connecting............: avg=577ns min=0s med=0s max=183µs p(90)=0s p(95)=0s
http_req_duration..............: avg=1.88ms min=637µs med=1.03ms max=252.7ms p(90)=1.37ms p(95)=1.5ms
{ expected_response:true }...: avg=1.88ms min=637µs med=1.03ms max=252.7ms p(90)=1.37ms p(95)=1.5ms
http_req_failed................: 0.00% ✓ 0 ✗ 317
http_req_receiving.............: avg=34.28µs min=18µs med=32µs max=147µs p(90)=46µs p(95)=56.2µs
http_req_sending...............: avg=15.51µs min=6µs med=14µs max=446µs p(90)=18µs p(95)=19.19µs
http_req_tls_handshaking.......: avg=0s min=0s med=0s max=0s p(90)=0s p(95)=0s
http_req_waiting...............: avg=1.83ms min=606µs med=988µs max=252.19ms p(90)=1.32ms p(95)=1.45ms
http_reqs......................: 317 85.200396/s
iteration_duration.............: avg=11.71ms min=1.38ms med=5.28ms max=255.35ms p(90)=24.4ms p(95)=40.3ms
iterations.....................: 317 85.200396/s
vus............................: 1 min=1 max=1
vus_max........................: 1 min=1 max=1
To Reproduce
Steps to reproduce the behavior. Please provide:
I have bunch of dependencies, not sure if it is side effects of any but all are up to date, here are majors.
Expected behavior
Actually
/health
endpoint shouldn't be blocked by normal traffic in reactive framework.The text was updated successfully, but these errors were encountered: