[Broker][Proxy][Function worker] Fix backpressure handling in Jetty web server configuration #14353
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
lhotari
added
area/broker
area/proxy
doc-not-needed
lhotari
force-pushed
the
lh-improve-jetty-configuration-to-handle-backpressure
branch
from
ca6e3d4
to
109cc71
Compare
|
btw. It would be useful to investigate the use of https://www.eclipse.org/jetty/javadoc/jetty-9/org/eclipse/jetty/servlets/DoSFilter.html for basic DoS protection. |
This was referenced Feb 17, 2022
|
The pr had no activity for 30 days, mark with Stale label. |
lhotari
force-pushed
the
lh-improve-jetty-configuration-to-handle-backpressure
branch
2 times, most recently
from
5ff348d
to
c9e17a0
Compare
lhotari
requested review from
eolivelli,
merlimat,
addisonj,
dave2wave,
dlg99 and
Technoboy-
lhotari
force-pushed
the
lh-improve-jetty-configuration-to-handle-backpressure
branch
2 times, most recently
from
9bf6c2a
to
2cf16f7
Compare
lhotari
changed the title
lhotari
changed the title
nicoloboschi
approved these changes
Apr 11, 2022
eolivelli
approved these changes
Apr 11, 2022
|
@merlimat @rdhabalia @codelipenghui would you mind taking a look ? |
michaeljmarshall
approved these changes
Apr 11, 2022
There was a problem hiding this comment.
Great work @lhotari! I think we should also add this back pressure handling to the WorkerServer, which also runs a Jetty server.
- Fix maxConcurrentHttpRequests by using QoSFilter to limit concurrent requests
- previous solution didn't limit concurrent http requests
- Replace previous hardcoded connection limit of 10000 http connections with configurable setting
- use Jetty's built-in connection limit instead of PulsarServerConnector's custom solution
- Rate limiting should happen in the beginning of the filter chain
- Let Jetty tune selectors and acceptors based on number of cores
- JETTY_AVAILABLE_PROCESSORS=n environment variable can be used to override the number of cores reported by the OS
- This is useful when CPU limit isn't set on k8s and the number of cores is the number of total cores available on the k8s node
- use can also use -XX:ActiveProcessorCount=n to make Java's Runtime.getRuntime().availableProcessors() return n
- Make accept queue capacity configurable
- Make thread pool queue capacity bounded and make it configurable
- Make hardcoded websocket thread pool size of 20 configurable
lhotari
force-pushed
the
lh-improve-jetty-configuration-to-handle-backpressure
branch
from
2cf16f7
to
04bfb4a
Compare
@michaeljmarshall Thanks for pointing that out. I added a commit to cover Function Worker with similar changes. |
lhotari
changed the title
lhotari
changed the title
codelipenghui
approved these changes
Apr 12, 2022
lhotari
added a commit
to datastax/pulsar
that referenced
this pull request
Apr 14, 2022
…eb server configuration (apache#14353) * [Broker] Improve Jetty configuration to handle backpressure - Fix maxConcurrentHttpRequests by using QoSFilter to limit concurrent requests - previous solution didn't limit concurrent http requests - Replace previous hardcoded connection limit of 10000 http connections with configurable setting - use Jetty's built-in connection limit instead of PulsarServerConnector's custom solution - Rate limiting should happen in the beginning of the filter chain - Let Jetty tune selectors and acceptors based on number of cores - JETTY_AVAILABLE_PROCESSORS=n environment variable can be used to override the number of cores reported by the OS - This is useful when CPU limit isn't set on k8s and the number of cores is the number of total cores available on the k8s node - use can also use -XX:ActiveProcessorCount=n to make Java's Runtime.getRuntime().availableProcessors() return n - Make accept queue capacity configurable - Make thread pool queue capacity bounded and make it configurable * [Functions] Add http backpressure handling for Functions worker's http server (cherry picked from commit ec38211)
aparajita89
pushed a commit
to aparajita89/pulsar
that referenced
this pull request
Apr 18, 2022
…eb server configuration (apache#14353) * [Broker] Improve Jetty configuration to handle backpressure - Fix maxConcurrentHttpRequests by using QoSFilter to limit concurrent requests - previous solution didn't limit concurrent http requests - Replace previous hardcoded connection limit of 10000 http connections with configurable setting - use Jetty's built-in connection limit instead of PulsarServerConnector's custom solution - Rate limiting should happen in the beginning of the filter chain - Let Jetty tune selectors and acceptors based on number of cores - JETTY_AVAILABLE_PROCESSORS=n environment variable can be used to override the number of cores reported by the OS - This is useful when CPU limit isn't set on k8s and the number of cores is the number of total cores available on the k8s node - use can also use -XX:ActiveProcessorCount=n to make Java's Runtime.getRuntime().availableProcessors() return n - Make accept queue capacity configurable - Make thread pool queue capacity bounded and make it configurable * [Functions] Add http backpressure handling for Functions worker's http server
Nicklee007
pushed a commit
to Nicklee007/pulsar
that referenced
this pull request
Apr 20, 2022
…eb server configuration (apache#14353) * [Broker] Improve Jetty configuration to handle backpressure - Fix maxConcurrentHttpRequests by using QoSFilter to limit concurrent requests - previous solution didn't limit concurrent http requests - Replace previous hardcoded connection limit of 10000 http connections with configurable setting - use Jetty's built-in connection limit instead of PulsarServerConnector's custom solution - Rate limiting should happen in the beginning of the filter chain - Let Jetty tune selectors and acceptors based on number of cores - JETTY_AVAILABLE_PROCESSORS=n environment variable can be used to override the number of cores reported by the OS - This is useful when CPU limit isn't set on k8s and the number of cores is the number of total cores available on the k8s node - use can also use -XX:ActiveProcessorCount=n to make Java's Runtime.getRuntime().availableProcessors() return n - Make accept queue capacity configurable - Make thread pool queue capacity bounded and make it configurable * [Functions] Add http backpressure handling for Functions worker's http server
4 tasks
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Motivation
The current Jetty configuration doesn't properly handle backpressure. This is a problem when asynchronous request handling is used and there's high load. The requests will get handled and the asynchronous calls are made to the backend system and this might cause all requests to fail.
One of the main reasons to configure backpressure is to ensure that the system doesn't get overwhelmed and can handle successfully a certain amount of requests and reject the other requests. When more and more asynchronous request handling is happening, it is importance of having a backpressure handling that prevents too much work entering the system.
It should be possible to limit the number of concurrent requests that are handled at a time. There is a setting
maxConcurrentHttpRequests, but that is not correctly implemented and has no impact.This PR fixes the backpressure handling and improves the settings around backpressure control. The changes are made consistently for broker, proxy, function worker and websocket proxy, where Jetty is used in Pulsar.
Modifications
Fix
maxConcurrentHttpRequestsby using QoSFilter to limit concurrent requestsmaxConcurrentHttpRequestswas used to configure accept queue size. This didn't limit concurrent http requests.Replace previous hardcoded connection limit of 10000 http connections with configurable setting
maxHttpServerConnectionsRate limiting should happen in the beginning of the filter chain
Let Jetty tune selectors and acceptors based on number of cores
Make accept queue capacity configurable by
httpServerAcceptQueueSizesettingmake the queue for Jetty's thread pool's bounded and configured with
httpServerThreadPoolQueueSizesetting