New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
KREST-10243 Add custom-request-logging to kafka-rest, and log error-codes for when various rate-limiters are triggered #1168
Conversation
1cedf14
to
71d5089
Compare
kafka-rest/src/main/java/io/confluent/kafkarest/CustomLogFields.java
Outdated
Show resolved
Hide resolved
kafka-rest/src/main/java/io/confluent/kafkarest/GlobalDosFilterListener.java
Outdated
Show resolved
Hide resolved
kafka-rest/src/main/java/io/confluent/kafkarest/CustomLogFields.java
Outdated
Show resolved
Hide resolved
kafka-rest/src/main/java/io/confluent/kafkarest/PerConnectionDosFilterListener.java
Outdated
Show resolved
Hide resolved
kafka-rest/src/main/java/io/confluent/kafkarest/RestCustomRequestLog.java
Outdated
Show resolved
Hide resolved
kafka-rest/src/main/java/io/confluent/kafkarest/RestCustomRequestLog.java
Outdated
Show resolved
Hide resolved
kafka-rest/src/main/java/io/confluent/kafkarest/ratelimit/FixedCostRateLimitRequestFilter.java
Outdated
Show resolved
Hide resolved
kafka-rest/src/main/java/io/confluent/kafkarest/resources/v3/ProduceRateLimiters.java
Outdated
Show resolved
Hide resolved
kafka-rest/src/main/java/io/confluent/kafkarest/RestCustomRequestLog.java
Outdated
Show resolved
Hide resolved
kafka-rest/src/main/java/io/confluent/kafkarest/RestCustomRequestLog.java
Outdated
Show resolved
Hide resolved
kafka-rest/src/main/java/io/confluent/kafkarest/RestCustomRequestLog.java
Outdated
Show resolved
Hide resolved
kafka-rest/src/main/java/io/confluent/kafkarest/RestCustomRequestLog.java
Outdated
Show resolved
Hide resolved
kafka-rest/src/main/java/io/confluent/kafkarest/RestCustomRequestLog.java
Outdated
Show resolved
Hide resolved
kafka-rest/src/main/java/io/confluent/kafkarest/RestCustomRequestLog.java
Outdated
Show resolved
Hide resolved
kafka-rest/src/main/java/io/confluent/kafkarest/RestCustomRequestLog.java
Outdated
Show resolved
Hide resolved
kafka-rest/src/main/java/io/confluent/kafkarest/KafkaRestApplication.java
Outdated
Show resolved
Hide resolved
kafka-rest/src/main/java/io/confluent/kafkarest/KafkaRestApplication.java
Outdated
Show resolved
Hide resolved
kafka-rest/src/main/java/io/confluent/kafkarest/KafkaRestApplication.java
Show resolved
Hide resolved
kafka-rest/src/main/java/io/confluent/kafkarest/KafkaRestApplication.java
Outdated
Show resolved
Hide resolved
kafka-rest/src/main/java/io/confluent/kafkarest/RestCustomRequestLog.java
Outdated
Show resolved
Hide resolved
@msn-tldr could you sync this branch with |
@trnguyencflt the build-failure is not do with Weirdly, the import is not in my local repo, only here. When i try to update the remote repo(even forcefully), github thinks everything is upto date, and so the unused import doesn't go away. Edit: Now fixed. Also had to rebase off master, since the version was bumped from 7.5.0 to 7.6.0, to pick latest build of rest-utils, and compile successfully. |
kafka-rest/src/main/java/io/confluent/kafkarest/KafkaRestApplication.java
Outdated
Show resolved
Hide resolved
kafka-rest/src/main/java/io/confluent/kafkarest/requestlog/CustomLog.java
Outdated
Show resolved
Hide resolved
kafka-rest/src/main/java/io/confluent/kafkarest/requestlog/CustomLog.java
Outdated
Show resolved
Hide resolved
kafka-rest/src/main/java/io/confluent/kafkarest/requestlog/CustomLog.java
Outdated
Show resolved
Hide resolved
kafka-rest/src/test/java/io/confluent/kafkarest/integration/CustomLogIntegrationTest.java
Show resolved
Hide resolved
kafka-rest/src/test/java/io/confluent/kafkarest/integration/CustomLogIntegrationTest.java
Outdated
Show resolved
Hide resolved
kafka-rest/src/test/java/io/confluent/kafkarest/integration/CustomLogIntegrationTest.java
Outdated
Show resolved
Hide resolved
kafka-rest/src/test/java/io/confluent/kafkarest/integration/CustomLogIntegrationTest.java
Outdated
Show resolved
Hide resolved
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM 👍
Following categories of changes are introduced in this PR.
CustomLog.java
, as a new Jetty request-logger, that also logs configuredrequest-attributes
.Example:
Existing
GET /kafka/v3/clusters/lkc-foobar/consumer-groups/foobar/lags HTTP/1.1" 429 408 "-" "Mozilla/5.0 (Windows NT; Windows NT 10.0; en-US) WindowsPowerShell/5.1.14393.5582
Vs NOTE the new log has the error-code
429007
at the end, which is error-code defined inRateLimitExceededException.java
class.GET /kafka/v3/clusters/lkc-foobar/consumer-groups/foobar/lags HTTP/1.1" 429 408 "-" "Mozilla/5.0 (Windows NT; Windows NT 10.0; en-US) WindowsPowerShell/5.1.14393.5582 429007
request-attribute
i.e.REST_ERROR_CODE
to be populated when a rate-limit is being triggered. All rate-limiters, when triggered have a corresponding "error-code" in request-log, this logic is tested inRestCustomRequestLogIntegrationTest.java
.RateLimitExceededException.java
has the error-codes that will be logged when rate-limit is triggered.GlobalDosFilterListener.java
&PerConnectionDosfilterListener.java
are DosFilter.Listeners that on global & per-ip Jetty rate-limiters populate request-attribute needed for logging.FixedCostRateLimitRequestFilter
has changes to log error-code when Jersey request rate-limits are triggered.ProduceRatelimiters
to log error-code when produce rate limits are triggered.use.custom.request.logging
. By default it's ON.