-
Notifications
You must be signed in to change notification settings - Fork 3.6k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[improve][misc] Upgrade Netty to 4.1.86.Final and Netty Tcnative to 2.0.54.Final #18599
Conversation
df50ac8
to
1f85c02
Compare
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Good!
f0d6153
to
8084010
Compare
Codecov Report
@@ Coverage Diff @@
## master #18599 +/- ##
=============================================
- Coverage 46.17% 28.13% -18.05%
+ Complexity 10359 5391 -4968
=============================================
Files 703 623 -80
Lines 68845 59108 -9737
Branches 7382 6147 -1235
=============================================
- Hits 31788 16628 -15160
- Misses 33448 40131 +6683
+ Partials 3609 2349 -1260
Flags with carried forward coverage won't be shown. Click here to find out more.
|
There are odd test failures with newer Netty version. I was investigating the issue with |
db271d0
to
bc69b20
Compare
Simple instructions to reproduce the issue: git clone https://github.com/apache/pulsar
cd pulsar
# checkout PR 18599 in detached mode
git fetch origin pull/18599/head && git checkout FETCH_HEAD
mvn -Pcore-modules,-main -T 1C clean install -DskipTests -Dspotbugs.skip=true
# run test in a loop until it fails. The problem usually reproduces in less than 5 runs (usually 1st or 2nd round).
while mvn -DredirectTestOutputToFile=false -DtestRetryCount=0 test -pl pulsar-broker -Dtest=PatternMultiTopicsConsumerTest#testNotifications; do echo "Rerunning..."; done Now downgrade Netty version to 4.1.84.Final perl -i -p -e 's/4.1.85.Final/4.1.84.Final/g' $(git grep -l 4.1.85.Final)
# run test in a loop, it doesn't fail with 4.1.84.Final
while mvn -DredirectTestOutputToFile=false -DtestRetryCount=0 test -pl pulsar-broker -Dtest=PatternMultiTopicsConsumerTest#testNotifications; do echo "Rerunning..."; done This proves that changes in netty/netty#12888 are the source of the regression in Pulsar: # revert any changes (like downgrade to 4.1.84.Final)
git reset --hard HEAD
pushd
# patch netty-common-4.1.85.Final.jar file
cd ~/.m2/repository/io/netty/netty-common/4.1.85.Final
cp -p netty-common-4.1.85.Final.jar netty-common-4.1.85.Final.jar.orig
unzip ../4.1.84.Final/netty-common-4.1.84.Final.jar 'io/netty/util/HashedWheelTimer*'
zip netty-common-4.1.85.Final.jar io/netty/util/HashedWheelTimer*.class
rm -rf io/
# go back to the checkout directory
popd
# run test in a loop, it doesn't fail with 4.1.85.Final patched with HashedWheelTimer class from 4.1.84.Final
while mvn -DredirectTestOutputToFile=false -DtestRetryCount=0 test -pl pulsar-broker -Dtest=PatternMultiTopicsConsumerTest#testNotifications; do echo "Rerunning..."; done Remember to restore the patched jar file: cd ~/.m2/repository/io/netty/netty-common/4.1.85.Final
mv netty-common-4.1.85.Final.jar.orig netty-common-4.1.85.Final.jar |
Although the Netty upgrade makes the problem appear in Pulsar, it might be a bug in Pulsar that causes the problem. @andrasbeni Do you have any ideas why PatternMultiTopicsConsumerTest fails with Netty 4.1.85? (there's instructions to reproduce in #18599 (comment)) |
@andrasbeni you can ignore the previous comment. This has been confirmed to be caused by a bug in Netty. I'm expecting that the fix will be available in the next Netty 4.1.x release 4.1.86 . |
|
bc69b20
to
7839fdf
Compare
- enabling auto read will trigger a read
7839fdf
to
2223362
Compare
….0.54.Final (apache#18599) (cherry picked from commit 1cd20c5) (cherry picked from commit 3cace2f)
### Motivation - see release notes for 4.1.85.Final: https://netty.io/news/2022/11/09/4-1-85-Final.html - see release notes for 4.1.86.Final: https://netty.io/news/2022/12/12/4-1-86-Final.html It is unknown whether there are fixes that specifically target Netty features used in Bookkeeper. One of the motivation of this PR is to upgrade Netty in Bookkeeper when the Netty version gets upgraded in Pulsar. The PR for Netty 4.1.86.Final upgrade in Pulsar is apache/pulsar#18599 ### Changes Upgrade Netty version to 4.1.86.Final
### Motivation - see release notes for 4.1.85.Final: https://netty.io/news/2022/11/09/4-1-85-Final.html - see release notes for 4.1.86.Final: https://netty.io/news/2022/12/12/4-1-86-Final.html It is unknown whether there are fixes that specifically target Netty features used in Bookkeeper. One of the motivation of this PR is to upgrade Netty in Bookkeeper when the Netty version gets upgraded in Pulsar. The PR for Netty 4.1.86.Final upgrade in Pulsar is apache/pulsar#18599 ### Changes Upgrade Netty version to 4.1.86.Final
### Motivation - see release notes for 4.1.85.Final: https://netty.io/news/2022/11/09/4-1-85-Final.html - see release notes for 4.1.86.Final: https://netty.io/news/2022/12/12/4-1-86-Final.html It is unknown whether there are fixes that specifically target Netty features used in Bookkeeper. One of the motivation of this PR is to upgrade Netty in Bookkeeper when the Netty version gets upgraded in Pulsar. The PR for Netty 4.1.86.Final upgrade in Pulsar is apache/pulsar#18599 ### Changes Upgrade Netty version to 4.1.86.Final
Motivation
Modifications
ctx.read()
which is no more required when autoread is switched on.Documentation
doc
doc-required
doc-not-needed
doc-complete
Matching PR in forked repository
PR in forked repository: lhotari#103