Define tcnative as optional again #12146
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This was referenced Mar 4, 2022
chrisvest
approved these changes
Mar 4, 2022
chrisvest
pushed a commit
that referenced
this pull request
Mar 4, 2022
srowen
pushed a commit
to apache/spark
that referenced
this pull request
Mar 19, 2022
### What changes were proposed in this pull request? This pr aims to upgrade netty to 4.1.75 and this pr add the explicit dependency on `netty-tcnative-classes` because the dependency on `netty-tcnative-classes` is changed to `Optional` in [netty](netty/netty#12146). ### Why are the changes needed? The release notes as follows: - https://netty.io/news/2022/03/10/4-1-75-Final.html There are 2 import changes in this version and both of these changes are to reduce memory-overhead: - [Reduce the default PooledByteBufAllocator chunk size from 16 MiB to 4 MiB](netty/netty#12108) - [Change default of io.netty.allocator.useCacheForAllThreads to false](netty/netty#12109) ### Does this PR introduce _any_ user-facing change? Yes, all changes are caused by the change of Netty. If the user wants to maintain the same behavior as the previous version, the user needs to make the following configuration: - add `-Dio.netty.allocator.useCacheForAllThreads=true` to enable `useCacheForAllThreads ` - add `-Dio.netty.allocator.maxOrder=11` to keep the chunk size of `PooledByteBufAllocator` to 16m ### How was this patch tested? Pass GA Closes #35835 from LuciferYang/upgrade-netty-4175. Authored-by: yangjie01 <yangjie01@baidu.com> Signed-off-by: Sean Owen <srowen@gmail.com>
raidyue
pushed a commit
to raidyue/netty
that referenced
this pull request
Jul 8, 2022
Motivation: netty#11745 introduced some change but missed to mark the tcnative classes jar as optional Modifications: Mark the dependency as optional again Result: Fixes netty#12101 and netty#12132
franz1981
pushed a commit
to franz1981/netty
that referenced
this pull request
Aug 22, 2022
Motivation: netty#11745 introduced some change but missed to mark the tcnative classes jar as optional Modifications: Mark the dependency as optional again Result: Fixes netty#12101 and netty#12132
This was referenced Nov 10, 2022
Closed
pan3793
added a commit
to apache/kyuubi
that referenced
this pull request
Nov 10, 2022
### _Why are the changes needed?_ Bump Netty to latest version. netty/netty#12146 explained that netty-tcnative-classes should be optional, otherwise will cause netty/netty#12101 ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [x] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request Closes #3796 from pan3793/netty. Closes #3796 2b29923 [Cheng Pan] license 447a8d0 [Cheng Pan] nit 62abe48 [Cheng Pan] nit 506f76b [Cheng Pan] Bump Netty from 4.1.73.Final to 4.1.84.Final Authored-by: Cheng Pan <chengpan@apache.org> Signed-off-by: Cheng Pan <chengpan@apache.org>
srowen
pushed a commit
to apache/spark
that referenced
this pull request
Nov 11, 2022
…ncyList ### What changes were proposed in this pull request? Remove `netty-tcnative-classes` from Spark dependencyList. ### Why are the changes needed? netty/netty#12146 explained that `netty-tcnative-classes` should be optional, otherwise will cause netty/netty#12101 It also not part of previous `netty-all`(fat jar) ### Does this PR introduce _any_ user-facing change? No. ### How was this patch tested? CI pass. Closes #38596 from pan3793/SPARK-41093. Authored-by: Cheng Pan <chengpan@apache.org> Signed-off-by: Sean Owen <srowen@gmail.com>
a0x8o
added a commit
to a0x8o/spark
that referenced
this pull request
Nov 11, 2022
…ncyList ### What changes were proposed in this pull request? Remove `netty-tcnative-classes` from Spark dependencyList. ### Why are the changes needed? netty/netty#12146 explained that `netty-tcnative-classes` should be optional, otherwise will cause netty/netty#12101 It also not part of previous `netty-all`(fat jar) ### Does this PR introduce _any_ user-facing change? No. ### How was this patch tested? CI pass. Closes #38596 from pan3793/SPARK-41093. Authored-by: Cheng Pan <chengpan@apache.org> Signed-off-by: Sean Owen <srowen@gmail.com>
pan3793
added a commit
to apache/kyuubi
that referenced
this pull request
Dec 2, 2022
### _Why are the changes needed?_ Bump Netty to latest version. netty/netty#12146 explained that netty-tcnative-classes should be optional, otherwise will cause netty/netty#12101 ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [x] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request Closes #3796 from pan3793/netty. Closes #3796 2b29923 [Cheng Pan] license 447a8d0 [Cheng Pan] nit 62abe48 [Cheng Pan] nit 506f76b [Cheng Pan] Bump Netty from 4.1.73.Final to 4.1.84.Final Authored-by: Cheng Pan <chengpan@apache.org> Signed-off-by: Cheng Pan <chengpan@apache.org>
SandishKumarHN
pushed a commit
to SandishKumarHN/spark
that referenced
this pull request
Dec 12, 2022
…ncyList ### What changes were proposed in this pull request? Remove `netty-tcnative-classes` from Spark dependencyList. ### Why are the changes needed? netty/netty#12146 explained that `netty-tcnative-classes` should be optional, otherwise will cause netty/netty#12101 It also not part of previous `netty-all`(fat jar) ### Does this PR introduce _any_ user-facing change? No. ### How was this patch tested? CI pass. Closes apache#38596 from pan3793/SPARK-41093. Authored-by: Cheng Pan <chengpan@apache.org> Signed-off-by: Sean Owen <srowen@gmail.com>
a0x8o
added a commit
to a0x8o/spark
that referenced
this pull request
Dec 30, 2022
…ncyList ### What changes were proposed in this pull request? Remove `netty-tcnative-classes` from Spark dependencyList. ### Why are the changes needed? netty/netty#12146 explained that `netty-tcnative-classes` should be optional, otherwise will cause netty/netty#12101 It also not part of previous `netty-all`(fat jar) ### Does this PR introduce _any_ user-facing change? No. ### How was this patch tested? CI pass. Closes #38596 from pan3793/SPARK-41093. Authored-by: Cheng Pan <chengpan@apache.org> Signed-off-by: Sean Owen <srowen@gmail.com>
a0x8o
added a commit
to a0x8o/spark
that referenced
this pull request
Dec 30, 2022
…ncyList ### What changes were proposed in this pull request? Remove `netty-tcnative-classes` from Spark dependencyList. ### Why are the changes needed? netty/netty#12146 explained that `netty-tcnative-classes` should be optional, otherwise will cause netty/netty#12101 It also not part of previous `netty-all`(fat jar) ### Does this PR introduce _any_ user-facing change? No. ### How was this patch tested? CI pass. Closes #38596 from pan3793/SPARK-41093. Authored-by: Cheng Pan <chengpan@apache.org> Signed-off-by: Sean Owen <srowen@gmail.com>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Motivation:
#11745 introduced some change but missed to mark the tcnative classes jar as optional
Modifications:
Mark the dependency as optional again
Result:
Fixes #12101 and #12132