Skip to content

Commit 1a6e7cc

Browse files
committed
[KYUUBI #2103] [1.5] Revert "[KYUUBI #1948] Upgrade thrift version to 0.16.0"
### _Why are the changes needed?_ This PR reverts "[KYUUBI #1948] Upgrade thrift version to 0.16.0" in branch-1.5 and some subsequent commits due to issues caused by breaking changes in Thrift, e.g. #2101, #2093 We need more time to evaluate it and add more test cases to ensure stability before the next release. ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [x] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request Closes #2103 from pan3793/revert. Closes #2103 Closes #1948 575dfe8 [Cheng Pan] properties order 4c94481 [Cheng Pan] fix deps bfbdeb8 [Cheng Pan] Add log4j deps for kyuubi-hive-jdbc test bad4e90 [Cheng Pan] surefire f2ae8b9 [Cheng Pan] Revert "[KYUUBI #1948] Upgrade thrift version to 0.16.0" 4501f74 [Cheng Pan] Revert "[KYUUBI #1948][FOLLOWUP] Remove TFramedTransport in extension-spark-3-1 and extension-spark-3-2" cc04df4 [Cheng Pan] Revert "[KYUUBI #1996] Clean up log4j2 conf and enhance surefire plugin conf" 52a20e1 [Cheng Pan] Revert "[KYUUBI #1948][FOLLOWUP] Relocate fb303 classes" Authored-by: Cheng Pan <chengpan@apache.org> Signed-off-by: Cheng Pan <chengpan@apache.org>
1 parent a1d5d58 commit 1a6e7cc

File tree

41 files changed

+472
-1142
lines changed

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

41 files changed

+472
-1142
lines changed

conf/log4j2.properties.template

Lines changed: 11 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -25,8 +25,17 @@ appender.console.name = STDOUT
2525
appender.console.target = SYSTEM_OUT
2626
appender.console.layout.type = PatternLayout
2727
appender.console.layout.pattern = %d{HH:mm:ss.SSS} %p %c: %m%n
28-
appender.console.filter.a.type = ThresholdFilter
29-
appender.console.filter.a.level = info
28+
29+
appender.console.filter.1.type = Filters
30+
31+
appender.console.filter.1.a.type = ThresholdFilter
32+
appender.console.filter.1.a.level = info
33+
34+
# SPARK-34128: Suppress undesirable TTransportException warnings, due to THRIFT-4805
35+
appender.console.filter.1.b.type = RegexFilter
36+
appender.console.filter.1.b.regex = .*Thrift error occurred during processing of message.*
37+
appender.console.filter.1.b.onMatch = deny
38+
appender.console.filter.1.b.onMismatch = neutral
3039

3140
# Set the default kyuubi-ctl log level to WARN. When running the kyuubi-ctl, the
3241
# log level for this class is used to overwrite the root logger's log level.

dev/dependencyList

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -68,7 +68,7 @@ jetty-util-ajax/9.4.41.v20210516//jetty-util-ajax-9.4.41.v20210516.jar
6868
jetty-util/9.4.41.v20210516//jetty-util-9.4.41.v20210516.jar
6969
jline/0.9.94//jline-0.9.94.jar
7070
libfb303/0.9.3//libfb303-0.9.3.jar
71-
libthrift/0.16.0//libthrift-0.16.0.jar
71+
libthrift/0.9.3//libthrift-0.9.3.jar
7272
log4j-1.2-api/2.17.2//log4j-1.2-api-2.17.2.jar
7373
log4j-api/2.17.2//log4j-api-2.17.2.jar
7474
log4j-core/2.17.2//log4j-core-2.17.2.jar

dev/kyuubi-extension-spark-3-1/src/test/resources/log4j2-test.properties

Lines changed: 21 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -26,15 +26,32 @@ appender.console.name = STDOUT
2626
appender.console.target = SYSTEM_OUT
2727
appender.console.layout.type = PatternLayout
2828
appender.console.layout.pattern = %d{HH:mm:ss.SSS} %p %c: %m%n
29-
appender.console.filter.a.type = ThresholdFilter
30-
appender.console.filter.a.level = fatal
29+
30+
appender.console.filter.1.type = Filters
31+
32+
appender.console.filter.1.a.type = ThresholdFilter
33+
appender.console.filter.1.a.level = fatal
34+
35+
# SPARK-34128: Suppress undesirable TTransportException warnings, due to THRIFT-4805
36+
appender.console.filter.1.b.type = RegexFilter
37+
appender.console.filter.1.b.regex = .*Thrift error occurred during processing of message.*
38+
appender.console.filter.1.b.onMatch = deny
39+
appender.console.filter.1.b.onMismatch = neutral
3140

3241
# File Appender
3342
appender.file.type = File
3443
appender.file.name = File
3544
appender.file.fileName = target/unit-tests.log
3645
appender.file.layout.type = PatternLayout
3746
appender.file.layout.pattern = %d{HH:mm:ss.SSS} %t %p %c{1}: %m%n
47+
48+
appender.file.filter.1.type = Filters
49+
50+
appender.file.filter.1.a.type = RegexFilter
51+
appender.file.filter.1.a.regex = .*Thrift error occurred during processing of message.*
52+
appender.file.filter.1.a.onMatch = deny
53+
appender.file.filter.1.a.onMismatch = neutral
54+
3855
# Set the logger level of File Appender to DEBUG
39-
appender.file.filter.a.type = ThresholdFilter
40-
appender.file.filter.a.level = debug
56+
appender.file.filter.1.b.type = ThresholdFilter
57+
appender.file.filter.1.b.level = debug

dev/kyuubi-extension-spark-3-2/src/test/resources/log4j2-test.properties

Lines changed: 21 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -26,15 +26,32 @@ appender.console.name = STDOUT
2626
appender.console.target = SYSTEM_OUT
2727
appender.console.layout.type = PatternLayout
2828
appender.console.layout.pattern = %d{HH:mm:ss.SSS} %p %c: %m%n
29-
appender.console.filter.a.type = ThresholdFilter
30-
appender.console.filter.a.level = fatal
29+
30+
appender.console.filter.1.type = Filters
31+
32+
appender.console.filter.1.a.type = ThresholdFilter
33+
appender.console.filter.1.a.level = fatal
34+
35+
# SPARK-34128: Suppress undesirable TTransportException warnings, due to THRIFT-4805
36+
appender.console.filter.1.b.type = RegexFilter
37+
appender.console.filter.1.b.regex = .*Thrift error occurred during processing of message.*
38+
appender.console.filter.1.b.onMatch = deny
39+
appender.console.filter.1.b.onMismatch = neutral
3140

3241
# File Appender
3342
appender.file.type = File
3443
appender.file.name = File
3544
appender.file.fileName = target/unit-tests.log
3645
appender.file.layout.type = PatternLayout
3746
appender.file.layout.pattern = %d{HH:mm:ss.SSS} %t %p %c{1}: %m%n
47+
48+
appender.file.filter.1.type = Filters
49+
50+
appender.file.filter.1.a.type = RegexFilter
51+
appender.file.filter.1.a.regex = .*Thrift error occurred during processing of message.*
52+
appender.file.filter.1.a.onMatch = deny
53+
appender.file.filter.1.a.onMismatch = neutral
54+
3855
# Set the logger level of File Appender to DEBUG
39-
appender.file.filter.a.type = ThresholdFilter
40-
appender.file.filter.a.level = debug
56+
appender.file.filter.1.b.type = ThresholdFilter
57+
appender.file.filter.1.b.level = debug

dev/kyuubi-extension-spark-common/src/test/java/org/apache/thrift/transport/TFramedTransport.java

Lines changed: 0 additions & 185 deletions
This file was deleted.

dev/kyuubi-extension-spark-common/src/test/resources/log4j2-test.properties

Lines changed: 21 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -26,15 +26,32 @@ appender.console.name = STDOUT
2626
appender.console.target = SYSTEM_OUT
2727
appender.console.layout.type = PatternLayout
2828
appender.console.layout.pattern = %d{HH:mm:ss.SSS} %p %c: %m%n
29-
appender.console.filter.a.type = ThresholdFilter
30-
appender.console.filter.a.level = fatal
29+
30+
appender.console.filter.1.type = Filters
31+
32+
appender.console.filter.1.a.type = ThresholdFilter
33+
appender.console.filter.1.a.level = fatal
34+
35+
# SPARK-34128: Suppress undesirable TTransportException warnings, due to THRIFT-4805
36+
appender.console.filter.1.b.type = RegexFilter
37+
appender.console.filter.1.b.regex = .*Thrift error occurred during processing of message.*
38+
appender.console.filter.1.b.onMatch = deny
39+
appender.console.filter.1.b.onMismatch = neutral
3140

3241
# File Appender
3342
appender.file.type = File
3443
appender.file.name = File
3544
appender.file.fileName = target/unit-tests.log
3645
appender.file.layout.type = PatternLayout
3746
appender.file.layout.pattern = %d{HH:mm:ss.SSS} %t %p %c{1}: %m%n
47+
48+
appender.file.filter.1.type = Filters
49+
50+
appender.file.filter.1.a.type = RegexFilter
51+
appender.file.filter.1.a.regex = .*Thrift error occurred during processing of message.*
52+
appender.file.filter.1.a.onMatch = deny
53+
appender.file.filter.1.a.onMismatch = neutral
54+
3855
# Set the logger level of File Appender to DEBUG
39-
appender.file.filter.a.type = ThresholdFilter
40-
appender.file.filter.a.level = debug
56+
appender.file.filter.1.b.type = ThresholdFilter
57+
appender.file.filter.1.b.level = debug

docs/deployment/settings.md

Lines changed: 15 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -211,9 +211,11 @@ Key | Default | Meaning | Type | Since
211211

212212
Key | Default | Meaning | Type | Since
213213
--- | --- | --- | --- | ---
214+
<code>kyuubi.frontend.backoff.slot.length</code>|<div style='width: 65pt;word-wrap: break-word;white-space: normal'>PT0.1S</div>|<div style='width: 170pt;word-wrap: break-word;white-space: normal'>(deprecated) Time to back off during login to the thrift frontend service.</div>|<div style='width: 30pt'>duration</div>|<div style='width: 20pt'>1.0.0</div>
214215
<code>kyuubi.frontend.bind.host</code>|<div style='width: 65pt;word-wrap: break-word;white-space: normal'>&lt;undefined&gt;</div>|<div style='width: 170pt;word-wrap: break-word;white-space: normal'>(deprecated) Hostname or IP of the machine on which to run the thrift frontend service via binary protocol.</div>|<div style='width: 30pt'>string</div>|<div style='width: 20pt'>1.0.0</div>
215216
<code>kyuubi.frontend.bind.port</code>|<div style='width: 65pt;word-wrap: break-word;white-space: normal'>10009</div>|<div style='width: 170pt;word-wrap: break-word;white-space: normal'>(deprecated) Port of the machine on which to run the thrift frontend service via binary protocol.</div>|<div style='width: 30pt'>int</div>|<div style='width: 20pt'>1.0.0</div>
216217
<code>kyuubi.frontend.connection.url.use.hostname</code>|<div style='width: 65pt;word-wrap: break-word;white-space: normal'>false</div>|<div style='width: 170pt;word-wrap: break-word;white-space: normal'>When true, frontend services prefer hostname, otherwise, ip address</div>|<div style='width: 30pt'>boolean</div>|<div style='width: 20pt'>1.5.0</div>
218+
<code>kyuubi.frontend.login.timeout</code>|<div style='width: 65pt;word-wrap: break-word;white-space: normal'>PT20S</div>|<div style='width: 170pt;word-wrap: break-word;white-space: normal'>(deprecated) Timeout for Thrift clients during login to the thrift frontend service.</div>|<div style='width: 30pt'>duration</div>|<div style='width: 20pt'>1.0.0</div>
217219
<code>kyuubi.frontend.max.message.size</code>|<div style='width: 65pt;word-wrap: break-word;white-space: normal'>104857600</div>|<div style='width: 170pt;word-wrap: break-word;white-space: normal'>(deprecated) Maximum message size in bytes a Kyuubi server will accept.</div>|<div style='width: 30pt'>int</div>|<div style='width: 20pt'>1.0.0</div>
218220
<code>kyuubi.frontend.max.worker.threads</code>|<div style='width: 65pt;word-wrap: break-word;white-space: normal'>999</div>|<div style='width: 170pt;word-wrap: break-word;white-space: normal'>(deprecated) Maximum number of threads in the of frontend worker thread pool for the thrift frontend service</div>|<div style='width: 30pt'>int</div>|<div style='width: 20pt'>1.0.0</div>
219221
<code>kyuubi.frontend.min.worker.threads</code>|<div style='width: 65pt;word-wrap: break-word;white-space: normal'>9</div>|<div style='width: 170pt;word-wrap: break-word;white-space: normal'>(deprecated) Minimum number of threads in the of frontend worker thread pool for the thrift frontend service</div>|<div style='width: 30pt'>int</div>|<div style='width: 20pt'>1.0.0</div>
@@ -226,8 +228,10 @@ Key | Default | Meaning | Type | Since
226228
<code>kyuubi.frontend.protocols</code>|<div style='width: 65pt;word-wrap: break-word;white-space: normal'>THRIFT_BINARY</div>|<div style='width: 170pt;word-wrap: break-word;white-space: normal'>A comma separated list for all frontend protocols <ul> <li>THRIFT_BINARY - HiveServer2 compatible thrift binary protocol.</li> <li>REST - Kyuubi defined REST API(experimental).</li> <li>MYSQL - MySQL compatible text protocol(experimental).</li> </ul></div>|<div style='width: 30pt'>seq</div>|<div style='width: 20pt'>1.4.0</div>
227229
<code>kyuubi.frontend.rest.bind.host</code>|<div style='width: 65pt;word-wrap: break-word;white-space: normal'>&lt;undefined&gt;</div>|<div style='width: 170pt;word-wrap: break-word;white-space: normal'>Hostname or IP of the machine on which to run the REST frontend service.</div>|<div style='width: 30pt'>string</div>|<div style='width: 20pt'>1.4.0</div>
228230
<code>kyuubi.frontend.rest.bind.port</code>|<div style='width: 65pt;word-wrap: break-word;white-space: normal'>10099</div>|<div style='width: 170pt;word-wrap: break-word;white-space: normal'>Port of the machine on which to run the REST frontend service.</div>|<div style='width: 30pt'>int</div>|<div style='width: 20pt'>1.4.0</div>
231+
<code>kyuubi.frontend.thrift.backoff.slot.length</code>|<div style='width: 65pt;word-wrap: break-word;white-space: normal'>PT0.1S</div>|<div style='width: 170pt;word-wrap: break-word;white-space: normal'>Time to back off during login to the thrift frontend service.</div>|<div style='width: 30pt'>duration</div>|<div style='width: 20pt'>1.4.0</div>
229232
<code>kyuubi.frontend.thrift.binary.bind.host</code>|<div style='width: 65pt;word-wrap: break-word;white-space: normal'>&lt;undefined&gt;</div>|<div style='width: 170pt;word-wrap: break-word;white-space: normal'>Hostname or IP of the machine on which to run the thrift frontend service via binary protocol.</div>|<div style='width: 30pt'>string</div>|<div style='width: 20pt'>1.4.0</div>
230233
<code>kyuubi.frontend.thrift.binary.bind.port</code>|<div style='width: 65pt;word-wrap: break-word;white-space: normal'>10009</div>|<div style='width: 170pt;word-wrap: break-word;white-space: normal'>Port of the machine on which to run the thrift frontend service via binary protocol.</div>|<div style='width: 30pt'>int</div>|<div style='width: 20pt'>1.4.0</div>
234+
<code>kyuubi.frontend.thrift.login.timeout</code>|<div style='width: 65pt;word-wrap: break-word;white-space: normal'>PT20S</div>|<div style='width: 170pt;word-wrap: break-word;white-space: normal'>Timeout for Thrift clients during login to the thrift frontend service.</div>|<div style='width: 30pt'>duration</div>|<div style='width: 20pt'>1.4.0</div>
231235
<code>kyuubi.frontend.thrift.max.message.size</code>|<div style='width: 65pt;word-wrap: break-word;white-space: normal'>104857600</div>|<div style='width: 170pt;word-wrap: break-word;white-space: normal'>Maximum message size in bytes a Kyuubi server will accept.</div>|<div style='width: 30pt'>int</div>|<div style='width: 20pt'>1.4.0</div>
232236
<code>kyuubi.frontend.thrift.max.worker.threads</code>|<div style='width: 65pt;word-wrap: break-word;white-space: normal'>999</div>|<div style='width: 170pt;word-wrap: break-word;white-space: normal'>Maximum number of threads in the of frontend worker thread pool for the thrift frontend service</div>|<div style='width: 30pt'>int</div>|<div style='width: 20pt'>1.4.0</div>
233237
<code>kyuubi.frontend.thrift.min.worker.threads</code>|<div style='width: 65pt;word-wrap: break-word;white-space: normal'>9</div>|<div style='width: 170pt;word-wrap: break-word;white-space: normal'>Minimum number of threads in the of frontend worker thread pool for the thrift frontend service</div>|<div style='width: 30pt'>int</div>|<div style='width: 20pt'>1.4.0</div>
@@ -407,8 +411,17 @@ appender.console.name = STDOUT
407411
appender.console.target = SYSTEM_OUT
408412
appender.console.layout.type = PatternLayout
409413
appender.console.layout.pattern = %d{HH:mm:ss.SSS} %p %c: %m%n
410-
appender.console.filter.a.type = ThresholdFilter
411-
appender.console.filter.a.level = info
414+
415+
appender.console.filter.1.type = Filters
416+
417+
appender.console.filter.1.a.type = ThresholdFilter
418+
appender.console.filter.1.a.level = info
419+
420+
# SPARK-34128: Suppress undesirable TTransportException warnings, due to THRIFT-4805
421+
appender.console.filter.1.b.type = RegexFilter
422+
appender.console.filter.1.b.regex = .*Thrift error occurred during processing of message.*
423+
appender.console.filter.1.b.onMatch = deny
424+
appender.console.filter.1.b.onMismatch = neutral
412425

413426
# Set the default kyuubi-ctl log level to WARN. When running the kyuubi-ctl, the
414427
# log level for this class is used to overwrite the root logger's log level.

externals/kyuubi-flink-sql-engine/pom.xml

Lines changed: 1 addition & 9 deletions
Original file line numberDiff line numberDiff line change
@@ -171,8 +171,7 @@
171171
<include>org.apache.curator:curator-framework</include>
172172
<include>org.apache.curator:curator-recipes</include>\
173173
<include>org.apache.hive:hive-service-rpc</include>
174-
<include>org.apache.thrift:libfb303</include>
175-
<include>org.apache.thrift:libthrift</include>
174+
<include>org.apache.thrift:*</include>
176175
<include>org.apache.zookeeper:*</include>
177176
</includes>
178177
</artifactSet>
@@ -220,13 +219,6 @@
220219
<include>org.apache.thrift.**</include>
221220
</includes>
222221
</relocation>
223-
<relocation>
224-
<pattern>com.facebook.fb303</pattern>
225-
<shadedPattern>${kyuubi.shade.packageName}.com.facebook.fb303</shadedPattern>
226-
<includes>
227-
<include>com.facebook.fb303.**</include>
228-
</includes>
229-
</relocation>
230222
<relocation>
231223
<pattern>org.apache.zookeeper</pattern>
232224
<shadedPattern>${kyuubi.shade.packageName}.org.apache.zookeeper</shadedPattern>

0 commit comments

Comments
 (0)