Skip to content

Commit 30dc84b

Browse files
pan3793pan3793
committed
[KYUUBI #2134] Respect Spark bundled log4j in extension modules
### _Why are the changes needed?_ As Spark 3.1/3.2 still use log4j 1.2, we should respect Spark bundled log4j in extension modules. ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [x] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request Closes #2134 from pan3793/log. Closes #2134 7429503 [Cheng Pan] Revert "reset mvn log level to info in CI" 694a159 [Cheng Pan] Revert "mvn --batch-mode" ba72d84 [Cheng Pan] log4j.properties 39f12c9 [Cheng Pan] mvn --batch-mode 4792226 [Cheng Pan] reset mvn log level to info in CI b337f71 [Cheng Pan] Respect Spark bundled log4j in extension modules Authored-by: Cheng Pan <chengpan@apache.org> Signed-off-by: Kent Yao <yao@apache.org>
1 parent e232a83 commit 30dc84b

File tree

7 files changed

+135
-173
lines changed

7 files changed

+135
-173
lines changed

‎dev/kyuubi-extension-spark-3-1/src/test/resources/log4j.properties

Lines changed: 44 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,44 @@
1+
#
2+
# Licensed to the Apache Software Foundation (ASF) under one or more
3+
# contributor license agreements. See the NOTICE file distributed with
4+
# this work for additional information regarding copyright ownership.
5+
# The ASF licenses this file to You under the Apache License, Version 2.0
6+
# (the "License"); you may not use this file except in compliance with
7+
# the License. You may obtain a copy of the License at
8+
#
9+
# http://www.apache.org/licenses/LICENSE-2.0
10+
#
11+
# Unless required by applicable law or agreed to in writing, software
12+
# distributed under the License is distributed on an "AS IS" BASIS,
13+
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
14+
# See the License for the specific language governing permissions and
15+
# limitations under the License.
16+
#
17+
18+
# Set everything to be logged to the file target/unit-tests.log
19+
log4j.rootLogger=INFO, CA, FA
20+
21+
# Console Appender
22+
log4j.appender.CA=org.apache.log4j.ConsoleAppender
23+
log4j.appender.CA.layout=org.apache.log4j.PatternLayout
24+
log4j.appender.CA.layout.ConversionPattern=%d{HH:mm:ss.SSS} %p %c: %m%n
25+
log4j.appender.CA.Threshold = FATAL
26+
27+
# File Appender
28+
log4j.appender.FA=org.apache.log4j.FileAppender
29+
log4j.appender.FA.append=false
30+
log4j.appender.FA.file=target/unit-tests.log
31+
log4j.appender.FA.layout=org.apache.log4j.PatternLayout
32+
log4j.appender.FA.layout.ConversionPattern=%d{HH:mm:ss.SSS} %t %p %c{2}: %m%n
33+
34+
# Set the logger level of File Appender to WARN
35+
log4j.appender.FA.Threshold = DEBUG
36+
37+
# SPARK-34128: Suppress undesirable TTransportException warnings involved in THRIFT-4805
38+
log4j.appender.CA.filter.1=org.apache.log4j.varia.StringMatchFilter
39+
log4j.appender.CA.filter.1.StringToMatch=Thrift error occurred during processing of message
40+
log4j.appender.CA.filter.1.AcceptOnMatch=false
41+
42+
log4j.appender.FA.filter.1=org.apache.log4j.varia.StringMatchFilter
43+
log4j.appender.FA.filter.1.StringToMatch=Thrift error occurred during processing of message
44+
log4j.appender.FA.filter.1.AcceptOnMatch=false

‎dev/kyuubi-extension-spark-3-1/src/test/resources/log4j2-test.properties

Lines changed: 0 additions & 57 deletions
This file was deleted.

‎dev/kyuubi-extension-spark-3-2/src/test/resources/log4j.properties

Lines changed: 44 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,44 @@
1+
#
2+
# Licensed to the Apache Software Foundation (ASF) under one or more
3+
# contributor license agreements. See the NOTICE file distributed with
4+
# this work for additional information regarding copyright ownership.
5+
# The ASF licenses this file to You under the Apache License, Version 2.0
6+
# (the "License"); you may not use this file except in compliance with
7+
# the License. You may obtain a copy of the License at
8+
#
9+
# http://www.apache.org/licenses/LICENSE-2.0
10+
#
11+
# Unless required by applicable law or agreed to in writing, software
12+
# distributed under the License is distributed on an "AS IS" BASIS,
13+
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
14+
# See the License for the specific language governing permissions and
15+
# limitations under the License.
16+
#
17+
18+
# Set everything to be logged to the file target/unit-tests.log
19+
log4j.rootLogger=INFO, CA, FA
20+
21+
# Console Appender
22+
log4j.appender.CA=org.apache.log4j.ConsoleAppender
23+
log4j.appender.CA.layout=org.apache.log4j.PatternLayout
24+
log4j.appender.CA.layout.ConversionPattern=%d{HH:mm:ss.SSS} %p %c: %m%n
25+
log4j.appender.CA.Threshold = FATAL
26+
27+
# File Appender
28+
log4j.appender.FA=org.apache.log4j.FileAppender
29+
log4j.appender.FA.append=false
30+
log4j.appender.FA.file=target/unit-tests.log
31+
log4j.appender.FA.layout=org.apache.log4j.PatternLayout
32+
log4j.appender.FA.layout.ConversionPattern=%d{HH:mm:ss.SSS} %t %p %c{2}: %m%n
33+
34+
# Set the logger level of File Appender to WARN
35+
log4j.appender.FA.Threshold = DEBUG
36+
37+
# SPARK-34128: Suppress undesirable TTransportException warnings involved in THRIFT-4805
38+
log4j.appender.CA.filter.1=org.apache.log4j.varia.StringMatchFilter
39+
log4j.appender.CA.filter.1.StringToMatch=Thrift error occurred during processing of message
40+
log4j.appender.CA.filter.1.AcceptOnMatch=false
41+
42+
log4j.appender.FA.filter.1=org.apache.log4j.varia.StringMatchFilter
43+
log4j.appender.FA.filter.1.StringToMatch=Thrift error occurred during processing of message
44+
log4j.appender.FA.filter.1.AcceptOnMatch=false

‎dev/kyuubi-extension-spark-3-2/src/test/resources/log4j2-test.properties

Lines changed: 0 additions & 57 deletions
This file was deleted.

‎dev/kyuubi-extension-spark-common/src/test/resources/log4j.properties

Lines changed: 44 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,44 @@
1+
#
2+
# Licensed to the Apache Software Foundation (ASF) under one or more
3+
# contributor license agreements. See the NOTICE file distributed with
4+
# this work for additional information regarding copyright ownership.
5+
# The ASF licenses this file to You under the Apache License, Version 2.0
6+
# (the "License"); you may not use this file except in compliance with
7+
# the License. You may obtain a copy of the License at
8+
#
9+
# http://www.apache.org/licenses/LICENSE-2.0
10+
#
11+
# Unless required by applicable law or agreed to in writing, software
12+
# distributed under the License is distributed on an "AS IS" BASIS,
13+
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
14+
# See the License for the specific language governing permissions and
15+
# limitations under the License.
16+
#
17+
18+
# Set everything to be logged to the file target/unit-tests.log
19+
log4j.rootLogger=INFO, CA, FA
20+
21+
# Console Appender
22+
log4j.appender.CA=org.apache.log4j.ConsoleAppender
23+
log4j.appender.CA.layout=org.apache.log4j.PatternLayout
24+
log4j.appender.CA.layout.ConversionPattern=%d{HH:mm:ss.SSS} %p %c: %m%n
25+
log4j.appender.CA.Threshold = FATAL
26+
27+
# File Appender
28+
log4j.appender.FA=org.apache.log4j.FileAppender
29+
log4j.appender.FA.append=false
30+
log4j.appender.FA.file=target/unit-tests.log
31+
log4j.appender.FA.layout=org.apache.log4j.PatternLayout
32+
log4j.appender.FA.layout.ConversionPattern=%d{HH:mm:ss.SSS} %t %p %c{2}: %m%n
33+
34+
# Set the logger level of File Appender to WARN
35+
log4j.appender.FA.Threshold = DEBUG
36+
37+
# SPARK-34128: Suppress undesirable TTransportException warnings involved in THRIFT-4805
38+
log4j.appender.CA.filter.1=org.apache.log4j.varia.StringMatchFilter
39+
log4j.appender.CA.filter.1.StringToMatch=Thrift error occurred during processing of message
40+
log4j.appender.CA.filter.1.AcceptOnMatch=false
41+
42+
log4j.appender.FA.filter.1=org.apache.log4j.varia.StringMatchFilter
43+
log4j.appender.FA.filter.1.StringToMatch=Thrift error occurred during processing of message
44+
log4j.appender.FA.filter.1.AcceptOnMatch=false

‎dev/kyuubi-extension-spark-common/src/test/resources/log4j2-test.properties

Lines changed: 0 additions & 57 deletions
This file was deleted.

‎pom.xml

Lines changed: 3 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -455,8 +455,8 @@
455455
<artifactId>hadoop-client</artifactId>
456456
</exclusion>
457457
<!--
458-
The module is only used in Kyuubi Spark Extensions, so we don't care about which
459-
version of Log4j it depends on.
458+
The module is only used in Kyuubi Spark Extensions, we should respect
459+
the Spark bundled log4j.
460460
-->
461461
</exclusions>
462462
</dependency>
@@ -1503,6 +1503,7 @@
15031503
</environmentVariables>
15041504
<systemProperties>
15051505
<log4j.ignoreTCL>true</log4j.ignoreTCL>
1506+
<log4j.configuration>file:src/test/resources/log4j.properties</log4j.configuration>
15061507
<log4j2.configurationFile>file:src/test/resources/log4j2-test.properties</log4j2.configurationFile>
15071508
<java.io.tmpdir>${project.build.directory}/tmp</java.io.tmpdir>
15081509
<spark.driver.memory>1g</spark.driver.memory>

0 commit comments

Comments
 (0)