Skip to content

Commit 53d59a0

Browse files
turboFeiturboFei
committed
[KYUUBI #1769] [BUILD] Migrate from log4j1 to log4j2
<!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> This patch proposes to migrate from log4j1 to log4j2 in Kyuubi. Refer the spark patch: apache/spark#34895 ### Does this PR introduce any user-facing change? Yes. Users may need to rewrite their log4j properties file for log4j2. As of version 2.4, log4j now supports configuration via properties files. Note that the property syntax is not the same as the syntax used in log4j 1, but during the migration I found the syntax is pretty close so the migration should be straightforward. ### _How was this patch tested?_ Passed all existing tests. Closes #1769 from turboFei/log4j2. Closes #1769 8613779 [Fei Wang] remove log4j dependencies from spark-sql-engine b2fe6db [Fei Wang] Use String to present default log level 762e2d0 [Fei Wang] only remove org.apache.logging.log4j:log4j-slf4j-impl 8a91208 [Fei Wang] remove dependencies from spark-sql-engine 7e3a498 [Fei Wang] address commments 051f49f [Fei Wang] address comments 85316a0 [Fei Wang] Keep compatible with log4j12 01d1a84 [Fei Wang] for log4j1 b9e17e1 [Fei Wang] refactor e24391e [Fei Wang] revert log count 3880300 [Fei Wang] add log4j2.properties.template 4f0b22f [Fei Wang] save 7ce8411 [Fei Wang] modify log level 1ea5ca5 [Fei Wang] add log4j to engine c4a86d4 [Fei Wang] use AbstractFilter 27b08b6 [Fei Wang] remove more 8cc15ae [Fei Wang] reformat c13ec29 [Fei Wang] save temporarily 33a38e2 [Fei Wang] exclude log4j12 from spark-sql 9129a64 [Fei Wang] refactor 5362b43 [Fei Wang] make it run at first 7f27f51 [Fei Wang] more 56f4f1f [Fei Wang] fix logging a74b6d3 [Fei Wang] start appender dea964a [Fei Wang] fix build erorr at first e20b750 [Fei Wang] address comments 2ec02b4 [Fei Wang] fix LogDivertAppender dded129 [Fei Wang] more c63e000 [Fei Wang] add log4j2.properties Authored-by: Fei Wang <fwang12@ebay.com> Signed-off-by: Fei Wang <fwang12@ebay.com>
1 parent 5ea0964 commit 53d59a0

File tree

31 files changed

+1230
-96
lines changed

31 files changed

+1230
-96
lines changed

‎conf/log4j2.properties.template

Lines changed: 51 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,51 @@
1+
#
2+
# Licensed to the Apache Software Foundation (ASF) under one or more
3+
# contributor license agreements. See the NOTICE file distributed with
4+
# this work for additional information regarding copyright ownership.
5+
# The ASF licenses this file to You under the Apache License, Version 2.0
6+
# (the "License"); you may not use this file except in compliance with
7+
# the License. You may obtain a copy of the License at
8+
#
9+
# http://www.apache.org/licenses/LICENSE-2.0
10+
#
11+
# Unless required by applicable law or agreed to in writing, software
12+
# distributed under the License is distributed on an "AS IS" BASIS,
13+
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
14+
# See the License for the specific language governing permissions and
15+
# limitations under the License.
16+
#
17+
18+
# Set everything to be logged to the file target/unit-tests.log
19+
rootLogger.level = info
20+
rootLogger.appenderRef.stdout.ref = STDOUT
21+
22+
# Console Appender
23+
appender.console.type = Console
24+
appender.console.name = STDOUT
25+
appender.console.target = SYSTEM_OUT
26+
appender.console.layout.type = PatternLayout
27+
appender.console.layout.pattern = %d{HH:mm:ss.SSS} %p %c: %m%n
28+
29+
appender.console.filter.1.type = Filters
30+
31+
appender.console.filter.1.a.type = ThresholdFilter
32+
appender.console.filter.1.a.level = info
33+
34+
# SPARK-34128: Suppress undesirable TTransportException warnings, due to THRIFT-4805
35+
appender.console.filter.1.b.type = RegexFilter
36+
appender.console.filter.1.b.regex = .*Thrift error occurred during processing of message.*
37+
appender.console.filter.1.b.onMatch = deny
38+
appender.console.filter.1.b.onMismatch = neutral
39+
40+
# Set the default kyuubi-ctl log level to WARN. When running the kyuubi-ctl, the
41+
# log level for this class is used to overwrite the root logger's log level.
42+
logger.ctl.name = org.apache.kyuubi.ctl.ServiceControlCli
43+
logger.ctl.level = error
44+
45+
# Analysis MySQLFrontend protocol traffic
46+
# logger.mysql.name = org.apache.kyuubi.server.mysql.codec
47+
# logger.mysql.level = trace
48+
49+
# Kyuubi BeeLine
50+
logger.beeline.name = org.apache.hive.beeline.KyuubiBeeLine
51+
logger.beeline.level = error

‎dev/dependencyList

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -72,6 +72,10 @@ jetty-util/9.4.41.v20210516//jetty-util-9.4.41.v20210516.jar
7272
jline/0.9.94//jline-0.9.94.jar
7373
libfb303/0.9.3//libfb303-0.9.3.jar
7474
libthrift/0.9.3//libthrift-0.9.3.jar
75+
log4j-1.2-api/2.17.1//log4j-1.2-api-2.17.1.jar
76+
log4j-api/2.17.1//log4j-api-2.17.1.jar
77+
log4j-core/2.17.1//log4j-core-2.17.1.jar
78+
log4j-slf4j-impl/2.17.1//log4j-slf4j-impl-2.17.1.jar
7579
log4j/1.2.17//log4j-1.2.17.jar
7680
metrics-core/4.1.1//metrics-core-4.1.1.jar
7781
metrics-jmx/4.1.1//metrics-jmx-4.1.1.jar

‎dev/kyuubi-extension-spark-3-1/src/test/resources/log4j2.properties

Lines changed: 57 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,57 @@
1+
#
2+
# Licensed to the Apache Software Foundation (ASF) under one or more
3+
# contributor license agreements. See the NOTICE file distributed with
4+
# this work for additional information regarding copyright ownership.
5+
# The ASF licenses this file to You under the Apache License, Version 2.0
6+
# (the "License"); you may not use this file except in compliance with
7+
# the License. You may obtain a copy of the License at
8+
#
9+
# http://www.apache.org/licenses/LICENSE-2.0
10+
#
11+
# Unless required by applicable law or agreed to in writing, software
12+
# distributed under the License is distributed on an "AS IS" BASIS,
13+
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
14+
# See the License for the specific language governing permissions and
15+
# limitations under the License.
16+
#
17+
18+
# Set everything to be logged to the file target/unit-tests.log
19+
rootLogger.level = info
20+
rootLogger.appenderRef.stdout.ref = STDOUT
21+
rootLogger.appenderRef.file.ref = File
22+
23+
# Console Appender
24+
appender.console.type = Console
25+
appender.console.name = STDOUT
26+
appender.console.target = SYSTEM_OUT
27+
appender.console.layout.type = PatternLayout
28+
appender.console.layout.pattern = %d{HH:mm:ss.SSS} %p %c: %m%n
29+
30+
appender.console.filter.1.type = Filters
31+
32+
appender.console.filter.1.a.type = ThresholdFilter
33+
appender.console.filter.1.a.level = debug
34+
35+
# SPARK-34128: Suppress undesirable TTransportException warnings, due to THRIFT-4805
36+
appender.console.filter.1.b.type = RegexFilter
37+
appender.console.filter.1.b.regex = .*Thrift error occurred during processing of message.*
38+
appender.console.filter.1.b.onMatch = deny
39+
appender.console.filter.1.b.onMismatch = neutral
40+
41+
# File Appender
42+
appender.file.type = File
43+
appender.file.name = File
44+
appender.file.fileName = target/unit-tests.log
45+
appender.file.layout.type = PatternLayout
46+
appender.file.layout.pattern = %d{HH:mm:ss.SSS} %t %p %c{1}: %m%n
47+
48+
appender.file.filter.1.type = Filters
49+
50+
appender.file.filter.1.a.type = RegexFilter
51+
appender.file.filter.1.a.regx = .*Thrift error occurred during processing of message.*
52+
appender.file.filter.1.a.onMatch = deny
53+
appender.file.filter.1.a.onMismatch = neutral
54+
55+
# Set the logger level of File Appender to DEBUG
56+
appender.file.filter.1.b.type = ThresholdFilter
57+
appender.file.filter.1.b.level = debug

‎dev/kyuubi-extension-spark-3-2/src/test/resources/log4j2.properties

Lines changed: 57 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,57 @@
1+
#
2+
# Licensed to the Apache Software Foundation (ASF) under one or more
3+
# contributor license agreements. See the NOTICE file distributed with
4+
# this work for additional information regarding copyright ownership.
5+
# The ASF licenses this file to You under the Apache License, Version 2.0
6+
# (the "License"); you may not use this file except in compliance with
7+
# the License. You may obtain a copy of the License at
8+
#
9+
# http://www.apache.org/licenses/LICENSE-2.0
10+
#
11+
# Unless required by applicable law or agreed to in writing, software
12+
# distributed under the License is distributed on an "AS IS" BASIS,
13+
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
14+
# See the License for the specific language governing permissions and
15+
# limitations under the License.
16+
#
17+
18+
# Set everything to be logged to the file target/unit-tests.log
19+
rootLogger.level = info
20+
rootLogger.appenderRef.stdout.ref = STDOUT
21+
rootLogger.appenderRef.file.ref = File
22+
23+
# Console Appender
24+
appender.console.type = Console
25+
appender.console.name = STDOUT
26+
appender.console.target = SYSTEM_OUT
27+
appender.console.layout.type = PatternLayout
28+
appender.console.layout.pattern = %d{HH:mm:ss.SSS} %p %c: %m%n
29+
30+
appender.console.filter.1.type = Filters
31+
32+
appender.console.filter.1.a.type = ThresholdFilter
33+
appender.console.filter.1.a.level = debug
34+
35+
# SPARK-34128: Suppress undesirable TTransportException warnings, due to THRIFT-4805
36+
appender.console.filter.1.b.type = RegexFilter
37+
appender.console.filter.1.b.regex = .*Thrift error occurred during processing of message.*
38+
appender.console.filter.1.b.onMatch = deny
39+
appender.console.filter.1.b.onMismatch = neutral
40+
41+
# File Appender
42+
appender.file.type = File
43+
appender.file.name = File
44+
appender.file.fileName = target/unit-tests.log
45+
appender.file.layout.type = PatternLayout
46+
appender.file.layout.pattern = %d{HH:mm:ss.SSS} %t %p %c{1}: %m%n
47+
48+
appender.file.filter.1.type = Filters
49+
50+
appender.file.filter.1.a.type = RegexFilter
51+
appender.file.filter.1.a.regx = .*Thrift error occurred during processing of message.*
52+
appender.file.filter.1.a.onMatch = deny
53+
appender.file.filter.1.a.onMismatch = neutral
54+
55+
# Set the logger level of File Appender to DEBUG
56+
appender.file.filter.1.b.type = ThresholdFilter
57+
appender.file.filter.1.b.level = debug

‎dev/kyuubi-extension-spark-common/src/test/resources/log4j2.properties

Lines changed: 57 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,57 @@
1+
#
2+
# Licensed to the Apache Software Foundation (ASF) under one or more
3+
# contributor license agreements. See the NOTICE file distributed with
4+
# this work for additional information regarding copyright ownership.
5+
# The ASF licenses this file to You under the Apache License, Version 2.0
6+
# (the "License"); you may not use this file except in compliance with
7+
# the License. You may obtain a copy of the License at
8+
#
9+
# http://www.apache.org/licenses/LICENSE-2.0
10+
#
11+
# Unless required by applicable law or agreed to in writing, software
12+
# distributed under the License is distributed on an "AS IS" BASIS,
13+
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
14+
# See the License for the specific language governing permissions and
15+
# limitations under the License.
16+
#
17+
18+
# Set everything to be logged to the file target/unit-tests.log
19+
rootLogger.level = info
20+
rootLogger.appenderRef.stdout.ref = STDOUT
21+
rootLogger.appenderRef.file.ref = File
22+
23+
# Console Appender
24+
appender.console.type = Console
25+
appender.console.name = STDOUT
26+
appender.console.target = SYSTEM_OUT
27+
appender.console.layout.type = PatternLayout
28+
appender.console.layout.pattern = %d{HH:mm:ss.SSS} %p %c: %m%n
29+
30+
appender.console.filter.1.type = Filters
31+
32+
appender.console.filter.1.a.type = ThresholdFilter
33+
appender.console.filter.1.a.level = debug
34+
35+
# SPARK-34128: Suppress undesirable TTransportException warnings, due to THRIFT-4805
36+
appender.console.filter.1.b.type = RegexFilter
37+
appender.console.filter.1.b.regex = .*Thrift error occurred during processing of message.*
38+
appender.console.filter.1.b.onMatch = deny
39+
appender.console.filter.1.b.onMismatch = neutral
40+
41+
# File Appender
42+
appender.file.type = File
43+
appender.file.name = File
44+
appender.file.fileName = target/unit-tests.log
45+
appender.file.layout.type = PatternLayout
46+
appender.file.layout.pattern = %d{HH:mm:ss.SSS} %t %p %c{1}: %m%n
47+
48+
appender.file.filter.1.type = Filters
49+
50+
appender.file.filter.1.a.type = RegexFilter
51+
appender.file.filter.1.a.regx = .*Thrift error occurred during processing of message.*
52+
appender.file.filter.1.a.onMatch = deny
53+
appender.file.filter.1.a.onMismatch = neutral
54+
55+
# Set the logger level of File Appender to DEBUG
56+
appender.file.filter.1.b.type = ThresholdFilter
57+
appender.file.filter.1.b.level = debug

‎docs/deployment/settings.md

Lines changed: 28 additions & 10 deletions
Original file line numberDiff line numberDiff line change
@@ -370,7 +370,7 @@ Please refer to the Spark official online documentation for [SET Command](http:/
370370

371371
## Logging
372372

373-
Kyuubi uses [log4j](https://logging.apache.org/log4j/2.x/) for logging. You can configure it using `$KYUUBI_HOME/conf/log4j.properties`.
373+
Kyuubi uses [log4j](https://logging.apache.org/log4j/2.x/) for logging. You can configure it using `$KYUUBI_HOME/conf/log4j2.properties`.
374374
```bash
375375
#
376376
# Licensed to the Apache Software Foundation (ASF) under one or more
@@ -389,22 +389,40 @@ Kyuubi uses [log4j](https://logging.apache.org/log4j/2.x/) for logging. You can
389389
# limitations under the License.
390390
#
391391

392-
# Set everything to be logged to the console
393-
log4j.rootCategory=INFO, console
394-
log4j.appender.console=org.apache.log4j.ConsoleAppender
395-
log4j.appender.console.target=System.err
396-
log4j.appender.console.layout=org.apache.log4j.PatternLayout
397-
log4j.appender.console.layout.ConversionPattern=%d{yyyy-MM-dd HH:mm:ss.SSS} %p %c{2}: %m%n
392+
# Set everything to be logged to the file target/unit-tests.log
393+
rootLogger.level = info
394+
rootLogger.appenderRef.stdout.ref = STDOUT
395+
396+
# Console Appender
397+
appender.console.type = Console
398+
appender.console.name = STDOUT
399+
appender.console.target = SYSTEM_OUT
400+
appender.console.layout.type = PatternLayout
401+
appender.console.layout.pattern = %d{HH:mm:ss.SSS} %p %c: %m%n
402+
403+
appender.console.filter.1.type = Filters
404+
405+
appender.console.filter.1.a.type = ThresholdFilter
406+
appender.console.filter.1.a.level = info
407+
408+
# SPARK-34128: Suppress undesirable TTransportException warnings, due to THRIFT-4805
409+
appender.console.filter.1.b.type = RegexFilter
410+
appender.console.filter.1.b.regex = .*Thrift error occurred during processing of message.*
411+
appender.console.filter.1.b.onMatch = deny
412+
appender.console.filter.1.b.onMismatch = neutral
398413

399414
# Set the default kyuubi-ctl log level to WARN. When running the kyuubi-ctl, the
400415
# log level for this class is used to overwrite the root logger's log level.
401-
log4j.logger.org.apache.kyuubi.ctl.ServiceControlCli=ERROR
416+
logger.ctl.name = org.apache.kyuubi.ctl.ServiceControlCli
417+
logger.ctl.level = error
402418

403419
# Analysis MySQLFrontend protocol traffic
404-
# log4j.logger.org.apache.kyuubi.server.mysql.codec=TRACE
420+
# logger.mysql.name = org.apache.kyuubi.server.mysql.codec
421+
# logger.mysql.level = trace
405422

406423
# Kyuubi BeeLine
407-
log4j.logger.org.apache.hive.beeline.KyuubiBeeLine=ERROR
424+
logger.beeline.name = org.apache.hive.beeline.KyuubiBeeLine
425+
logger.beeline.level = error
408426
```
409427

410428
## Other Configurations

‎externals/kyuubi-flink-sql-engine/src/test/resources/log4j2.properties

Lines changed: 57 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,57 @@
1+
#
2+
# Licensed to the Apache Software Foundation (ASF) under one or more
3+
# contributor license agreements. See the NOTICE file distributed with
4+
# this work for additional information regarding copyright ownership.
5+
# The ASF licenses this file to You under the Apache License, Version 2.0
6+
# (the "License"); you may not use this file except in compliance with
7+
# the License. You may obtain a copy of the License at
8+
#
9+
# http://www.apache.org/licenses/LICENSE-2.0
10+
#
11+
# Unless required by applicable law or agreed to in writing, software
12+
# distributed under the License is distributed on an "AS IS" BASIS,
13+
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
14+
# See the License for the specific language governing permissions and
15+
# limitations under the License.
16+
#
17+
18+
# Set everything to be logged to the file target/unit-tests.log
19+
rootLogger.level = debug
20+
rootLogger.appenderRef.stdout.ref = STDOUT
21+
rootLogger.appenderRef.file.ref = File
22+
23+
# Console Appender
24+
appender.console.type = Console
25+
appender.console.name = STDOUT
26+
appender.console.target = SYSTEM_OUT
27+
appender.console.layout.type = PatternLayout
28+
appender.console.layout.pattern = %d{HH:mm:ss.SSS} %p %c: %m%n
29+
30+
appender.console.filter.1.type = Filters
31+
32+
appender.console.filter.1.a.type = ThresholdFilter
33+
appender.console.filter.1.a.level = debug
34+
35+
# SPARK-34128: Suppress undesirable TTransportException warnings, due to THRIFT-4805
36+
appender.console.filter.1.b.type = RegexFilter
37+
appender.console.filter.1.b.regex = .*Thrift error occurred during processing of message.*
38+
appender.console.filter.1.b.onMatch = deny
39+
appender.console.filter.1.b.onMismatch = neutral
40+
41+
# File Appender
42+
appender.file.type = File
43+
appender.file.name = File
44+
appender.file.fileName = target/unit-tests.log
45+
appender.file.layout.type = PatternLayout
46+
appender.file.layout.pattern = %d{HH:mm:ss.SSS} %t %p %c{1}: %m%n
47+
48+
appender.file.filter.1.type = Filters
49+
50+
appender.file.filter.1.a.type = RegexFilter
51+
appender.file.filter.1.a.regx = .*Thrift error occurred during processing of message.*
52+
appender.file.filter.1.a.onMatch = deny
53+
appender.file.filter.1.a.onMismatch = neutral
54+
55+
# Set the logger level of File Appender to DEBUG
56+
appender.file.filter.1.b.type = ThresholdFilter
57+
appender.file.filter.1.b.level = debug

0 commit comments

Comments
 (0)