[feat][broker] Support dynamic update log level at runtime #14200

ethqunzhong · 2022-02-10T03:41:13Z

Motivation

As default, Broker log level set by log4j2.yaml with pulsar.log.level=info, if developers want to update log level to debug, need to Modify log42j.yaml#pulsar.log.level=debug and then restart Broker service.

This PR support developers dynamic update Broker log level at runtime,It is very useful for online trouble shooting without having to stop Broker service and start it again.

Modifications

add admin cli (bin/pulsar-admin brokers update-logger-level --classname ${CLASSNAME} --level ${LEVEL})
add rest endpoints (/admin/v2/brokers/log4j/{classname}/{level})

Verifying this change

Make sure that the change passes the CI checks.

This change added tests and can be verified as follows:org.apache.pulsar.admin.cli.PulsarAdminToolTest#brokers
works as expected on product environment.

Does this pull request potentially affect one of the following parts:

If yes was chosen, please highlight the changes

Dependencies (does it add or upgrade a dependency): (no)
The public API: (no)
The schema: (no)
The default values of configurations: (no)
The wire protocol: (no)
The rest endpoints: (yes)
The admin cli options: (yes)
Anything that affects deployment: (no)

Documentation

doc-not-needed

pulsar-client-tools/src/main/java/org/apache/pulsar/admin/cli/CmdBrokers.java

pulsar-broker/src/main/java/org/apache/pulsar/broker/admin/impl/BrokersBase.java

Jason918 · 2022-02-11T07:27:28Z

conf/log4j2.yaml

      additivity: true
      AppenderRef:
        - ref: "${sys:pulsar.log.appender}"
-          level: "${sys:pulsar.log.level}"


This should not be changed as we should keep default behavior.

as default pulsar.log.level=info
i have do a test in product env as follow:

remove this item seems do not change default log4j behavior.

as answer in [stackoverflow:programmatically-change-log-level-in-log4j2 :

i have try all mentioned way to change logger level by programmatically way, it not work unless remove level: "${sys:pulsar.log.level}"

Have you ever had similar problems?

User may already have set pulsar.log.level=warn in their deployment. Will this PR change the settings?

i have set pulsar.log.level=warn for test. but not what i expected. described as this issue #14298
update: 🤲
i have a test, result as follow:
as log4j2.yaml using way,

pulsar.log.level go into effect only set by ENV property PULSAR_LOG_LEVEL which set in pulsar_env.sh while service starting.
while service running, change pulsar.log.level in log4j2.yaml is not work, pulsar.log.level is unmodifiable now.

User may already have set pulsar.log.level=warn in their deployment. Will this PR change the settings?

i have a test

if User set pulsar.log.level=warn in their deployment while starting service.

this PR will not change the settings , all loggers's level is warn, meanwhile , the PR featrue also works.

~~level: "${sys:pulsar.log.level}"~~

this line config is redundant and will disturb the programmatically way.

this PR will not change the settings

I am a little confused that if you removed this, how would user's previous setting (set by ENV property PULSAR_LOG_LEVEL in pulsar_env.sh) take effect?

i have a test in standalone mode after removed this line.

pulsar.log.level set by ENV property PULSAR_LOG_LEVEL in pulsar_env.sh to warn

2. then start pulsar service in standalone mode. the service log level is warn, worked as desired.
3. then run cmd `bin/pulsar-admin brokers update-logger-level --classname org.apache.zookeeper.server.FinalRequestProcessor --level debug` to validation feature in this PR. workd as desired.

so remove this line would not invalidate user's previous setting (set by ENV property PULSAR_LOG_LEVEL in pulsar_env.sh

Can you explain from the code, why pulsar.log.level this system env could still take effect after we delete from log4j2.yaml? Did we pass this variable to log4j somewhere else?

Can you explain from the code, why pulsar.log.level this system env could still take effect after we delete from log4j2.yaml? Did we pass this variable to log4j somewhere else?

+1, from the source code here https://github.com/apache/pulsar/blob/master/bin/pulsar#L299, I think we can't remove this line here.

eolivelli

Very interesting work.
I left some suggestions, PTAL

pulsar-broker/src/main/java/org/apache/pulsar/broker/admin/impl/BrokersBase.java

pulsar-client-tools/src/main/java/org/apache/pulsar/admin/cli/CmdBrokers.java

wolfstudy · 2022-02-21T09:31:20Z

cc @eolivelli @Jason918 PTAL again thanks

ethqunzhong · 2022-02-22T07:18:59Z

/pulsarbot run-failure-checks

eolivelli

IIRC log4j supports automatic reloading of configuration files.
Isn't it enough for you?

pulsar-broker/src/main/java/org/apache/pulsar/broker/admin/impl/BrokersBase.java

ethqunzhong · 2022-02-22T10:31:33Z

IIRC log4j supports automatic reloading of configuration files. Isn't it enough for you?

yes, Log4j has the ability to automatically configure itself during initialization and reload config while config files changed.
but it is more convenient for developers update logger level by programmatically way.

codelipenghui · 2022-03-21T07:47:25Z

conf/log4j2.yaml

      additivity: true
      AppenderRef:
        - ref: "${sys:pulsar.log.appender}"
-          level: "${sys:pulsar.log.level}"


Can you explain from the code, why pulsar.log.level this system env could still take effect after we delete from log4j2.yaml? Did we pass this variable to log4j somewhere else?

+1, from the source code here https://github.com/apache/pulsar/blob/master/bin/pulsar#L299, I think we can't remove this line here.

codelipenghui · 2022-03-21T07:50:10Z

pulsar-broker/src/main/java/org/apache/pulsar/broker/admin/impl/BrokersBase.java

+      @ApiResponse(code = 403, message = "You don't have admin permission to update logger level."),
+      @ApiResponse(code = 500, message = "Internal server error")})
+    public void updateLoggerLevelDynamically(@Suspended final AsyncResponse asyncResponse,
+                                             @PathParam("classname") String classname,


It should be the logger name here, not the class name. Usually, we will use the class name as the logger name, but not 100%

pulsar-broker/src/main/java/org/apache/pulsar/broker/admin/impl/BrokersBase.java

codelipenghui · 2022-03-21T07:54:32Z

pulsar-broker/src/main/java/org/apache/pulsar/broker/admin/impl/BrokersBase.java

+      @ApiResponse(code = 204, message = "class logger level updated successfully"),
+      @ApiResponse(code = 403, message = "You don't have admin permission to update logger level."),
+      @ApiResponse(code = 500, message = "Internal server error")})
+    public void updateLoggerLevelDynamically(@Suspended final AsyncResponse asyncResponse,


Is it able to provide a get method to get the current applied log levels?

codelipenghui · 2022-03-21T07:55:07Z

pulsar-client-admin-api/src/main/java/org/apache/pulsar/client/admin/Brokers.java

+     *
+     * @throws PulsarAdminException if update logger level failed.
+     */
+    void updateLoggerLevel(String classname, String level) throws PulsarAdminException;


Suggested change

void updateLoggerLevel(String classname, String level) throws PulsarAdminException;

void updateLoggerLevel(String loggerName, String level) throws PulsarAdminException;

codelipenghui · 2022-03-21T07:55:17Z

pulsar-client-admin-api/src/main/java/org/apache/pulsar/client/admin/Brokers.java

+     * Reset logger level dynamically in runtime asynchronously.
+     * @return CompletableFuture
+     */
+    CompletableFuture<Void> updateLoggerLevelAsync(String classname, String level);


Suggested change

CompletableFuture<Void> updateLoggerLevelAsync(String classname, String level);

CompletableFuture<Void> updateLoggerLevelAsync(String loggerName, String level);

codelipenghui · 2022-03-21T07:55:38Z

pulsar-client-tools/src/main/java/org/apache/pulsar/admin/cli/CmdBrokers.java

+    private class UpdateLoggerLevelCmd extends CliCommand {
+        @Parameter(names = {"-c", "--classname"}, description =
+          "The except class name,if set \"ROOT\" will take effect to rootLogger", required = true)
+        private String classname;


Suggested change

private String classname;

private String loggerName;

codelipenghui · 2022-03-21T07:55:50Z

pulsar-client-tools/src/main/java/org/apache/pulsar/admin/cli/CmdBrokers.java


+    @Parameters(commandDescription = "Dynamic update logger level in runtime by classname.")
+    private class UpdateLoggerLevelCmd extends CliCommand {
+        @Parameter(names = {"-c", "--classname"}, description =


Suggested change

@Parameter(names = {"-c", "--classname"}, description =

@Parameter(names = {"-l", "--logger-name"}, description =

github-actions · 2022-04-21T02:12:07Z

The pr had no activity for 30 days, mark with Stale label.

github-actions · 2022-05-28T02:11:18Z

The pr had no activity for 30 days, mark with Stale label.

qunzhong added 2 commits February 9, 2022 20:14

Broker: support dynamic update log level at runtime

6d69b68

fix conflict

fbd4d97

github-actions bot assigned ethqunzhong Feb 10, 2022

github-actions bot added the doc-not-needed Your PR changes do not impact docs label Feb 10, 2022

Jason918 reviewed Feb 10, 2022

View reviewed changes

mattisonchao reviewed Feb 11, 2022

View reviewed changes

ethqunzhong force-pushed the dynamic_adjust_broker_log_level branch from 7d28dd8 to 041f15e Compare February 11, 2022 06:13

ethqunzhong requested a review from Jason918 February 11, 2022 06:28

Jason918 reviewed Feb 11, 2022

View reviewed changes

eolivelli requested changes Feb 15, 2022

View reviewed changes

ethqunzhong force-pushed the dynamic_adjust_broker_log_level branch from 041f15e to ce21dc7 Compare February 18, 2022 09:10

ethqunzhong requested a review from eolivelli February 18, 2022 09:13

ethqunzhong force-pushed the dynamic_adjust_broker_log_level branch from ce21dc7 to 0172610 Compare February 18, 2022 10:25

eolivelli requested changes Feb 22, 2022

View reviewed changes

pulsar-broker/src/main/java/org/apache/pulsar/broker/admin/impl/BrokersBase.java Outdated Show resolved Hide resolved

add async method & fix checkstyle & update discription

c1f5ea4

ethqunzhong force-pushed the dynamic_adjust_broker_log_level branch from 0172610 to c1f5ea4 Compare February 22, 2022 10:44

ethqunzhong requested a review from eolivelli February 22, 2022 10:45

codelipenghui added this to the 2.11.0 milestone Mar 15, 2022

ethqunzhong mentioned this pull request Mar 20, 2022

change pulsar.log.level in log4j2.yaml not work? #14298

Open

codelipenghui reviewed Mar 21, 2022

View reviewed changes

github-actions bot added the lifecycle/stale label Apr 21, 2022

codelipenghui modified the milestones: 2.11.0, 2.12.0 Jul 26, 2022

tisonkun added ready-to-test and removed lifecycle/stale labels Dec 15, 2022

poorbarcode added this to the 3.1.0 milestone Apr 10, 2023

Technoboy- modified the milestones: 3.1.0, 3.2.0 Jul 31, 2023

Technoboy- modified the milestones: 3.2.0, 3.3.0 Dec 22, 2023

coderzc modified the milestones: 3.3.0, 3.4.0 May 8, 2024

lhotari modified the milestones: 4.0.0, 4.1.0 Oct 14, 2024

lhotari added the release/4.0.1 label Oct 14, 2024

lhotari added release/4.0.2 and removed release/4.0.1 labels Dec 6, 2024

lhotari added release/4.0.3 and removed release/4.0.2 labels Jan 6, 2025

lhotari added release/4.0.4 and removed release/4.0.3 labels Feb 19, 2025

lhotari added release/4.0.5 and removed release/4.0.4 labels Mar 31, 2025

lhotari added release/4.0.6 and removed release/4.0.5 labels May 21, 2025

lhotari added release/4.0.7 and removed release/4.0.6 labels Jul 30, 2025

coderzc modified the milestones: 4.1.0, 4.2.0 Sep 1, 2025

lhotari added release/4.0.8 and removed release/4.0.7 labels Sep 27, 2025

lhotari added release/4.0.9 and removed release/4.0.8 labels Nov 11, 2025

lhotari added release/4.0.10 and removed release/4.0.9 labels Feb 9, 2026

	void updateLoggerLevel(String classname, String level) throws PulsarAdminException;
	void updateLoggerLevel(String loggerName, String level) throws PulsarAdminException;

	CompletableFuture<Void> updateLoggerLevelAsync(String classname, String level);
	CompletableFuture<Void> updateLoggerLevelAsync(String loggerName, String level);

	@Parameter(names = {"-c", "--classname"}, description =
	@Parameter(names = {"-l", "--logger-name"}, description =

[feat][broker] Support dynamic update log level at runtime #14200

Are you sure you want to change the base?

[feat][broker] Support dynamic update log level at runtime #14200

Conversation

ethqunzhong commented Feb 10, 2022 • edited by tisonkun Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Motivation

Modifications

Verifying this change

Does this pull request potentially affect one of the following parts:

Documentation

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

ethqunzhong Feb 15, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

eolivelli left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

wolfstudy commented Feb 21, 2022

Uh oh!

ethqunzhong commented Feb 22, 2022

Uh oh!

eolivelli left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

ethqunzhong commented Feb 22, 2022

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

github-actions bot commented Apr 21, 2022

Uh oh!

github-actions bot commented May 28, 2022

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

ethqunzhong commented Feb 10, 2022 •

edited by tisonkun

Loading

ethqunzhong Feb 15, 2022 •

edited

Loading