[improve] [broker] Do not print an Error log when responding to `HTTP-404` when calling `Admin API` and the topic does not exist. #21995

poorbarcode · 2024-01-30T17:53:26Z

Motivation

Pulsar printed an Error log when responding to HTTP-404 when calling Admin API and the topic does not exist.

The behavior is expected, so it should not print an error log.

2024-01-30T15:52:17,143+0000 [bookkeeper-ml-scheduler-OrderedScheduler-2-0] ERROR org.apache.pulsar.broker.admin.v2.PersistentTopics - [flink-cluster@belvedere.auth.streamnative.cloud] Failed to get stats for persistent://xx/xx/xx-partition-0
java.util.concurrent.CompletionException: org.apache.pulsar.broker.web.RestException: Topic partitions were not yet created
        at java.util.concurrent.CompletableFuture.encodeThrowable(CompletableFuture.java:315) ~[?:?]
        at java.util.concurrent.CompletableFuture.uniApplyNow(CompletableFuture.java:687) ~[?:?]
        at java.util.concurrent.CompletableFuture.uniApplyStage(CompletableFuture.java:662) ~[?:?]
        at java.util.concurrent.CompletableFuture.thenApply(CompletableFuture.java:2168) ~[?:?]
        at org.apache.pulsar.broker.admin.impl.PersistentTopicsBase.topicNotFoundReasonAsync(PersistentTopicsBase.java:4519) ~[io.streamnative-pulsar-broker-3.0.2.1.jar:3.0.2.1]
        at org.apache.pulsar.broker.admin.impl.PersistentTopicsBase.lambda$getTopicReferenceAsync$438(PersistentTopicsBase.java:4499) ~[io.streamnative-pulsar-broker-3.0.2.1.jar:3.0.2.1]
        at java.util.Optional.orElseGet(Optional.java:364) ~[?:?]
        at org.apache.pulsar.broker.admin.impl.PersistentTopicsBase.lambda$getTopicReferenceAsync$439(PersistentTopicsBase.java:4499) ~[io.streamnative-pulsar-broker-3.0.2.1.jar:3.0.2.1]
        at java.util.concurrent.CompletableFuture$UniCompose.tryFire(CompletableFuture.java:1150) ~[?:?]
        at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:510) ~[?:?]
        at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:2147) ~[?:?]
        at org.apache.pulsar.broker.service.BrokerService$2.openLedgerFailed(BrokerService.java:1800) ~[io.streamnative-pulsar-broker-3.0.2.1.jar:3.0.2.1]
        at org.apache.bookkeeper.mledger.impl.ManagedLedgerFactoryImpl.lambda$asyncOpen$8(ManagedLedgerFactoryImpl.java:421) ~[io.streamnative-managed-ledger-3.0.2.1.jar:3.0.2.1]
        at java.util.concurrent.CompletableFuture.uniExceptionally(CompletableFuture.java:990) ~[?:?]
        at java.util.concurrent.CompletableFuture$UniExceptionally.tryFire(CompletableFuture.java:974) ~[?:?]
        at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:510) ~[?:?]
        at java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:2162) ~[?:?]
        at org.apache.bookkeeper.mledger.impl.ManagedLedgerFactoryImpl$2.initializeFailed(ManagedLedgerFactoryImpl.java:416) ~[io.streamnative-managed-ledger-3.0.2.1.jar:3.0.2.1]
        at org.apache.bookkeeper.mledger.impl.ManagedLedgerImpl$1.operationFailed(ManagedLedgerImpl.java:458) ~[io.streamnative-managed-ledger-3.0.2.1.jar:3.0.2.1]
        at org.apache.bookkeeper.mledger.impl.MetaStoreImpl.lambda$getManagedLedgerInfo$3(MetaStoreImpl.java:150) ~[io.streamnative-managed-ledger-3.0.2.1.jar:3.0.2.1]
        at java.util.concurrent.CompletableFuture$UniAccept.tryFire(CompletableFuture.java:718) ~[?:?]
        at java.util.concurrent.CompletableFuture$Completion.run(CompletableFuture.java:482) ~[?:?]
        at org.apache.bookkeeper.common.util.OrderedExecutor$TimedRunnable.run(OrderedExecutor.java:201) ~[org.apache.bookkeeper-bookkeeper-common-4.16.3.jar:4.16.3]
        at org.apache.bookkeeper.common.util.SingleThreadSafeScheduledExecutorService$SafeRunnable.run(SingleThreadSafeScheduledExecutorService.java:46) ~[org.apache.bookkeeper-bookkeeper-common-4.16.3.jar:4.16.3]
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539) ~[?:?]
        at java.util.concurrent.FutureTask.run(FutureTask.java:264) ~[?:?]
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304) ~[?:?]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) ~[?:?]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) ~[?:?]
        at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) ~[io.netty-netty-common-4.1.100.Final.jar:4.1.100.Final]
        at java.lang.Thread.run(Thread.java:840) ~[?:?]     
Caused by: org.apache.pulsar.broker.web.RestException: Topic partitions were not yet created
        at org.apache.pulsar.broker.admin.impl.PersistentTopicsBase.lambda$topicNotFoundReasonAsync$442(PersistentTopicsBase.java:4521) ~[io.streamnative-pulsar-broker-3.0.2.1.jar:3.0.2.1]
        at java.util.concurrent.CompletableFuture.uniApplyNow(CompletableFuture.java:684) ~[?:?]
        ... 29 more
2024-01-30T15:52:17,143+0000 [bookkeeper-ml-scheduler-OrderedScheduler-2-0] INFO  org.eclipse.jetty.server.RequestLog - 127.0.0.6 - - [30/Jan/2024:15:52:17 +0000] "GET /admin/v2/persistent/xx/xx/xx-partition-0/stats?getPreciseBacklog=true&authoritative=false&subscriptionBacklogSize=true&getEarliestTimeInBacklog=false HTTP/1.1" 404 50 "-" "Pulsar-Java-v2.11.1" 7

Modifications

Admin API does not print an Error log when responding to Http-404.

Documentation

doc
doc-required
doc-not-needed
doc-complete

Matching PR in forked repository

PR in forked repository: x

pulsar-broker/src/main/java/org/apache/pulsar/broker/admin/AdminResource.java

Technoboy- · 2024-01-31T07:05:41Z

Also reported by #21848

poorbarcode · 2024-01-31T08:16:50Z

Since the improvement "Do not try to open ML when the topic meta does not exist and do not expect to create a new one", will change the behavior of TopicEventsListener, I split this change to a separate PR #22004.

Behavior changes of `TopicEventsListener`

Before changes: the listener will receive a Topic Load event when calling pulsar-admin topics stats <topic name> even if the topic was not created.
After changes: the listener will not receive a Topic Load event when calling pulsar-admin topics stats <topic name> if the topic was not created.

… exist and do not expect to create a new one. #21995 (#22004) Co-authored-by: Jiwe Guo <technoboy@apache.org>

…-404` when calling `Admin API` and the topic does not exist. (#21995)

… exist and do not expect to create a new one. #21995 (#22004) Co-authored-by: Jiwe Guo <technoboy@apache.org>

heesung-sn · 2024-02-27T19:08:14Z

@poorbarcode could you help to cherry-pick this PR to branch-3.0? I see conflicts.

… exist and do not expect to create a new one. #21995 (#22004) Co-authored-by: Jiwe Guo <technoboy@apache.org> (cherry picked from commit 1c652f5)

…-404` when calling `Admin API` and the topic does not exist. (apache#21995) (cherry picked from commit 5ab1c05)

… exist and do not expect to create a new one. apache#21995 (apache#22004) Co-authored-by: Jiwe Guo <technoboy@apache.org> (cherry picked from commit d18831f)

…-404` when calling `Admin API` and the topic does not exist. (apache#21995) (cherry picked from commit 5ab1c05)

… exist and do not expect to create a new one. apache#21995 (apache#22004) Co-authored-by: Jiwe Guo <technoboy@apache.org> (cherry picked from commit d18831f)

…-404` when calling `Admin API` and the topic does not exist. (#21995) (cherry picked from commit 48b4481)

codelipenghui · 2024-03-07T01:17:26Z

@heesung-sn I have cherry-picked to branch-3.0

…-404` when calling `Admin API` and the topic does not exist. (#21995) (cherry picked from commit 48b4481)

dojiong · 2024-03-27T03:22:15Z

pulsar-broker/src/main/java/org/apache/pulsar/broker/admin/impl/PersistentTopicsBase.java

@@ -4634,7 +4634,7 @@ protected void internalGetLastMessageId(AsyncResponse asyncResponse, boolean aut
                    });
                }).exceptionally(ex -> {
                    // If the exception is not redirect exception we need to log it.
-                    if (!isRedirectException(ex)) {
+                    if (!isNot307And404Exception(ex)) {


it seems that this line only log the 307 and 404 exceptions. not(not 307 and not 404)

Good pointer! Now the implementation is only not 307 | 404(so far, seems broker will not throw other 30x)

jaak-pruulmann-sympower · 2024-04-02T11:48:15Z

pulsar-broker/src/main/java/org/apache/pulsar/broker/admin/v2/NonPersistentTopics.java

@@ -132,7 +132,7 @@ public void getInternalStats(
                })
                .thenAccept(asyncResponse::resume)
                .exceptionally(ex -> {
-                    if (!isRedirectException(ex)) {
+                    if (!isNot307And404Exception(ex)) {


Looks like this line and all the others where !isRedirectException(ex) has been replaced with !isNot307.. are wrong, as the new line should NOT have the negation exclamation mark any more.

poorbarcode added 3 commits January 31, 2024 01:13

[improve] [admin] improve topic stats if not exists

c024bae

do not print error log if respond 404

202d367

-

faedb06

poorbarcode self-assigned this Jan 30, 2024

github-actions bot added the doc-not-needed Your PR changes do not impact docs label Jan 30, 2024

poorbarcode requested review from lhotari, hangc0276, codelipenghui, liangyepianzhou and mattisonchao January 30, 2024 17:58

poorbarcode added this to the 3.3.0 milestone Jan 30, 2024

poorbarcode added release/3.0.3 release/3.1.3 release/3.2.1 labels Jan 30, 2024

lhotari requested changes Jan 30, 2024

View reviewed changes

pulsar-broker/src/main/java/org/apache/pulsar/broker/admin/AdminResource.java Outdated Show resolved Hide resolved

poorbarcode added 2 commits January 31, 2024 10:48

fix bug

29b8bcb

fix bug

aaa960b

poorbarcode requested a review from lhotari January 31, 2024 02:53

lhotari approved these changes Jan 31, 2024

View reviewed changes

poorbarcode added 3 commits January 31, 2024 15:23

fix test

cd50855

fix test

e1e43be

split to 2 PRs

1fe7ba8

poorbarcode changed the title ~~[improve] [broker] Do not try to open ML when the topic meta does not exist and do not expect to create a new one.~~ [improve] [broker] Do not print an Error log when responding to HTTP-404 when calling Admin API and the topic does not exist. Jan 31, 2024

poorbarcode added release/2.10.6 release/2.11.4 labels Jan 31, 2024

Technoboy- added the category/performance Performance issues fix or improvements label Jan 31, 2024

congbobo184 approved these changes Jan 31, 2024

View reviewed changes

Technoboy- approved these changes Jan 31, 2024

View reviewed changes

dao-jun pushed a commit that referenced this pull request Feb 23, 2024

[improve] [broker] Do not try to open ML when the topic meta does not…

1c652f5

… exist and do not expect to create a new one. #21995 (#22004) Co-authored-by: Jiwe Guo <technoboy@apache.org>

Technoboy- added a commit that referenced this pull request Feb 26, 2024

[improve] [broker] Do not try to open ML when the topic meta does not…

b0910fd

… exist and do not expect to create a new one. #21995 (#22004) Co-authored-by: Jiwe Guo <technoboy@apache.org>

Technoboy- pushed a commit that referenced this pull request Feb 27, 2024

[improve] [broker] Do not print an Error log when responding to `HTTP…

5ab1c05

…-404` when calling `Admin API` and the topic does not exist. (#21995)

Technoboy- added the cherry-picked/branch-3.1 label Feb 27, 2024

Technoboy- added a commit that referenced this pull request Feb 27, 2024

[improve] [broker] Do not try to open ML when the topic meta does not…

d18831f

… exist and do not expect to create a new one. #21995 (#22004) Co-authored-by: Jiwe Guo <technoboy@apache.org>

mukesh-ctds pushed a commit to datastax/pulsar that referenced this pull request Mar 1, 2024

[improve] [broker] Do not print an Error log when responding to `HTTP…

8a37714

…-404` when calling `Admin API` and the topic does not exist. (apache#21995) (cherry picked from commit 5ab1c05)

lhotari added release/2.10.7 release/2.11.5 release/2.10.6 release/2.11.4 and removed release/2.10.6 release/2.11.4 release/2.10.7 release/2.11.5 labels Mar 5, 2024

mukesh-ctds pushed a commit to datastax/pulsar that referenced this pull request Mar 6, 2024

[improve] [broker] Do not print an Error log when responding to `HTTP…

cb0703d

…-404` when calling `Admin API` and the topic does not exist. (apache#21995) (cherry picked from commit 5ab1c05)

lhotari pushed a commit that referenced this pull request Mar 6, 2024

[improve] [broker] Do not print an Error log when responding to `HTTP…

ae8d149

…-404` when calling `Admin API` and the topic does not exist. (#21995) (cherry picked from commit 48b4481)

lhotari added the cherry-picked/branch-2.11 label Mar 6, 2024

codelipenghui pushed a commit that referenced this pull request Mar 7, 2024

[improve] [broker] Do not print an Error log when responding to `HTTP…

d9be026

…-404` when calling `Admin API` and the topic does not exist. (#21995) (cherry picked from commit 48b4481)

codelipenghui added the cherry-picked/branch-3.0 label Mar 7, 2024

codelipenghui pushed a commit that referenced this pull request Mar 7, 2024

[improve] [broker] Do not print an Error log when responding to `HTTP…

a080aeb

…-404` when calling `Admin API` and the topic does not exist. (#21995) (cherry picked from commit 48b4481)

codelipenghui added the cherry-picked/branch-2.10 label Mar 7, 2024

dojiong reviewed Mar 27, 2024

View reviewed changes

jaak-pruulmann-sympower reviewed Apr 2, 2024

View reviewed changes

This was referenced Apr 3, 2024

[Bug][broker] Broker contains invalid condition for logging exceptions #22409

Closed

[fix][broker] Fix invalid condition in logging exceptions #22412

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[improve] [broker] Do not print an Error log when responding to `HTTP-404` when calling `Admin API` and the topic does not exist. #21995

[improve] [broker] Do not print an Error log when responding to `HTTP-404` when calling `Admin API` and the topic does not exist. #21995

poorbarcode commented Jan 30, 2024 •

edited

Loading

Technoboy- commented Jan 31, 2024

poorbarcode commented Jan 31, 2024

heesung-sn commented Feb 27, 2024

codelipenghui commented Mar 7, 2024

dojiong Mar 27, 2024 •

edited

Loading

poorbarcode Mar 27, 2024

jaak-pruulmann-sympower Apr 2, 2024

[improve] [broker] Do not print an Error log when responding to HTTP-404 when calling Admin API and the topic does not exist. #21995

[improve] [broker] Do not print an Error log when responding to HTTP-404 when calling Admin API and the topic does not exist. #21995

Conversation

poorbarcode commented Jan 30, 2024 • edited Loading

Motivation

Modifications

Documentation

Matching PR in forked repository

Technoboy- commented Jan 31, 2024

poorbarcode commented Jan 31, 2024

Behavior changes of TopicEventsListener

heesung-sn commented Feb 27, 2024

codelipenghui commented Mar 7, 2024

dojiong Mar 27, 2024 • edited Loading

Choose a reason for hiding this comment

poorbarcode Mar 27, 2024

Choose a reason for hiding this comment

jaak-pruulmann-sympower Apr 2, 2024

Choose a reason for hiding this comment

[improve] [broker] Do not print an Error log when responding to `HTTP-404` when calling `Admin API` and the topic does not exist. #21995

[improve] [broker] Do not print an Error log when responding to `HTTP-404` when calling `Admin API` and the topic does not exist. #21995

poorbarcode commented Jan 30, 2024 •

edited

Loading

Behavior changes of `TopicEventsListener`

dojiong Mar 27, 2024 •

edited

Loading