Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[improve] [broker] improve read entry error log for troubleshooting #21169

Merged
merged 1 commit into from Sep 13, 2023

Conversation

poorbarcode
Copy link
Contributor

@poorbarcode poorbarcode commented Sep 12, 2023

Motivation

2023-09-12T05:35:43,141+0000 [pulsar-msg-expiry-monitor-25-1] ERROR org.apache.bookkeeper.mledger.impl.ManagedLedgerImpl - [public/default/persistent/xxx] Failed to get message with ledger 7213399:0 the ledgerId does not belong to this topic or has been deleted.
2023-09-12T05:35:43,141+0000 [pulsar-msg-expiry-monitor-25-1] WARN  org.apache.pulsar.broker.service.persistent.PersistentTopic - [persistent://public/default/xxx] Error while getting the oldest message
2023-09-12T05:35:43,292+0000 [pulsar-msg-expiry-monitor-25-1] WARN  org.apache.pulsar.broker.service.persistent.PersistentTopic - [persistent://public/default/xxx] Error while getting the oldest message
2023-09-12T05:35:43,292+0000 [pulsar-msg-expiry-monitor-25-1] ERROR org.apache.bookkeeper.mledger.impl.ManagedLedgerImpl - [public/default/persistent/xxx] Failed to get message with ledger 7210605:2 the ledgerId does not belong to this topic or has been deleted.
2023-09-12T05:35:43,292+0000 [pulsar-msg-expiry-monitor-25-1] ERROR org.apache.bookkeeper.mledger.impl.ManagedLedgerImpl - [public/default/persistent/xxx] Failed to get message with ledger 7210605:2 the ledgerId does not belong to this topic or has been deleted.
2023-09-12T05:35:43,292+0000 [pulsar-msg-expiry-monitor-25-1] WARN  org.apache.pulsar.broker.service.persistent.PersistentTopic - [persistent://public/default/xxx] Error while getting the oldest message
2023-09-12T05:35:43,529+0000 [pulsar-msg-expiry-monitor-25-1] ERROR org.apache.bookkeeper.mledger.impl.ManagedLedgerImpl - [public/PRICING/persistent/xxx] Failed to get message with ledger 7168220:3818 the ledgerId does not belong to this topic or has been deleted.
2023-09-12T05:35:43,529+0000 [pulsar-msg-expiry-monitor-25-1] WARN  org.apache.pulsar.broker.service.persistent.PersistentTopic - [persistent://public/PRICING/xxx] Error while getting the oldest message

There is a log of error logs indicating that the broker tries to read a deleted ledger. Since there are a lot of subscriptions under this topic(durable and non-durable) and the ledger id already removed from the topic's ledgers list, we do not know what happened.

Modifications

If the read entry fails, add the cursor name to the error log.

Documentation

  • doc
  • doc-required
  • doc-not-needed
  • doc-complete

Matching PR in forked repository

PR in forked repository: x

@codecov-commenter
Copy link

Codecov Report

Merging #21169 (c6310a8) into master (550476f) will increase coverage by 36.34%.
Report is 4 commits behind head on master.
The diff coverage is 43.24%.

Impacted file tree graph

@@              Coverage Diff              @@
##             master   #21169       +/-   ##
=============================================
+ Coverage     36.84%   73.18%   +36.34%     
- Complexity    12179    32466    +20287     
=============================================
  Files          1698     1887      +189     
  Lines        130330   140064     +9734     
  Branches      14247    15427     +1180     
=============================================
+ Hits          48016   102509    +54493     
+ Misses        75966    29460    -46506     
- Partials       6348     8095     +1747     
Flag Coverage Δ
inttests 24.04% <27.02%> (-0.11%) ⬇️
systests 24.98% <21.62%> (-0.12%) ⬇️
unittests 72.47% <43.24%> (+40.51%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Files Changed Coverage Δ
...che/bookkeeper/mledger/impl/ManagedCursorImpl.java 78.96% <0.00%> (+33.33%) ⬆️
...che/bookkeeper/mledger/impl/ManagedLedgerImpl.java 81.27% <0.00%> (+31.27%) ⬆️
...keeper/mledger/impl/ReadOnlyManagedLedgerImpl.java 53.62% <0.00%> (+53.62%) ⬆️
...pulsar/broker/admin/impl/PersistentTopicsBase.java 63.81% <0.00%> (+53.63%) ⬆️
...va/org/apache/pulsar/broker/service/ServerCnx.java 71.28% <0.00%> (+29.84%) ⬆️
...roker/service/persistent/PersistentReplicator.java 68.43% <0.00%> (+25.29%) ⬆️
...ker/service/persistent/PersistentSubscription.java 75.93% <0.00%> (+25.07%) ⬆️
...sar/broker/service/persistent/PersistentTopic.java 80.54% <0.00%> (+26.19%) ⬆️
...r/impl/SnapshotSegmentAbortedTxnProcessorImpl.java 78.51% <0.00%> (+78.51%) ⬆️
...rg/apache/pulsar/client/impl/PulsarClientImpl.java 74.38% <66.66%> (+14.26%) ⬆️
... and 3 more

... and 1438 files with indirect coverage changes

@Technoboy- Technoboy- merged commit 65706c6 into apache:master Sep 13, 2023
67 of 71 checks passed
poorbarcode added a commit that referenced this pull request Sep 14, 2023
poorbarcode added a commit that referenced this pull request Sep 14, 2023
poorbarcode added a commit that referenced this pull request Sep 14, 2023
mukesh-ctds pushed a commit to datastax/pulsar that referenced this pull request Apr 15, 2024
mukesh-ctds pushed a commit to datastax/pulsar that referenced this pull request Apr 17, 2024
mukesh-ctds pushed a commit to datastax/pulsar that referenced this pull request Apr 19, 2024
srinath-ctds pushed a commit to datastax/pulsar that referenced this pull request Apr 23, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

6 participants