Allow to configure the managed ledger cache eviction frequency #4066
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
merlimat
added
the
type/enhancement
merlimat
force-pushed
the
conf-cache-eviction-freq
branch
from
4f12504
to
7facb33
Compare
rdhabalia
reviewed
Apr 23, 2019
managed-ledger/src/main/java/org/apache/bookkeeper/mledger/impl/ManagedLedgerFactoryImpl.java
Outdated
Show resolved
Hide resolved
managed-ledger/src/main/java/org/apache/bookkeeper/mledger/impl/ManagedLedgerFactoryImpl.java
Outdated
Show resolved
Hide resolved
managed-ledger/src/main/java/org/apache/bookkeeper/mledger/impl/EntryImpl.java
Show resolved
Hide resolved
managed-ledger/src/main/java/org/apache/bookkeeper/mledger/impl/ManagedLedgerImpl.java
Outdated
Show resolved
Hide resolved
merlimat
force-pushed
the
conf-cache-eviction-freq
branch
from
22eaac4
to
60cda05
Compare
rdhabalia
approved these changes
Apr 24, 2019
| // activate caught up cursors | ||
| cursors.forEach(cursor -> { | ||
| if (cursor.getNumberOfEntries() < maxActiveCursorBacklogEntries) { | ||
| if (cursor.getNumberOfEntries() < backloggedCursorThresholdEntries) { |
pulsar-broker/src/test/java/org/apache/pulsar/client/api/SimpleProducerConsumerTest.java
Outdated
Show resolved
Hide resolved
|
@rdhabalia Updated with an additional time threshold and check for slowest active consumer. Please take another look. |
rdhabalia
reviewed
Apr 24, 2019
managed-ledger/src/main/java/org/apache/bookkeeper/mledger/util/RangeCache.java
Outdated
Show resolved
Hide resolved
managed-ledger/src/main/java/org/apache/bookkeeper/mledger/impl/ManagedLedgerFactoryImpl.java
Outdated
Show resolved
Hide resolved
| @@ -136,9 +136,6 @@ public synchronized void updateStats( | |||
| } catch (Exception e) { | |||
| log.error("Failed to generate topic stats for topic {}: {}", name, e.getMessage(), e); | |||
| } | |||
| // this task: helps to activate inactive-backlog-cursors which have caught up and | |||
| // connected, also deactivate active-backlog-cursors which has backlog | |||
| ((PersistentTopic) topic).getManagedLedger().checkBackloggedCursors(); | |||
There was a problem hiding this comment.
if we have removed from here then who triggers checkBackloggedCursors(..)?
|
@merlimat we may also want to add config to standalone-config file and site-documentation file. |
Which one are you referring? I added into broker.conf |
may be we can add configs to : |
|
Good point, Added. |
ivankelly
approved these changes
Apr 26, 2019
|
@ivankelly Good point. Added explicit test for time-based eviction. |
|
run java8 tests |
This was referenced Jan 13, 2020
Closed
sijie
pushed a commit
that referenced
this pull request
Jan 18, 2020
### Motivation Some parameters are added in the `broker.conf` and `standalone.conf` files. However, those parameters are not updated in the docs. See the following PRs for details: #4150, #4066, #4197, #3819, #4261, #4273, #4320. ### Modifications Add those parameter info, and sync docs with the code. Does not update the description quite much, there are two reasons for this: 1. Keep doc content consistent with code. We need to update the description for those parameters in the code first, and then sync them in docs. 2. Will adopt a generator to generate those content automatically in the near future.
huangdx0726
pushed a commit
to huangdx0726/pulsar
that referenced
this pull request
Aug 24, 2020
### Motivation Some parameters are added in the `broker.conf` and `standalone.conf` files. However, those parameters are not updated in the docs. See the following PRs for details: apache#4150, apache#4066, apache#4197, apache#3819, apache#4261, apache#4273, apache#4320. ### Modifications Add those parameter info, and sync docs with the code. Does not update the description quite much, there are two reasons for this: 1. Keep doc content consistent with code. We need to update the description for those parameters in the code first, and then sync them in docs. 2. Will adopt a generator to generate those content automatically in the near future.
3 tasks
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Motivation
Currently there's a rate limiter that avoid evicting from cache each time the read position is updated but rather only does that every 1 sec at max. The primary reason is to avoid mutex contention across different cursors trying to trigger the eviction in parallel.
The frequency should be higher though, to make sure we promptly discard all entries once all active cursors have moved past a particular point.
Modifications
broker.conf