[pip] PIP-430: Pulsar Broker cache improvements: refactoring eviction and adding a new cache strategy based on expected read count

[lhotari]

Motivation

I'd like to propose PIP-430, which addresses performance and
efficiency issues in Pulsar broker's entry cache eviction mechanisms
and introduces a more efficient caching strategy.

The current broker entry cache implementation has several
production-impacting issues. The size-based eviction doesn't guarantee
removal of globally oldest entries, leading to suboptimal cache
utilization. More critically, the timestamp-based eviction iterates
through all ManagedLedgers every 10ms by default, causing high CPU
utilization in brokers with many topics. Mixed read patterns like
tailing, catch-up, and Key_Shared replays break eviction assumptions,
resulting in unnecessary BookKeeper and S3 reads that increase
operational costs.

PIP-430 introduces two main improvements.
First, a centralized eviction mechanism using a global
RangeCacheRemovalQueue that tracks all cached entries in insertion
order. This replaces the expensive per-ledger iteration with a single
periodic task and ensures true oldest-first eviction globally. The
implementation PR for this part is
#24363.
Second, a new "expected read count" cache strategy where entries track
how many active cursors are anticipated to read them. This allows the
cache to intelligently retain entries that have higher utility,
especially in high fan-out catch-up read scenarios and Key_Shared
subscriptions.

The benefits include reduced CPU overhead, improved cache hit rates
through better eviction decisions, and proper handling of diverse read
patterns. The new strategy is configurable via
cacheEvictionByExpectedReadCount (default: true) and maintains full
backward compatibility with no client-facing API changes.

This addresses long-standing performance issues that particularly
affect production deployments with high topic counts or diverse
consumption patterns. The refactored architecture also provides a
solid foundation for future cache optimizations.

The full proposal can be found at: #24444
Rendered PIP document:
https://github.com/lhotari/pulsar/blob/lh-pip-430/pip/pip-430.md

I welcome your feedback and discussion on this proposal. Please share
your thoughts, concerns, or suggestions.

Mailing list discussion: https://lists.apache.org/thread/o1ozbg468kxfd38pxk2ppzsstdnxnok2

Documentation

• doc
• doc-required
• doc-not-needed
• doc-complete

[lhotari]

Implementation related:

• First part is implemented in [refactor][ml] Replace cache eviction algorithm with centralized removal queue and job #24363
• There's a WIP PoC of the second part in [WIP] Add cacheEvictionByExpectedReadCount solution lhotari/pulsar#209
  • This doesn't implement managedLedgerCacheEvictionTimeThresholdMillisMax yet.