Topic partition's segmented storage space calculation error

[xuesongxs]

Describe the bug
The storage space of partition is much larger than 4G and entries is much larger than 50000 (managedLedgerMaxSizePerLedgerMbytes=4096, managedLedgerMaxEntriesPerLedger=50000 in broker.conf).

Pulsar v2.8.1

broker.conf
managedLedgerMaxEntriesPerLedger=50000
managedLedgerMaxSizePerLedgerMbytes=4096

producer code:
            producer = client.newProducer(Schema.STRING)
                    .topic(topic)
                    .producerName(producerName)
                    .sendTimeout(0, TimeUnit.SECONDS)
                    .enableBatching(true)
                    .batchingMaxMessages(1000)
                    .compressionType(CompressionType.ZSTD)
                    .blockIfQueueFull(true)
                    .create();

To Reproduce
Steps to reproduce the behavior:

1. Start producer
2. View the segmented storage of the partition in pulsar manager
3. The storage space of partition is much larger than 4G

[gaozhangmin]

rollover will be triggered only when max entries have been written to the ledger and at least min-time has passed.

Minimum time between ledger rollover for a topic
managedLedgerMinLedgerRolloverTimeMinutes=10

[xuesongxs]

en max entries have been written to the ledger and at least

However, the actual storage space exceeds the configured threshold too much. Is this reasonable?

[hezhangjian]

en max entries have been written to the ledger and at least

However, the actual storage space exceeds the configured threshold too much. Is this reasonable?

The managedLedgerMinLedgerRolloverTimeMinutes is to avoild frequencyly ledger rollover, if you want exactly less than size, you can tune this to 0

[xuesongxs]

en max entries have been written to the ledger and at least

However, the actual storage space exceeds the configured threshold too much. Is this reasonable?

The managedLedgerMinLedgerRolloverTimeMinutes is to avoild frequencyly ledger rollover, if you want exactly less than size, you can tune this to 0

I tested that if the producer does not use compression, it scrolls according to the entries and size of the ledger, but after the producer uses compression, it does not scroll according to the entries and size of the ledger. It can only rely on the configuration of managedledgerminledger rollovertimeminutes. I want to express this meaning.

[hezhangjian]

en max entries have been written to the ledger and at least

However, the actual storage space exceeds the configured threshold too much. Is this reasonable?

The managedLedgerMinLedgerRolloverTimeMinutes is to avoild frequencyly ledger rollover, if you want exactly less than size, you can tune this to 0

I tested that if the producer does not use compression, it scrolls according to the entries and size of the ledger, but after the producer uses compression, it does not scroll according to the entries and size of the ledger. It can only rely on the configuration of managedledgerminledger rollovertimeminutes. I want to express this meaning.

interesting, let me try to reproduce it

[github-actions]

The issue had no activity for 30 days, mark with Stale label.