Documentation: units for storageSize

[candlerb]

Is your feature request related to a problem? Please describe.

ubuntu@ldex-pulsar:~$ apache-pulsar-2.4.1/bin/pulsar-admin topics stats public/default/avro-topic | grep storageSize
  "storageSize" : 1980,
ubuntu@ldex-pulsar:~$ apache-pulsar-2.4.1/bin/pulsar-admin topics stats public/default/my-topic | grep storageSize
  "storageSize" : 3579,

What are those units? Bytes, KiB, MiB, GiB, something else?

Describe the solution you'd like
Document at https://pulsar.apache.org/docs/en/pulsar-admin/#stats-1 and/or https://pulsar.apache.org/admin-rest-api/?version=2.4.1#operation/getStats

[Jennifer88huang-zz]

@candlerb The unit is Byte. We'll add it in docs. Thanks for your question.

[candlerb]

That doesn't make sense for my test cluster. When I got the figures posted at the top of this mail, my data/ directory was about 5.5GB in size. I had posted many test messages, each 1MB in size, although some of these had expired and were awaiting garbage collection.

But I see large figures at site2/docs/admin-api-persistent-topics.md which do make sense in bytes. I'm just trying to understand what I see in my cluster.

[candlerb]

I've made a simple test for this. Before run:

ubuntu@ldex-pulsar:~$ apache-pulsar-2.4.1/bin/pulsar-admin topics stats temp
{
  "msgRateIn" : 0.0,
  "msgThroughputIn" : 0.0,
  "msgRateOut" : 0.0,
  "msgThroughputOut" : 0.0,
  "averageMsgSize" : 0.0,
  "storageSize" : 0,
  "publishers" : [ ],
  "subscriptions" : { },
  "replication" : { },
  "deduplicationStatus" : "Disabled"
}
ubuntu@ldex-pulsar:~$ du -sck apache-pulsar-2.4.1/data/
2371392	apache-pulsar-2.4.1/data/
2371392	total

Run program which publishes 500 x 1MB messages to topic "temp" (see below)

After run:

ubuntu@ldex-pulsar:~$ apache-pulsar-2.4.1/bin/pulsar-admin topics stats temp
{
  "msgRateIn" : 0.0,
  "msgThroughputIn" : 0.0,
  "msgRateOut" : 0.0,
  "msgThroughputOut" : 0.0,
  "averageMsgSize" : 0.0,
  "storageSize" : 0,
  "publishers" : [ ],
  "subscriptions" : { },
  "replication" : { },
  "deduplicationStatus" : "Disabled"
}
ubuntu@ldex-pulsar:~$ du -sck apache-pulsar-2.4.1/data/
3348208	apache-pulsar-2.4.1/data/
3348208	total

Weird. Stats are supposed to update every minute, but:

ubuntu@ldex-pulsar:~$ sleep 60; apache-pulsar-2.4.1/bin/pulsar-admin topics stats temp
{
  "msgRateIn" : 0.0,
  "msgThroughputIn" : 0.0,
  "msgRateOut" : 0.0,
  "msgThroughputOut" : 0.0,
  "averageMsgSize" : 0.0,
  "storageSize" : 0,
  "publishers" : [ ],
  "subscriptions" : { },
  "replication" : { },
  "deduplicationStatus" : "Disabled"
}
ubuntu@ldex-pulsar:~$ apache-pulsar-2.4.1/bin/pulsar-admin topics stats persistent://public/default/temp
{
  "msgRateIn" : 0.0,
  "msgThroughputIn" : 0.0,
  "msgRateOut" : 0.0,
  "msgThroughputOut" : 0.0,
  "averageMsgSize" : 0.0,
  "storageSize" : 0,
  "publishers" : [ ],
  "subscriptions" : { },
  "replication" : { },
  "deduplicationStatus" : "Disabled"
}
ubuntu@ldex-pulsar:~$ apache-pulsar-2.4.1/bin/pulsar-admin topics stats-internal persistent://public/default/temp
{
  "entriesAddedCounter" : 500,
  "numberOfEntries" : 2632,
  "totalSize" : 2134036790,
  "currentLedgerEntries" : 500,
  "currentLedgerSize" : 500012872,
  "lastLedgerCreatedTimestamp" : "2019-11-20T09:24:19.756Z",
  "waitingCursorsCount" : 0,
  "pendingAddEntriesCount" : 0,
  "lastConfirmedEntry" : "122673:499",
  "state" : "LedgerOpened",
  "ledgers" : [ {
    "ledgerId" : 108697,
    "entries" : 1633,
    "size" : 1632046009,
    "offloaded" : false
  }, {
    "ledgerId" : 109540,
    "entries" : 499,
    "size" : 1977909,
    "offloaded" : false
  }, {
    "ledgerId" : 122673,
    "entries" : 0,
    "size" : 0,
    "offloaded" : false
  } ],
  "cursors" : { }
}

If I give an invalid topic name to stats (e.g. tempz) it tells me the topic does not exist, so I must be looking at the right topic. The retention period on this namespaces is set to 1440 minutes, so the lack of consumers shouldn't be an issue; and stats-internal shows storage in the ledgers.

So the problem more likely is: I don't understand what the "storageSize" parameter of "stats" is actually representing.

---

from collections import defaultdict
import pulsar
import time

NUM_MESSAGES = 500
MESSAGE_SIZE = 1_000_000

client = pulsar.Client('pulsar://localhost:6650')

producer = client.create_producer('temp', producer_name='fred', compression_type=pulsar.CompressionType.NONE)

sent = 0
bytes = 0
results = defaultdict(lambda: 0)
def ack(res, msg):
    global sent, bytes
    sent += 1
    bytes += len(msg.data())
    results[str(res)] += 1

for i in range(NUM_MESSAGES):
    data = b"x" * MESSAGE_SIZE
    producer.send_async(data, callback=ack)

print("Flushing...")
producer.flush()
print("Flush complete")
for i in range(600):
    print("Sent: %d messages %d bytes" % (sent, bytes))
    if sent == NUM_MESSAGES:
        break
    time.sleep(0.1)
else:
    print("Never got all the messages!")
print("Results: %r" % dict(results))

producer.close()
client.close()

[Jennifer88huang-zz]

@jiazhai could you please help on this issue? Thank you.

[sijie]

@candlerb

1. first question, how to understand "storage size"?

The total size = sum(size of each ledger) + currentLedgerSize.

"size of each ledger" is the size field of a ledger stat in ledgers list. The size of the last ledger (active and not-closed ledger) is shown in currentLedgerSize.

The size of each ledger is the total of the entry size. Each entry is comprised of size of entry data and extra metadata (a couple of bytes).

Each entry is a message batch. Each message in a message batch also carries additional metadata (msg metadata plus checksum).

Because additional metadata is added to each message and each entry (message batch), the storage size will be larger than (number message * message size).

2. understand the disk usage under /data.

the storage size doesn't simply equal to the disk usage under /data.

the disk usage of /data is comprised of two parts, one is journal, the other is ledger.

1. The journal is reclaimed after the data is persisted to ledger directories, however bookies keep journalMaxBackups number of journals for disaster recovery.

2. The ledger is comprised of entry data and index. The index is stored in rocksdb. So the disk usage is also larger than the storage size you will see in the stats.

---

Hope this gives you a better view about storage size and disk usage.

---

Back to the question about the unit of StorageSize, it is bytes. @jennifer88huang you can proceed the fix #5707. We can have a separate pull request to document what does storage size mean.

[jiazhai]

There was a fix for the storageSize always 0 issue: #5108

[jiazhai]

But #5108 is not contained in 2.4.1

[Jennifer88huang-zz]

Since I've merged #5707, this issue is closed automatically.
@candlerb If you have any further question on this, you can reopen this issue.