high publish latency of Pulsar on HDD

[krumo]

I deployed a Pulsar cluster on 6 identical machines according to Cluster Setup. Each machine is equipped with a Intel(R) Xeon(R) CPU E5645 @ 2.40GHz, 32G RAM, 4*1T 7200rpm HDD and 1Gbit NIC. And I utilized 3 machines to run a zookeeper ensemble and brokers, 2 machines to run Bookkeepers( journal, ledger and index files are located in different disk devices) and 1 machine to run a pulsar-perf produce Command.

It's my plan to repeat the test posted in Announcement post on Yahoo Eng Blog. Thus, I used one client to publish messages to 1 topic for 1 hour with different message size and rate. My result is as attached:

It's obvious that my result is much worse than the official benchmark. By analyzing the logs I find that the most of latency is spent on writing entries to Bookkeeper. So I compared the client publish latency and bookie write latency for several batches and visualize it as follows:

Plus, the disk IO stat for bookie is also recorded and shown as follows:

From my stat we can see most of the time the bookie is under low load so I think I still could do some thing to improve the performance of Pulsar on HDD.

However, there is nearly nothing about optimizing the performance of bookkeeper on the Internet. Although I did try modifying the parameters in config file of bookkeeper, I don't see any improvement. All of my tests are implemented with default parameters in Pulsar.

So I wonder is there anybody trying to do the similar test like me? And do you know how to improve the performance of Pulsar on HDD? I would be grateful if somebody could give me practical advice.

System configuration

OS: CentOS 6.4
File System: ext4
Raid: No Raid
Pulsar version: 1.16.2

[agarman]

Edit: ignore comment, based on misreading of OP

Bookkeeper should have a dedicated disk to journalDir.
See: http://bookkeeper.apache.org/docs/r4.0.0/bookkeeperConfigParams.html

[krumo]

@agarman Thank you for your reply~Yes, for each bookkeeper, I put journal, ledger and index files on different disk devices, but my result is still very poor.

[merlimat]

Hi @krumo, I think the main difference resides in the disk config and also the 1Gbps NIC.

For example with 1Gbps in the broker you're not able to send 100K msg/s at 1KB payload size, because that's already over capacity.

For disk latency, the reason is that in BookKeeper every entry is flushed to disk before an ack is returned. This is to prevent any data loss in case of system crash or power failure.

The downside of it is that on normal HDDs you're seeing high latency, even if you use a dedicated disk for the journal.

As stated in the Cluster Setup :

It is critical to have fast fsync operation on this device for sequential writes. Typically, a small and fast SSDs will be fine, or HDDs with RAID controller and battery backed write cache. Both solutions can reach fsync latency of ~0.4 ms.

So you have 2 options:

1. Use SSD for journal. Latency should be good, even though we have seen different performance numbers for different SSD disks.
2. Use HDD with RAID and write cache. The battery on the RAID controller will make sure that all the data in the write cache gets flushed to disk in case of crashes or power loss.

Finally, we have been using XFS for both journal and ledger devices since it was leading to better latencies (especially at 99pct and on).

Can you also post the bookie internal stats? That should give more insight to understand where the latency is coming from. If you use the default config, bookies should dump the stats every 1min into a bookie-stats.json file.

[krumo]

@merlimat Thank you for your detailed explanation. It's a sad story that my lab only have machines with HDDs. So now I have to test the performance of Pulsar on HDDs. And I will suggest my advisor to buy some SSDs to present the real power of Pulsar.

Attached is my bookie-stats file generated by one of the bookies during the test. Hope it can help to reflect what is the bottleneck of the bookie.
bookie-stats.zip

[sijie]

it seems that the question is answered. so close it now. please reopen it if there are more information provided.