doc: document sizing for block.db

[alfredodeza]

This expands a lot into what is needed to understand to properly size a block.db device, depending on different cluster workloads, and also simplifying the most basic use cases, like provisioning using only spinning drives.

It also adds a bit of LVM commands that can be used with ceph-volume that might help until the batch sub-command is ready.

/cc @bengland2

[alfredodeza]

jenkins render docs

[liewegas]

One possible formula to choosing a size for block.db includes the following items:

[liewegas]

At a very high level, one can consider three distinct workloads:

[liewegas]

Note that all of the figures below are very much approximations based on one person's experience and are unlikely to match your workload precisely.

[bengland2]

True, but it's better than 1 GB!!! Also, I think it should mention that both Cephfs and RGW chop up large files/objects into 4 MB chunks, so if your average file/object size is >> 4 MB, then 4 MB is a good estimate for RADOS object size, which makes the arithmetic simpler.

For existing (Filestore) clusters, we can just ask Ceph what the number of objects and the amount of space used is, and calculate average object size from that.

Finally, for Cephfs, the average metadata per object might increase to be more like RBD for files accessed with random writes >= bluestore_min_alloc_size, because Bluestore will then have many more extents to deal with, right? For example, database files. But for sequential access cases, the metadata per object should be low, more like RGW, correct?

@markhpc ?

[liewegas]

Anything is better than 1 GB :) I don't think I'm disagreeing. I'm just generally uncomfortable with this entire section.. I don't think users should be thinking about any of this, even if they are expert users. Instead, we should suggest that they use as much SSD as they have. Maybe with a ceiling of something like 30%... or maybe not. But I think the simple message of "more is better" is probably enough?

[liewegas]

In fact, even the RGW case is misleading, because for RGW index pools the block.db usage is unbounded (could be the entire data set!).

I think the only reason not to do "more is better" is that we don't currently put data on the SSD. But we may very well do that in the future (case in point, ttps://github.com//pull/23208). And even if we leaned on dm-cache to do it, that detail would be hidden by ceph-volume ("hi ceph-volume, here is my SSD and HDD, do the right thing").

[alfredodeza]

Part of the problem here is that "as big as possible" leaves a lot of room for interpretation and can cause doubt about what is correct vs. what may be problematic. I agree that these formulas are still hard to digest, but they do give us some pointers.

If what we want is to prevent users from figuring out some number (while causing confusion) at one end, and at the other end just being naive with ceph-volume and creating as-large-as-possible LVs, could we perhaps recommend a minimum size?

Given these examples, the largest block.db is the RGW example using close to 4% of the block device size. Wouldn't that be a reasonable default? Something like:

It is recommended that the block.db size be 4% of the size of the block device. For a 1TB block, that would be a 40GB block.db LV. Anything less than 4% may incur in a performance penalty.

We would still prevent the performance penalty, we would have something pretty clear, and we wouldn't confuse users with formulas and average metadata/object sizes

[liewegas]

Well, it will never really cause a performance penalty, since the SSD is always strictly faster than teh HDD. I think 1-2GB is probably the real minimum as that's what required to get the wal on the SSD (for fast metadata commits). In practice, what really happens is that performance is good while the OSD is mostly empty (all the metadata fits on the SSD, yay!) and then things slow down once it fills up enough to not fit. But it's still (a lot) better than no SSD at all.

[leseb]

Taking the entire device and split it into equal parts for block.db can be tricky. Perhaps having guidance/formula just as suggested is a useful approach. I assume we have warnings in Ceph to tell: "hey osd.0 block.db is almost running out of space, please expand the block" or "hey osd.0 block.db is running out of space, we now write metadata on the block drive". Since we do everything with LVM it should be trivial to increase the size of the block, or is it something impossible and the initial size is set in stone? What I'm afraid about the "as big as possible" approach is that users might just divide the SSD into equal parts. This has pros and cons:

• Pros: we might never run out of space and thus fewer things to care on the operator side
• Cons: we can not scale our OSDs (disk on the machine) without adding a new SS

Also, the block.db fillup rate might differ depending on which application is writing to the OSD (rgw/rbd/cephfs)

We (hopefully) have a good set of tools to monitor the status of each block.db/wal at any given time so operators can react quickly.

My 2 cents.

[alfredodeza]

@leseb we can expand block.db with LVM, but once data has spilled over to the slower device it isn't trivial to move it back (this is my understanding from talking to @bengland2 ). So monitoring would only allow you to destroy it and reprovision the OSD (!)

My concern is also the scaling of OSDs, the batch subcommand could (by default) create block.db as big as possible but would be nice if we could also carve out the size from the config option in ceph.conf that ceph-disk used.

My two issues are that:

1. There is no clear guidance to what is needed ("as big as possible" works for some users, doesn't give a better resource allocation idea for advanced users)

2. We can't really say "2GB" because this is all tied to the size of the block size. A 10TB disk would see the spill over problem faster (without filling up the OSD).

I'm fine if we want to do without the formula, but would love to find some agreement on a recommendation on size (in addition to "as large as possible" which I find unclear without a path for expansion)

[liewegas]

My problem is that "need" and "problem" are the wrong words here. You don't "need" an SSD to be any particular size; it's just a question of how much it will help. Similarly, the spill-over effect isn't really a "problem"--it's just that the SSD didn't provide as much benefit as it could have.

The 2GB minimum is independent of the block device... it's based on how big the rocksdb wal files are (currently 512MB, and we need 2, plus some room or slop and for the bluefs metadata journal). This is almost entirely independent of the main device's size. Basically anything bigger than that is better, and how much better depends on how big.

"Big enough" is so wildly dependent on workload and data set I don't know that we can give useful guidance. I think that, like the idea that you might want ot add another OSD to the host, this is really a hardware question: how many slots are in my box, and what is the ratio of HDD:SSD I'm willing to pay for? If you have 2 empty slots with no HDDs, then we could leave some space on the SSD for those. (This seems like a useful feature of the batch-prepare but not required for an MVP.)

It feels to me like the guidance how to pick that ratio (or go all-SSD) is a higher level recommendation. "For RGW clusters, for best performance we recommend a dedicated pool of SSDs for bucket indexes. In real-world deployments the capacity ratio between objects and bucket metadata appears to range between 1:100 and 1:200" (I'm making that up, not sure what the real number is, but we can easily get a ballpark from current deployments.)

"For block workloads, best performance comes from all SSD pools. For hybrid pools that mix HDDs and SSDs, an SSD:HDD ratio between 1:5 and 1:8 seems to work well." That isn't quite right for NVMe, though, because it's more about capacity, and NVMes are getting big. So maybe 1 NVMe for 12 OSDs is okay. Which means the probably bigger consideration is around reliability and the blast radius of an NVMe failure.

Regardless, I'm still skeptical that we should (1) get into the weeds about block.db size, and if we do, (2) be so prescriptive when the workload descriptions are so imprecise and (3) bluestore behavior may change in the future.

[wjwithagen]

What everybody seems to forget is that SSDs have something called "lifetime". It is eitehr expressed in Writes/per day or something... (like: Drive Writes Per Day (DWPD) ) But it boils down to that you can only write a cell so may times.
Now if you overdimension the SSD, the internal software in the SSD controller will balance the writes over all cells. Partitioning has nothing to do with it, under water the SSD does writes where it pleases.

So in effect if you have a 120GB SSD, but only "use" 40Gb from the disk-side of things, you have extended the SSDs lifetime by a factor of 3.... So overdimentioning is not really a waste.... It just extends lifetime. We have servers where we run thru SSDs every 1,5 year. Going to the double size has make them last almost 3 years.

So then it becomes a question of servicability <> pricing <> reliability.

[ceph-jenkins]

Doc render available at http://docs.ceph.com/ceph-prs/23210/

[bengland2]

We can simplify the RocksDB volume size formula - allocate space proportional to the size of the slow device (HDD). If done right, this will cover 90% of the use cases, and the user can tune this ratio for the remaining 10%.

I absolutely agree that users shouldn't be thinking about this. But the difficulty is that there is no ability to adjust space usage on the fly to unexpected user workloads, and people often don't know in advance what they are going to do with a cluster. Reformatting OSDs is expensive and risky in production systems. And RocksDB spillover causes a severe latency penalty to Bluestore, removing all the advantages that it has over Filestore, as John Harrigan's data shows.

dm-cache IMHO is an idea whose time has come... and gone. Either we use collocated OSDs and give ALL the NVM space to dm-cache, or why bother?

• When RocksDB and WAL are on SSD, some of the benefit of dm-cache (or iCAS) is lost.
• Unless dm-cache has SSD space >> OSD cache size, the OSD cache will intercept most reads before they reach dm-cache (exception: immediately after OSD restart).
• dm-cache cannot accelerate writes.
• for block storage, client-side caching makes more sense
• for RGW data lakes, RGW caching closer to the clients makes more sense.

As for PR 23208, that's a head-scratcher - I don't know how you plan for that, or how you prioritize metadata over data. Is there some kind of background activity to move data from RocksDB in SSD to HDD? Am going to ignore for now.

The problem with "use as much as you have" is that there are 2 kinds of use for NVM devices:

• Bluestore rocksDB and WAL for OSDs that are HDD-resident
• NVM-backed OSDs

I also agree that we are increasingly relying on all-flash storage pools, or should be doing so. Examples include RGW bucket index, Gnocchi "metrics" pool, and cephfs_metadata pool.

If you have a 2-TB NVM SSD device, it's generally a terrible waste of space to just use it for RocksDB and WAL, according to the calculations that we mostly all agree with (@nwl first brought this to my attention). For example, even with RGW with 12 HDDs and 40 GB RocksDB space per OSD, we only use 480 GB, or 1/4 of a 2-TB device. These are still relatively expensive devices, and there are plenty of candidate storage pools that really benefit from high IOPS, low latency, but use relatively little storage space.

John Harrigan ran into this problem - he had 2 NVM devices/host, but ceph-ansible only supports using 1 of them as a Filestore journal device and the other as an OSD device (for bucket index pool). This artificial and unnecessary restriction kills performance on writes since the journal NVM device becomes oversubscribed with 24 OSDs on it, while the other NVM device was mostly idle.

Proposal: generously allocate space to RocksDB, ~10 GB/slow-TB volume size for HDD-backed OSDs, in order to prevent RocksDB spillover, and then create 1-2 OSDs on the remaining space (default 1 if RocksDB partitions present, default 2 if none), if there is > 30% space left over. This gets the most out of your NVM devices, and if you have > 1 NVM device/host (between 2 and 4 is not unreasonable), it allows you to evenly spread load across them, regardless of what the workload is at that time. And it allows you to scale up the number of OSDs/host farther than you could otherwise.

RBD: 12 KB/object x 250K 4-MB objects/TB ~= 3 GB metadata space used
RGW: 10 GB / 3 KB/object = 10/3 million objects, and
1 TB/(10/3) million obj ~= 330 KB average object size
Cephfs would probably be more like RGW in metadata/object.

If you need to fill your HDDs with objects/files smaller than 330 KB, then the site would need to increase RocksDB-GB/slow-TB.

Example: with 6-TB HDDs, you would allocate 60 GB/OSD for RocksDB. With 12 OSDs/NVM device, you would use ~720 GB of space for RocksDB, but ~2/3 of your 2-TB NVM device is usable as an OSD.

background: bzs 1591074, 1583839, 1591791

[liewegas]

Hrm, we didn't consider the case where a single nvme would be shared across hybrid osds and also a standalone nvme ssd. :/

I wonder, though, if the hybrid OSDs end up with a ton of nvme block.db space, if the dedicated index pool could be skipped? Because the index metadata (omap) would all go on the ssds on the hybrid osds anyway. Feels fragile, though, since you can't easily scale the amount of metadata space or independently scale performance.

[ifed01]

Yet another aspect of DB sizing that probably should be covered here is the fact that RocksDB allocates its data depending on the level granularity they belong to. And for each level RocksDB has a maximum size calculated using some basis value and level multiplier. If underlying fast storage isn't sufficient to keep that maximum for the given level - all data for that level are spilled over to slow device.
With default settings max level sizes are something like:
level 1 = 250MB, level2 = 2500MB, level 3 = 25000MB, etc.
Hence IMO there is no much sense to allocate DB device in arbitrary sizes, e.g. 10GB - levels < 3 wouldn't completely use it while level 3+ wouldn't land there at all.

Still an open question is what are the excesses RocksDB can admit for each level which definitely exists Still investigating that but it looks like up to ~100% excess is possible...

[bengland2]

@ifed01 , thanks for pointing out that levels are either all in SSD or all in HDD, I did not know that. Can these 3 parameters be aligned with one another? For example if the GB_per_Slow_TB parameter was exposed to Ceph, could it create RocksDB with the basis level and multiplier such that up to N levels fit within this amount of space on the volume? If these are input parameters to RocksDB, then it seems straightforward to do this. Please let's all consider that mere mortals have to plan these clusters, and they don't know anything about RocksDB, nor should they have to.

[liewegas]

Yeah, I don't think we should plan teh block.db size based on the rocksdb stairstep pattern. A better solution would be to tweak the rocksdb level sizes at mkfs time based on the block.db size!

Again, I think we're better off having hte user think about the high-level planning and not tuning based on the internal behaviors (that may change in the future, possibly without notice).

My suggestion is to axe this whole block.db section and instead say something like "bigger is better". Then, point to a new section that is a higher-level "hardware planning" guide that helps users think about the many issues contirbuting to what hardware to buy. All SSD? All HDD? Hybrid? What ratio? How do you factor in the key considerations: (1) size of SSD, (2) life expectancy of SSD (how many HDDs' write streams do you point at it?), (3) reliability due to blast radius of a failed SSD shared by many OSDs, (4) share the SSD to make hybrid OSDs or do as separate class of all-SSD OSDs, (5) how many slots are in your chassis, etc. This would inform teh high-level hardware and planning, the output of which is the drive groups, which ceph-volume then consumes to Do The Right Thing...

[jdurgin]

Is there any reason to use a wal partition instead of a db partition if you only have 2 devices (even if only a wal will fit on the faster one)?

That would simplify this section a bit - we could mention separate wal partitions at the end, only for a case where you have an even faster 3rd device

[alfredodeza]

I don't think there is a reason, no. I agree pushing them to the end, since we are concentrating in block.db here

[jdurgin]

I agree a higher level planning guide would be very helpful. Updating the ancient http://docs.ceph.com/docs/master/start/hardware-recommendations/ page would be a good start - it references 2012 as a 'recent' deployment.

I think for existing hardware or clusters that are converting to bluestore, it would help to have some simplistic guidelines here. Rather than guessing based on the workload, we could provide a simple rule that would be safe for most cases (for more optimal design you really need to look at the whole server and workload which gets too complex and uncertain quickly).

I like @alfredodeza's suggestion of a minimum recommended block.db size, e.g. 4% of the data device size. To account for the blast radius of a failure and limit write streams/cpu requirements, we could additionally recommend a maximum number of partitions per fast device, e.g. 6 block.dbs / SSD.

So the simple rule for this page could be:
For a hybrid flash/HDD setup, partition your SSD/NVMe device into at most 6 block.db partitions, each with a minimum size of 4% of the HDD.

This results in using ~1TB flash for 6x 4TB drives, ~2TB for 6x 8TB drives, etc.

More complex scenarios like adding pure-NVMe OSDs on the same devices could be left for the higher-level planning guide, since they'll need to take CPU and workload into account more.

[liewegas]

So, to move this conversation along.. can we merge this PR without the block.db sizing section for now, and follow up with a discussion about how to address documenting the high-level hardware selection?

[bengland2]

I like Josh's suggestion for now, but if we see that RocksDB partitions are all very empty, we may need to tweak the percentage.

One alternative, and I think Federico Lucifredi suggested this, is to let the various downstream vendors that use Ceph work it out and define it in their own documentation, if we can't reach consensus here. This seems inefficient but perhaps different vendors will experiment with different policies and we'll see which one works best. Opinions?

[alfredodeza]

I am fine with nixing most of the RocksDB section, but would at least like some percentage being mentioned as a reference. Can we agree on leaving that suggestion and nixing the rest?

[liewegas]

Sure

[alfredodeza]

@liewegas updated, nixing the formulas, is the wording better now? or do you have anything else I need to address

[bengland2]

@alfredodeza IMHO the commits above are way better than before, we can quibble about what the exact right percentage is and people can experiment, but I think most people will be much happier following these guidelines than using the current defaults, and hopefully Ceph installers can start implementing these guidelines as defaults. The decreasing cost and increasing size of NVM SSD takes the pressure off of us to be exactly right and ultra-efficient.

[liewegas]

Looks great, thanks!