Hybrid Pool questions

[2fst4u]

Not really a Q&A but just more general curiosity.

In the docs it is stated:

WARNING Hybrid storage pools are likely to suffer from lower availability if a node goes down. The data across the two tiers may actually end up on the same node, instead of being spread across unique nodes (or failure domains) as expected. 

Can this be elaborated on? Why is it possible for copies to end up on the same node or failure domain if you specifically tell it not to? Shouldn't that be built into the logic of creating a hybrid pool so this doesn't happen?

Second question I have is about capacities. If you use a hybrid pool, do your different storage class devices have to have the same storage size essentially for it to work? You can't have small fast devices for caching and larger slow storage for deep storage?

Third question, I've read that with a regular pool, writes are as slow as the slowest of the replicas to be written. In a hybrid pool is this also true? Does that mean they're always as slow as the secondary tier?

Seems like hybrid pools have a lot of potential that isn't used yet.

[travisn]

For more context on the challenges with hybrid pools I'd recommend reading through #9298.

[2fst4u]

I have read through there and there is some good information, but I don't think it fully rounds out the answers to the questions I'm looking for.

I see in there it was discovered that two copies can be on the same host, but my question there is how can that happen if the failure domain is host? Shouldn't that not be possible and is it possible to fix this issue in future?

There are a couple of other parts in my initial question that would be good to get some info on.

[travisn]

It's possible to happen or not happen, depending on the topology. Let's say you have six OSDs, three for each of hdd and ssd. If they are all on six unique nodes, you would never have a problem. But let's say the topology is with two on each node:

node-a: 
  osd.0: ssd
  osd.1: hdd
node-b: 
  osd.0: ssd
  osd.1: hdd
node-c: 
  osd.0: ssd
  osd.1: hdd  

With the hybrid pool, one replica would be placed on an ssd, and the other replica on an hdd. IIRC from #9298, there is no awareness of placing the replicas on different nodes since the different devices classes use different crush trees internally. Not sure about the other questions

[2fst4u]

Interesting. Is it a rook specific implementation for them to be on different CRUSH trees? I'm not too familiar with ceph's inner workings so I struggle to separate ceph from rook implementation of ceph.

[travisn]

Everything about CRUSH belongs to Ceph, and Rook just tries to hide the complexity of CRUSH and other details by creating the rules for you. This topic may help explain the device classes related to CRUSH.