Better cardinality estimation for queries with many predicates (especially those not matched to any indices)

[dyemanov]

• Refactor selectivity estimations
• Apply exponential backoff to the selectivity combined from multiple booleans

In real-world SQL queries, conjuncts (ANDed booleans) are often inter-dependent and simple multiplication of selectivities (that we used so far) results to a very low final selectivity value, thus causing the stream cardinality being under-estimated. To avoid this, apply exponential backoff adjustment:

sel = sel1 * sqrt(sel2) * sqrt(sqrt(sel3)) * ... where sel1 is the least (best) selectivity and selN is the biggest (worst) one

I don't pretend this is the best appoach possible, but it seems working fine for MSSQL, so it's worth trying (actually, it was already tested in production -- with quite good results so far).

[mrotteveel]

@dyemanov For clarification, where you say boolean, do you mean predicate? Or are you literally talking about columns of type BOOLEAN?

[dyemanov]

Of course, I meant predicates. Inside the code, you may find conjunct or boolean meaning a predicate.

[mrotteveel]

@dyemanov Another question, the title talks about cardinality, while the body talks about selectivity. Although those are somewhat related, they are different things. Looking at the code, I think this is actually about selectivity, right?

[mrotteveel]

And the mention of "low selectivity" vs "cardinality under-estimated" throws me off, because I assume a low selectivity implies an over-estimated cardinality, not an under-estimated cardinality.

[dyemanov]

@dyemanov Another question, the title talks about cardinality, while the body talks about selectivity. Although those are somewhat related, they are different things. Looking at the code, I think this is actually about selectivity, right?

Internally, fix is about selectivity. But from the user POV, only the cardinality difference is visible. I.e. changed selectivity affects cardinality, changed cardinality affects cost.

And the mention of "low selectivity" vs "cardinality under-estimated" throws me off, because I assume a low selectivity implies an over-estimated cardinality, not an under-estimated cardinality.

Lower selectivity means smaller estimated cardinality, thus under-estimation.

0.1 * 1000 = 100
0.001 (lower selectivity) * 1000 = 1

I think the confusion is about what selective means. Very selective means more unique values and in turn implies lower selectivity value which is calculated as 1 / <number of unique values>.

[mrotteveel]

OK, clear. Now I need to think how to phrase that, because in English, low selectivity is the opposite of what in Firebird code is a "low selectivity value", so the release notes need to be understandable with the general meaning in mind, and the meaning in the context of Firebird code for those that think in its internals.