opt: add GeneratePartialIndexScans rule

[mgartner]

This commit introduces a new exploration rule,
GeneratePartialIndexScans. This rule generates unconstrained Scans over
partial indexes when a Select filter implies the partial index
predicate.

For example, consider the table and query:

CREATE TABLE t (
  i INT,
  s STRING,
  INDEX pidx (i) STORING (s) WHERE s = 'foo'
)

SELECT i FROM t WHERE s = 'foo'

GeneratePartialIndexScans will generate a scan over the partial index:

project
 ├── columns: i:1
 └── scan t@pidx
      └── columns: i:1 s:3!null

It is capable of handling cases where the index does not cover the
selected columns, and where additional filtering is required after the
partial index scan. For example:

CREATE TABLE t (
  i INT,
  s STRING,
  f FLOAT,
  INDEX pidx (i) WHERE s = 'foo'
)

SELECT i FROM t WHERE s = 'foo' AND f = 2.5

In this case, GeneratePartialIndexScans will generate the following
expression with an added Select and IndexJoin:

project
 ├── columns: i:1
 └── index-join t
      ├── columns: i:1 s:2!null f:3!null
      └── select
           ├── columns: i:1 rowid:4!null
           ├── scan t@pidx
           │    └── columns: i:1 rowid:4!null
           └── filters
                └── f:3 = 2.5

Fixes #50232

Release note: None

[cockroach-teamcity]

This change is 

[RaduBerinde]

Nice work!

Reviewable status: complete! 1 of 0 LGTMs obtained (waiting on @mgartner)

---

pkg/sql/opt/xform/custom_funcs.go, line 134 at r2 (raw file):

// GeneratePartialIndexScans generates unconstrained index scans over all
// partial indexes with predicates that are implied by the filters. Partial
// indexes with predicates cannot be proven to be implied by the filters are

[nit] which cannot

---

pkg/sql/opt/xform/custom_funcs.go, line 138 at r2 (raw file):

//
// When a filter completely matches the predicate, the remaining filters are
// simplified so that they do not include the filter. A redundnat filter is

redundant

---

pkg/sql/opt/xform/custom_funcs.go, line 193 at r2 (raw file):

		// If index is covering and there are no remaining filters, construct a
		// new Scan in the same group as the original Select.
		if remainingFilters.ChildCount() == 0 && covering {

[nit] len(remainingFilters)

---

pkg/sql/opt/xform/custom_funcs.go, line 196 at r2 (raw file):

			scan := memo.ScanExpr{ScanPrivate: *scanPrivate}
			scan.Index = iter.IndexOrdinal()
			c.e.mem.AddScanToGroup(&scan, grp)

[nit] Doesn't indexScanBuilder cover this case too?

---

pkg/sql/opt/xform/custom_funcs.go, line 226 at r2 (raw file):

		// If there are remaining filters, construct a Select with them.
		if remainingFilters.ChildCount() != 0 {

We don't need this check (addSelect already checks it)

---

pkg/sql/opt/xform/partial_index.go, line 20 at r2 (raw file):

)

// predicateImpliedByFilters attempts to prove that a partial index predicate is

I think this is accidentally left over

---

pkg/sql/opt/xform/testdata/rules/select, line 223 at r2 (raw file):

      └── filters
           └── NOT b:4 [outer=(4), constraints=(/4: [/false - /false]; tight), fd=()-->(4)]

Maybe add a test where the table has multiple partial indexes (as well as some non-partial indexes).

[rytaft]

Nice! once you address Radu's comments.

Reviewed 5 of 5 files at r1, 6 of 6 files at r2.
Reviewable status: complete! 2 of 0 LGTMs obtained (waiting on @mgartner)

---

pkg/sql/opt/testutils/opttester/opt_tester.go, line 830 at r2 (raw file):

			return false
		}
		return true

While you're making changes here, probably worth adding the ruleName to ot.seenRules like we do in Optimize above

---

pkg/sql/opt/xform/custom_funcs.go, line 144 at r2 (raw file):

// For each partial index, there are two conditions to consider: whether or not
// the index "covers" the columns needed and whether or not the remaining filter
// is empty. There two conditions combine to create four possible cases:

There two -> These two

---

pkg/sql/opt/xform/custom_funcs.go, line 148 at r2 (raw file):

//   - The index is covering and the remaining filter is empty. In this case, A
//     single unconstrained Scan operator is added to the same group as the
//     original Select operator).

[nit] extra ")"

---

pkg/sql/opt/xform/custom_funcs.go, line 191 at r2 (raw file):

		covering := iter.IsCovering()

		// If index is covering and there are no remaining filters, construct a

[nit] It would help to number the 4 cases above and refer to the case numbers in the comments here.

[mgartner]

Reviewable status: complete! 0 of 0 LGTMs obtained (and 2 stale) (waiting on @mgartner and @rytaft)

---

pkg/sql/opt/testutils/opttester/opt_tester.go, line 830 at r2 (raw file):

Previously, rytaft (Rebecca Taft) wrote…

While you're making changes here, probably worth adding the ruleName to ot.seenRules like we do in Optimize above

I've undid these changes for now, because I no longer need them.

---

pkg/sql/opt/xform/custom_funcs.go, line 134 at r2 (raw file):

Previously, RaduBerinde wrote…

[nit] which cannot

Done.

---

pkg/sql/opt/xform/custom_funcs.go, line 191 at r2 (raw file):

Previously, rytaft (Rebecca Taft) wrote…

[nit] It would help to number the 4 cases above and refer to the case numbers in the comments here.

Turns out there's actually 6 cases. I've rewritten this comment to be more clear, and I've rewritten the code to be correct and also less verbose. LMK what you think!

---

pkg/sql/opt/xform/custom_funcs.go, line 196 at r2 (raw file):

Previously, RaduBerinde wrote…

[nit] Doesn't indexScanBuilder cover this case too?

Thanks for the pointers. I've simplified this code, and fixed a bug—it wasn't splitting the filters into 2 Selects when part of the filters must be applied before the IndexJoin and the remaining filters must be applied after.

---

pkg/sql/opt/xform/partial_index.go, line 20 at r2 (raw file):

Previously, RaduBerinde wrote…

I think this is accidentally left over

Oops!

---

pkg/sql/opt/xform/testdata/rules/select, line 223 at r2 (raw file):

Previously, RaduBerinde wrote…

Maybe add a test where the table has multiple partial indexes (as well as some non-partial indexes).

Done.

[rytaft]

Reviewed 5 of 5 files at r3.
Reviewable status: complete! 0 of 0 LGTMs obtained (and 2 stale)

---

pkg/sql/opt/xform/custom_funcs.go, line 191 at r2 (raw file):

Previously, mgartner (Marcus Gartner) wrote…

Turns out there's actually 6 cases. I've rewritten this comment to be more clear, and I've rewritten the code to be correct and also less verbose. LMK what you think!

Looks good!

[mgartner]

bors r+

[craig]

Build failed (retrying...)

• GitHub CI (Cockroach)

[craig]

Build succeeded

• GitHub CI (Cockroach)