MMR reranking via mmr_lambda hidden column

[MayCXC]

Rebased version of asg017#267 (issue: asg017#266) for this fork. Adds a mmr_lambda hidden column to vec0 for Maximal Marginal Relevance reranking in KNN queries.

SELECT rowid, distance FROM vec_items
WHERE embedding MATCH ? AND k = 10 AND mmr_lambda = 0.5;

Composes with this fork's distance constraints, partition keys, and all vector types/metrics.

Also fixes the pre-existing test_shadow snapshot failure (missing ORDER BY on pragma_table_list).

Full design and rationale in the upstream PR.

[mceachen]

Thanks for this! Claude (and I) just merged it into my fork.

One fix it made: in vec0_mmr_rerank, the copy-back loop at the end iterates k_target times, but the greedy selection loop can terminate early (via if (best_idx < 0) break), leaving the tail of out_rowids/out_distances uninitialized. We added an n_selected counter and an out_n_selected output parameter so only the actually-selected entries are copied back. The caller now sets k_used = n_selected instead of k_used = k_original. (You can see my referenced commit for details)

[vlasky]

Thanks for this contribution! Merged.

I added a follow-up commit (8d4ef9e) to normalize the diversity term in the MMR greedy loop. The relevance term was already normalized to [0,1] by dividing by max_dist, but the inter-candidate similarity used raw distances (1 - d). For cosine distance this is fine since values are already in [0,1], but for L2 and L1 where distances are unbounded, the two terms in the MMR score were on different scales - meaning mmr_lambda didn't have a consistent effect across distance metrics.

The fix divides the inter-candidate distance by max_dist to match, so both terms are on the same scale regardless of metric.

[MayCXC]

@mceachen @vlasky thank you both for the very rapid fixes :)