perf: Optimize array_position for scalar needle

[neilconway]

Which issue does this PR close?

• Closes Improve performance of array_position with scalar needle #20530

Rationale for this change

The previous implementation of array_position used compare_element_to_list for every input row. When the needle is a scalar (quite common), we can do much better by searching over the entire flat haystack values array with a single call to arrow_ord::cmp::not_distinct. We can then iterate over the resulting set bits to determine per-row results.

This is ~5-10x faster than the previous implementation for typical inputs.

What changes are included in this PR?

• Implement new fast path for array_position with scalar needle
• Improve docs for array_position
• Don't use internal_err to report a user-visible error

Are these changes tested?

Yes, and benchmarked. Additional tests added in a separate PR (#20531)

Are there any user-facing changes?

No.

[neilconway]

Benchmarks:

  ┌────────────────┬─────────┬─────────┬─────────────┐
  │   Benchmark    │ Before  │  After  │ Improvement │
  ├────────────────┼─────────┼─────────┼─────────────┤
  │ found_once/10  │ 1.73 ms │ 55.6 µs │ -96.6%      │
  ├────────────────┼─────────┼─────────┼─────────────┤
  │ found_many/10  │ 1.61 ms │ 68.5 µs │ -95.5%      │
  ├────────────────┼─────────┼─────────┼─────────────┤
  │ not_found/10   │ 1.80 ms │ 47.0 µs │ -97.3%      │
  ├────────────────┼─────────┼─────────┼─────────────┤
  │ found_once/100 │ 1.90 ms │ 156 µs  │ -91.2%      │
  ├────────────────┼─────────┼─────────┼─────────────┤
  │ found_many/100 │ 2.05 ms │ 459 µs  │ -76.5%      │
  ├────────────────┼─────────┼─────────┼─────────────┤
  │ not_found/100  │ 2.13 ms │ 144 µs  │ -93.2%      │
  ├────────────────┼─────────┼─────────┼─────────────┤
  │ found_once/500 │ 2.39 ms │ 586 µs  │ -75.4%      │
  ├────────────────┼─────────┼─────────┼─────────────┤
  │ found_many/500 │ 2.68 ms │ 2.34 ms │ -12.8%      │
  ├────────────────┼─────────┼─────────┼─────────────┤
  │ not_found/500  │ 4.12 ms │ 585 µs  │ -85.8%      │
  └────────────────┴─────────┴─────────┴─────────────┘

found_many/500 is not surprising as a worst-case: the bitmap we get back from arrow_ord::cmp::not_distinct is dense, so we aren't able to save much work. But at least it's still a win.

[neilconway]

Note that this PR includes the commit from #20531; happy to remove that or we can land that change as part of this PR and close #20531, whatever folks prefer.