I went with the first idea. Now ivy-restrict-to-matches works for things like counsel-rg.
But still doesn't work for swiper-isearch. The reason is this code:
(ivy--filter ivy-text ivy--all-candidates)
Usually, ivy--all-candidates is a list of strings, and ivy--filter knows how to filter those.
But for swiper-isearch, for performance purposes, ivy--all-candidates is a list of points. So then we need to either extend ivy-read with a way to customize :filter, or maybe hack swiper-isearch candidates to become strings.
The first is preferable, maybe we can leverage that to gain better performance. Right now, for each new input, we search through the whole buffer once more. But with :filter, we could for each candidate: