Can't scroll when swiper minibuffer has focus

[muirdm]

Steps to reproduce:

1. Have a buffer open with multiple pages of contents
2. Do a swiper search on a middle page
3. Attempt to scroll up in content buffer while swiper minibuffer still has focus (either through C-M-v or sloppy mouse scrolling)

I expect to be able to scroll around, but it doesn't let scroll the current match off screen. If I switch to the content buffer then I am able to scroll around freely with the swiper still active.

[abo-abo]

Thanks.