`counsel-switch-buffer' leaves Ivy in broken state when unable to preview virtual buffer

[d125q]

1. There is a virtual buffer pointing to a file exceeding large-file-warning-threshold.
2. Invoke counsel-switch-buffer, and navigate to aforementioned virtual buffer.
3. When asked whether to open the file since it exceeds large-file-warning-threshold, answer No.
4. Ivy is left in a broken state.

Similarly Ivy's state gets garbled if a recursive minibuffer is opened at the time a virtual buffer is previewed, e.g., due to lsp-mode asking about the project's root.

[abo-abo]

Thanks, please test.