It appears to be caused by that ivy places the cursor at the end of current match in buffer. When query-replace starts, it can only "look forward" for the match to replace. This has the consequence that swiper-query-replace is never able to replace the first match (or the only match) in buffer.
Verified with MELPA version 20150610.638 in Emacs 24.5.
swiper.el (swiper-query-replace): Don't miss the first
* swiper.el (swiper-query-replace): Since the point is always after the
matching thing in swiper, it's necessary to move it before it in order
for `perform-replace' not to skip it.