It seems that the annotation position is computed in data coordinates before xlims (and presumably ylims) are applied, so if you specify a location as :bottomright etc. the annotation may be off-plot.
This works:
plot(rand(20), ann=[(:bottomright, "foobar")], xlims=(0,20))
But this has the annotation going offscreen:
plot(rand(20), ann=[(:bottomright, "foobar")], xlims=(0,18))