msg/simple: apply prefetch policy more precisely

[xiexingguo]

We shall apply prefetch policy based on the residual length aftering
checking cache instead of the original request length.

E.g., if the reading sequences are 1K, 5K, 2K, the improved logic
will trigger another prefetch of 4K(as 5K - 3K(from recv_buf) == 2K, and we
now have 8K prefetched data total) by the second 5K reading(which we
don't do this according to the old logic), and thus the last reading request
which asks for 2K data can be also benefited from this prefetch,
which is good for performance.

Signed-off-by: xie xingguo xie.xingguo@zte.com.cn

[gregsfortytwo]

Doesn't this change contradict the comment right there about not prefetching large reads?

[xiexingguo]

The key difference is how do we define large reads.
Previously we honour the origin length of read only when applying prefetch policy. But here we reconsider the residual length of read after reading from the cache instead.

E.g., suppose the recv_max_prefetch is 4K, and the read sequences are 1K, 5K, 2K

Before this change:

1. read 1K, as 1K < recv_max_prefetch, we prefetch 4K, and the 1K read itself is hit in the cache after the prefetch is done.
2. read 5K, the first 3K is hit in the cache and the cache is now empty, as 5K > recv_max_prefetch, we don't prefetch and trigger a 2K read instead.
3. read 2K, the cache is now empty, as 2K > recv_max_prefetch, we trigger another prefetch and get 2K from the cache after prefetch is done.

After this change:

1. read 1K, as 1K < recv_max_prefetch, we prefetch 4K, and the 1K read itself is hit in the cache after the prefetch is done.
2. read 5K, the first 3K is hit in the cache and the cache is now empty and we have 5K-3K = 2K to read, as 2K < recv_max_prefetch, we prefetch again and get 2K from the cache after prefetch is done, the cache has 2K data remaining.
3. read 2K, which is directly hit in the cache.

From the above example, we need exactly 2 (prefetch)reads now instead of 3 reads which we need before.

[xiexingguo]

@gregsfortytwo Thoughts?

[gregsfortytwo]

Yeah, that makes sense. I was worried about the performance impact of the change but going down a few function frames it all looks good to me!
Reviewed-by:

[yuriw]

test run http://pulpito.ceph.com/yuriw-2016-07-27_11:33:48-rados-wip-yuri-testing_2016_7_27-distro-basic-smithi/
ready for merge
@jdurgin pls merge