rgw_file: fix write error when the write offset overlaps

[yaozongyou]

fix bug in http://tracker.ceph.com/issues/21455

Signed-off-by: Yao Zongyou yaozongyou@vip.qq.com

[yaozongyou]

The nfs client writes data to system's page cache, and the page cache will be flushed to rgw_file by nfs-ganesha. The page cache is page size aligned, so the write offset may overlap in a page size.

[yaozongyou]

@tchaikov, would you please have a look at this?

[tchaikov]

@yaozongyou i am not familiar with rgw, thus not qualified to review your change.

[mattbenjamin]

@yaozongyou does this change attempt to address a behavior seen in Linux clients mounting -osync?

[yaozongyou]

yes, as above image shows, even mounted using -osync, the offset may overlap between two requests.

[mattbenjamin]

@yaozongyou do you have a simple reproducer from the client side?

[mattbenjamin]

of course, the client's pagesize (like its mount options) is not knowable, right? if we unconditionally attempt this correction, do we do just as well?

[yaozongyou]

@mattbenjamin a simple reproducer from the client side:
https://gist.github.com/yaozongyou/5f5ed0d362ccef97942a09496f4e2a8c

[yaozongyou]

@mattbenjamin can you reproduce it in your environment?

[mattbenjamin]

@yaozongyou I'll attempt it--if my results are consistent, this should be ok (with the possible caveat about pagesize flagged above)

[mattbenjamin]

@yaozongyou this doesn't error in my environment :(

[mattbenjamin]

@yaozongyou so, we've discussed this here a bit; it's known that there may be kernels/clients that don't conform to our expectation that WRITEs will be strictly sequential even when a) the mount is -osync and b) the application i/o is strictly sequential--but the kernels we've explicitly tested do so. I'd be interested to know which distro and kernel version you're doing this, on what architecture. My inclination is not to apply this or previous suggestions like this, but rather to continue with our work in progress which will allow async and random i/o. Please note that iiuc, your posix write calls are not guaranteed to write all bytes in one invocation--you need to be parepared for short writes. In any case, I will expedite getting that finished up, becausel clearly it is an issue for users.

Matt

[yaozongyou]

@mattbenjamin We are using CentOS Linux release 7.3.1611, the kernel version is 3.10.0-514.el7.x86_64.
We have a script using the system command ftp downloading files from remote ftp server directly into local mounted rgw buckets. The ftp command gives error. After stracing, we make a simple reproducer as above.
We mount using mount -t nfs -o sync 127.0.0.1:/ /mnt and mount -t nfs -osync,vers=4.1 127.0.0.1:/ /mnt. Both of them didn't work.
In /etc/ganesha/ganesha.conf , the Path is the bucket name. the command showmount -e shows as follows:

Export list for localhost:
testbucket (everyone)

[mattbenjamin]

Based on the evidence presented, I want to merge this change, but I am concerned about the use of sysconf(_pagesize). It seems that this conditional clause an just be removed without risk?

[yaozongyou]

@mattbenjamin I have made some changes about the problem you mentioned. And I have tested in our environment, it works very well.

[mattbenjamin]

per discussion, this change has been motivated and appears necessary given its ability to give a correct result for these observed cases, until full support for write at any offset and order is merged

I have hand-verified this change