File cache either returns too much data or fails with error when reading until EOF

[illingwo]

If you try to read the entire file until the end of file with the file cache then it can either fail with an error, or silently return more data than is in the source file:

[illingwo@fermicloud058 ~]$ export XROOTD_VMP=mwt2-stashcache.campuscluster.illinois.edu:/stash=
[illingwo@fermicloud058 ~]$ export LD_PRELOAD=/usr/lib64/libXrdPosixPreload.so
[illingwo@fermicloud058 ~]$ ls -l /stash/pnfs/fnal.gov/usr/nova/data/flux/gsimple/nova_v06/nd/mn/rhc/gsimple_NOvA-ND_mn000z-200i_rp11_bs1.1_pnut_lowth_f11f093bbird_target_q72_00001.root
-r-------- 1 illingwo e740 346178381 Apr  9 19:16 /stash/pnfs/fnal.gov/usr/nova/data/flux/gsimple/nova_v06/nd/mn/rhc/gsimple_NOvA-ND_mn000z-200i_rp11_bs1.1_pnut_lowth_f11f093bbird_target_q72_00001.root
[illingwo@fermicloud058 ~]$ XRD_LOGLEVEL=Error cat /stash/pnfs/fnal.gov/usr/nova/data/flux/gsimple/nova_v06/nd/mn/rhc/gsimple_NOvA-ND_mn000z-200i_rp11_bs1.1_pnut_lowth_f11f093bbird_target_q72_00001.root | wc -c
346195610

or

$ XRD_LOGLEVEL=Error wc /stash/pnfs/fnal.gov/usr/nova/data/flux/gsimple/nova_v06/nd/mn/rhc/gsimple_NOvA-ND_mn000z-200i_rp11_bs1.1_pnut_lowth_f11f093bbird_target_q52_00001.root
[2015-07-06 14:47:30.029650 -0500][Error  ][XRootD            ] [mwt2-stashcache.campuscluster.illinois.edu:1094] Overflow data while reading response to kXR_read (handle: 0x00000000, offset: 346235956, size: 16384): expected: 16384, got 17146 bytes
[2015-07-06 14:47:30.029839 -0500][Error  ][XRootD            ] [mwt2-stashcache.campuscluster.illinois.edu:1094] Handling response to kXR_read (handle: 0x00000000, offset: 346235956, size: 16384): user supplied buffer is too small for the received data.
[2015-07-06 14:47:30.029875 -0500][Error  ][File              ] [0x1b68af0@root://mwt2-stashcache.campuscluster.illinois.edu:1094//pnfs/fnal.gov/usr/nova/data/flux/gsimple/nova_v06/nd/mn/rhc/gsimple_NOvA-ND_mn000z-200i_rp11_bs1.1_pnut_lowth_f11f093bbird_target_q52_00001.root] Fatal file state error. Message kXR_read (handle: 0x00000000, offset: 346235956, size: 16384) returned with [ERROR] Invalid response
wc: /stash/pnfs/fnal.gov/usr/nova/data/flux/gsimple/nova_v06/nd/mn/rhc/gsimple_NOvA-ND_mn000z-200i_rp11_bs1.1_pnut_lowth_f11f093bbird_target_q52_00001.root: Bad message
  1211905   7186282 346235956 /stash/pnfs/fnal.gov/usr/nova/data/flux/gsimple/nova_v06/nd/mn/rhc/gsimple_NOvA-ND_mn000z-200i_rp11_bs1.1_pnut_lowth_f11f093bbird_target_q52_00001.root

In XrdFileCache/XrdFileCacheIOEntireFile.cc IOEntireFile::Read(), if the call to m_prefetch->Read(buff, off, size); returns less data than size because it's at the end of the file, then the number of bytes actually read is added to bytes_read twice, the first time at line 97, then the second time at line 106.

The corresponding function in XrdFileCache/XrdFileCacheIOFileBlock.cc also appears to be incorrect in this situation, although in a different way: if the actual bytes read doesn't equal the requested read size it returns the size of the final read, regardless of any data it may have previously put into the buffer from previous calls to read.

[alja]

@illingwo

Thanks for the report. I see the problem in IOEntireFile::Read().

From the report it is not clear to me how did you test file caching plugin by exporting only LD_PRELOAD and XROOTD_VMP. As I understand LD_PRELOAD=/usr/lib64/libXrdPosixPreload.so enables posix like interface, but does enables file caching.

[illingwo]

mwt2-stashcache.campuscluster.illinois.edu is a server running with the file caching plugin.