Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Many errors, corrupted filenames with external data introduced in cvmfs-2.11.0 #3418

Closed
DrDaveD opened this issue Oct 16, 2023 · 0 comments · Fixed by #3419
Closed

Many errors, corrupted filenames with external data introduced in cvmfs-2.11.0 #3418

DrDaveD opened this issue Oct 16, 2023 · 0 comments · Fixed by #3419
Assignees
Milestone

Comments

@DrDaveD
Copy link
Contributor

DrDaveD commented Oct 16, 2023

So far I have only been able to reproduce this at Fermilab, not at CERN, but I can quite reliably reproduce it in cvmfs-2.11.0 and 2.11.1 in less than a minute with 5 parallel processes reading from different files in one osgstorage.org repository hosted at Fermilab. /var/log/messages shows junk binary data at the end of the filename in the error message, for example:

Oct 16 16:39:38 dwdosgdev cvmfs2: (uboone.osgstorage.org) failed to fetch /pnfs/fnal.gov/usr/uboone/persistent/stash/uboonebeam/beam_data/bnb_data/00/00/97/19/beamdata_00009719.rootd���#017�#013W#011 (hash: d51f473dec6608c999a8ab9b7352008bbdf9c4bd, error 9 [host returned HTTP error])

Debug logs show that it's not just that error message, that the file name is corrupted in operations before that. This is an excerpt showing the file name transitioning from OK to corrupted:

(catalog) found entry '/pnfs/fnal.gov/usr/uboone/persistent/stash/uboonebeam/beam_data/bnb_data/00/00/97/19/beamdata_00009719.root' in catalog '/pnfs/fnal.gov/usr/uboone/persistent/stash/uboonebeam/beam_data'    [10-16-2023 16:39:33 CDT]
(lru) insert md5 --> dirent: 8f613e69a813b4d3c6e69ce618edcb1c -> 'beamdata_0000919.root'    [10-16-2023 16:39:33 CDT]
(cvmfs) cvmfs_open on inode: 124280    [10-16-2023 16:39:33 CDT]
(lru) lookup inode --> path: 124280 (hit)    [10-16-2023 16:39:33 CDT]
(lru) lookup inode --> dirent: 124280 (hit)    [10-16-2023 16:39:33 CDT]
(cache) miss ./d5/1f473dec6608c999a8ab9b7352008bbdf9c4bd (-2)    [10-16-2023 16:39:33 CDT]
(cache) miss ./d5/1f473dec6608c999a8ab9b7352008bbdf9c4bd (-2)    [10-16-2023 16:39:33 CDT]
(cache) downloading /pnfs/fnal.gov/usr/uboone/persistent/stash/uboonebeam/beam_data/bnb_data/00/00/97/19/beamdata_00009719.rootd<93><FF><FF>^O<D3>^KW       [10-16-2023 16:39:33 CDT]

I can reproduce the problem in the July 12 nightly build but not July 8, so I'm quite confident that the problem was introduced in #3263. CVMFS_STREAMING_CACHE is not set.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants