cvmfs server data chunk existence check is very slow

[DrDaveD]

As I mentioned in a #3553 comment, doing cvmfs_server check without -c is far slower than with -c; in fact with adding a complete scrub with -i it is 7.5 times slower, and it's probably a much worse multiplier without adding -i, although I haven't tried it. That measurement was with cvmfs-server-2.10.1, which included #3167 which made a huge difference by avoiding re-checking files that had already been checked.

I know that using http to read locally hosted files makes for a more flexible and elegant design, but given that the performance hit is so huge, could we have an optimization to directly check files when they're available, bypassing using the web server? I imagine that it would also help at least garbage collection.

[DrDaveD]

Jakob suggested in today's developer call to try using a file:// url. I tried setting

CVMFS_STRATUM1=file:///cvmfs/config-egi.egi.eu

in /etc/cvmfs/repositories.d/config-egi.egi.eu/server.conf but unfortunately then a snapshot failed:

CernVM-FS: replicating from http://cvmfs-stratum0.gridpp.rl.ac.uk:8000/cvmfs/config-egi.egi.eu
CernVM-FS: using public key(s) /etc/cvmfs/keys/egi.eu.pub
HTTP transfer error 7 (HTTP code -1): file:///cvmfs/config-egi.egi.eu/.cvmfsreflog
failed loading reflog (3 - network failure)
cvmfs_swissknife: /builddir/build/BUILD/cvmfs-2.11.2/cvmfs/swissknife_pull.cc:714: virtual int swissknife::CommandPull::Main(const ArgumentList&): Assertion `reflog != __null' failed.
/bin/cvmfs_server: line 7528: 2937099 Aborted                 $user_shell "$(__swissknife_cmd dbg) pull -m $name         -u $stratum0                                           -w $stratum1                                           -r ${upstream}                                         -x ${spool_dir}/tmp                                    -k $public_key                                         -n $num_workers                                        -t $timeout                                            -a $retries $with_history $with_reflog                    $initial_snapshot_flag $timestamp_threshold $log_level"