Fix releasing of gc lock

[DrDaveD]

The new gc lock added in #2850 introduced in 2.9.1 is never getting released. This PR fixes that.

I'm afraid this bug is bad enough to require a 2.9.3 release. The tests added fail without the fix.

[cernvm-bot]

Can one of the admins verify this patch?

[jblomer]

@cernvm-bot cloudtest

[jblomer]

Thanks for the patch! Just curious, how did you find out?

After the merge, I can cherry-pick the commit in the 2.9 branch and we can prepare a 2.9.3 release.

[cernvm-bot]

building for cloudtests finished: SUCCESS
https://lcgapp-services.cern.ch/cvmfs-jenkins/job/CvmfsFullBuildDocker/3386/

[cernvm-bot]

cloudtests finished: FAILURE
https://lcgapp-services.cern.ch/cvmfs-jenkins/job/CvmfsCloudTesting/1393/

[DrDaveD]

On the Nebraska stratum one where I am running cvmfs_server check -a I wanted to redo a check on one of the repositories after restoring some missing catalogs. When I did that it said it was waiting for garbage collection to complete but I checked and there was no garbage collection running. While investigating that I found that many repositories still had that lock and I looked at the code to find out how that could happen.

I think that systems that only run cvmfs_server gc -a will not be likely to notice this because each run is a different process and the lock will get overridden because the previous process is no longer running. The process ID that goes into the lock file turns out to be the gc -a parent process, not each subprocess (because $$ stays at the parent process even inside a parentheses-created subshell) so there was a long running gc -a that was making all the locks be seen as valid, even when gc wasn't running on that particular repository.

[radupopescu]

I don't think we can release this as 2.9.3, as we recently pushed a nightly build tagged 2.10.0-0.384 to the cvmfs-testing repositories. We can, however, also publish this to the testing repo as a new nightly build.

[DrDaveD]

You're saying you can't also release a 2.9.3 to the testing repository? Even if there's a newer one such that people can't install it with just yum install cvmfs-server, they could still install it when asking for a particular version number.

[jblomer]

It's also backported to the 2.9 branch

[radupopescu]

@DrDaveD Ok, in this case, if manually installing version 2.9.3 is useful, we can release it.