Fix spack locking on some NFS systems

[sethrj]

It turns out that on some filesystem/os combinations, calling os.makedirs can raise PermissionError (errno.EPERM) if the directory is extant and readable, but not writable. This call happened before trying to activate environments.

Fixes #17407

[tgamblin]

I think catching OSError and just warning if the directory already exists is too broad. We're trying to be really specific in llnl.util.lock and I think it's better to fail fast in the corner cases than to let something potentially dangerous slip through.

#17407 is giving you errno 13, which is EACCES. If that is the same error you see with NFSv4, I think we should check for that errno specifically, and allow it if it happens that the directory exists.

This is ok since as you note, we may be trying to get a read-only lock. I think we could avoid this issue altogether in Python 3 with the exist_ok parameter to os.makedirs but sadly we (for the time being) need to support 2.7. It'd be interesting to know if exist_ok gets rid of the permissions error if you are trying this with Python 3.

[tgamblin]

Ok actually I just looked at the cpython source, and it's broad. So why don't we just go with that. Here is what they do in Python 3:

    try:
        mkdir(name, mode)
    except OSError:
        # Cannot rely on checking for EEXIST, since the operating system
        # could give priority to other errors like EACCES or EROFS
        if not exist_ok or not path.isdir(name):
            raise

So apparently we care about EROFS and maybe others too.

[sethrj]

Thanks @tgamblin !

[tpdownes]

@tgamblin the way I read things, this change is not presently slated to make it into v0.18.2. Is it possible to make that happen? We have been hitting it regularly maintaining shared Spack installations for our HPC users.

GoogleCloudPlatform/cluster-toolkit#539

[haampie]

Me and @alalazo are usually the only people doing backports, but I've largely stopped doing that since #24718 because pretty much nothing applies cleanly.

This is a small fix, so we can probably do it.