RequestExecutingAgent and StorageElementCache problems #3727
Description
When the RequestExecutingAgent (REA) is executing requests for non-shifters, it gets proxies for the user and then deletes them after the execution of the operation. This causes problems when StorageElements are involved, because when they are created and a gfal2 context is created this context picks up the location of the proxy ($X509_USER_PROXY) and keeps that for its lifetime, which is equal to the lifetime of the StorageElement. However, this proxy file is deleted after the execution of the request that caused the creation of the StorageElement.
I noticed this behaviour in the development of the MultiVO transformations, because I was then only running requests as a "user" proxy. The resulting error messages from GFAL2 are somewhat cryptic
GError: srm-ifce err: Communication error on send, err: [SE][Ls][] httpg://srm-public.cern.ch:8443/srm/managerv2: CGSI-gSOAP running on pclcd50.dyndns.cern.ch reports No such file or directory
(No such file or directory refers to the proxy, not to the file on the storage, took me way too long to understand what was happening...)
When running with predominantly shifterproxies for the execution of the Requests the error messages don't show up, because the user requests will simply use the shifter proxy associated to the StorageElement already in the cache. The problem is still there. that the wrong proxy is used for the request when calling storage elements.
The gfal2 context storing the proxy location can be reproduced in just python
import os, gfal2
ctx = gfal2.creat_context() # this picks up the proxy from $X509_USER_PROXY, so export before starting python
ctx.stat("srm://srm-public.cern.ch:8443/srm/managerv2?SFN=/castor/cern.ch/grid/ilc/prod/clic/500gev/Z_uds/CLIC_o3_v14/SIM/00010494/000/Z_uds_sim_10494_304.slcio")
os.environ['X509_USER_PROXY' ] = '/no/such/file' # this is basically what the REA is doing except it points to an existing proxy, it just won't be used
ctx.stat("srm://srm-public.cern.ch:8443/srm/managerv2?SFN=/castor/cern.ch/grid/ilc/prod/clic/500gev/Z_uds/CLIC_o3_v14/SIM/00010494/000/Z_uds_sim_10494_304.slcio") # still works because we use the original proxy
ctx = gfal2.creat_context() # picks up the location now in $509_USER_PROXY
ctx.stat("...") # fails with file not found