v8.0.28: No delegation found - what do I do now ?

[marianne013]

Hi,

I have an odd problem with an ARC-CE. We have not been able to run pilots on this CE ever since we upgraded the production server to v8.0.28. It complains about "Response: 404 - No delegation found". Full log below.
I don't see this problem on our pre-prod server, which also runs v8.0.28, and there I can submit jobs to the same CE without any problems. The same site (Lancaster) has a second CE to which I can submit from both the prod and the pre-prod server. I checked the proxy mentioned in the logs, it's valid and looks otherwise inoffensive. It's not VO dependent, I tried several.
Does anyone have any idea on how to debug this ?

2023-09-25 13:43:08 UTC WorkloadManagement/SiteDirectorNA62/WorkloadManagement/SiteDirectorNA62 INFO: Going to submit pilots (a maximum of 16 pilots to grendel.hec.lancs.ac.uk_nordugrid-sungridengine-grid7 queue)
2023-09-25 13:43:08 UTC WorkloadManagement/SiteDirectorNA62/WorkloadManagement/SiteDirectorNA62 INFO: DIRAC project will be installed by pilots
2023-09-25 13:43:08 UTC WorkloadManagement/SiteDirectorNA62/WorkloadManagement/SiteDirectorNA62 VERBOSE: pilotOptions: -S GridPP --pythonVersion=3 -e GridPP -N grendel.hec.lancs.ac.uk -Q nordugrid-sungridengine-grid7 -n LCG.UKI-NORTHGRID-LANCS-HEP.uk
2023-09-25 13:43:09 UTC WorkloadManagement/SiteDirectorNA62/grendel.hec.lancs.ac.uk DEBUG: Set proxy variable X509_USER_PROXY to /tmp/tmp32l0ig36
2023-09-25 13:43:09 UTC WorkloadManagement/SiteDirectorNA62/grendel.hec.lancs.ac.uk VERBOSE: Executable file path: /opt/dirac/work/WorkloadManagement/SiteDirectorNA62/DIRAC_p5l4m3b4_pilotwrapper.py
2023-09-25 13:43:10 UTC WorkloadManagement/SiteDirectorNA62/grendel.hec.lancs.ac.uk ERROR: Issue while interacting with the delegations. Response: 404 - No delegation found
2023-09-25 13:43:10 UTC WorkloadManagement/SiteDirectorNA62/WorkloadManagement/SiteDirectorNA62 ERROR: Failed submission to queue Queue grendel.hec.lancs.ac.uk_nordugrid-sungridengine-grid7
Issue while interacting with the delegations
2023-09-25 13:43:10 UTC WorkloadManagement/SiteDirectorNA62/WorkloadManagement/SiteDirectorNA62 VERBOSE: Committing pilot submission to accounting
2023-09-25 13:43:10 UTC WorkloadManagement/SiteDirectorNA62/WorkloadManagement/SiteDirectorNA62 INFO: Failed pilot submission Queue: grendel.hec.lancs.ac.uk_nordugrid-sungridengine-grid7

[aldbr]

Hi,

We have a similar issue on our side (LHCb) and this is my fault, I haven't taken the time to investigate until now.
I am pretty sure I found the issue, this is minor. I am going to prepare a PR.

Thanks for reporting the problem!

[aldbr]

#7219 solves the problem but needs to be monitored just to make sure it does not create many useless delegations (only 1 delegation per VO should be required).