Dirac proxy multi VO problems #45

alexanderrichards · 2015-11-20T14:36:18Z

Here is an email from simon quoting an LSST user:

We've been given the long report below from a user testing the LSST VO
using ganga + our DIRAC server. The gist of it seems to be that ganga is
getting a vanilla proxy, which the DIRAC server will then attach a VOMS
proxy to at job submission time. Unfortunately this user is a member of
multiple VOs and DIRAC sometimes picks a different VO to the one they're
trying to test... I guess the questions we need to answer are:

 - Is this behaviour reproducible by us?
 - Is there some way to get ganga to get a VOMS proxy so that there is no
   room for the DIRAC server to make any decisions on the VO?

Would you be able to have a look at this?

The original email is below. Hopefully this is a small fix but obviously the new credentials system will be the proper solution.

Most of the jobs following those 4 failed with a mixture of

Stalling for more than 11700 sec and Job stalled: pilot not running

at all sites but Birmingham where they weren't supposed to run.

Since I put the right dirac-proxy-init in .gangarc I looked a bit better at what happens and it seems >not to care, it just generates a plain proxy.

if I run the dirac command standalone I get this proxy
{quote}
aforti@vm7>dirac-proxy-init -g lsst_user -M
Generating proxy...
Enter Certificate password:
Added VOMS attribute /lsst
Uploading proxy for lsst_user...
Proxy generated:
subject : /C=UK/O=eScience/OU=Manchester/L=HEP/CN=alessandra forti/CN=proxy/CN=proxy
issuer : /C=UK/O=eScience/OU=Manchester/L=HEP/CN=alessandra forti/CN=proxy
identity : /C=UK/O=eScience/OU=Manchester/L=HEP/CN=alessandra forti
timeleft : 23:53:59
DIRAC group : lsst_user
path : /tmp/x509up_u500
username : alessandra.forti
properties : NormalUser
VOMS : True
VOMS fqan : ['/lsst']

Proxies uploaded:
DN | Group | Until (GMT)
/C=UK/O=eScience/OU=Manchester/L=HEP/CN=alessandra forti | vo.northgrid.ac.uk_user | 2016/11/03 11:48
/C=UK/O=eScience/OU=Manchester/L=HEP/CN=alessandra forti | gridpp_user | 2016/11/03 11:48
/C=UK/O=eScience/OU=Manchester/L=HEP/CN=alessandra forti | lsst_user | 2016/11/03 11:48
aforti@vm7>voms-proxy-info -all
subject : /C=UK/O=eScience/OU=Manchester/L=HEP/CN=alessandra forti/CN=proxy/CN=proxy
issuer : /C=UK/O=eScience/OU=Manchester/L=HEP/CN=alessandra forti/CN=proxy
identity : /C=UK/O=eScience/OU=Manchester/L=HEP/CN=alessandra forti/CN=proxy
type : proxy
strength : 1024 bits
path : /tmp/x509up_u500
timeleft : 23:53:42
key usage : Digital Signature, Key Encipherment, Data Encipherment
=== VO lsst extension information ===
VO : lsst
subject : /C=UK/O=eScience/OU=Manchester/L=HEP/CN=alessandra forti
issuer : /DC=com/DC=DigiCert-Grid/O=Open Science Grid/OU=Services/CN=voms1.fnal.gov
attribute : /lsst/Role=NULL/Capability=NULL
timeleft : 23:53:42
uri : voms1.fnal.gov:15003
{quote}

when I put that command in ganga this is what happen instead

{quote}
aforti@vm7>grep dirac-proxy-init .gangarc
[defaults_GridCommand]init = dirac-proxy-init -g lsst_user -M

aforti@vm7>ganga
Your identity: /C=UK/O=eScience/OU=Manchester/L=HEP/CN=alessandra forti
Enter GRID pass phrase for this identity:
Creating proxy ........................................................................................................................... Done
Your proxy is valid until: Fri Nov 20 23:16:25 2015

*** Welcome to Ganga ***
Version: Ganga-6-1-6-hotfix1
Documentation and support: http://cern.ch/ganga
Type help() or help('index') for online help.

This is free software (GPL), and you are welcome to redistribute it
under certain conditions; type license() for details.

Ganga.Utility.Config : INFO reading config file /home/aforti/.gangarc

In [1]:
Do you really want to exit ([y]/n)? y
Ganga.Core.MonitoringComponent : INFO Stopping the monitoring component...
aforti@vm7>voms-proxy-info -all
subject : /C=UK/O=eScience/OU=Manchester/L=HEP/CN=alessandra forti/CN=400330830
issuer : /C=UK/O=eScience/OU=Manchester/L=HEP/CN=alessandra forti
identity : /C=UK/O=eScience/OU=Manchester/L=HEP/CN=alessandra forti
type : RFC compliant proxy
strength : 1024 bits
path : /tmp/x509up_u500
timeleft : 23:59:43
key usage : Digital Signature, Key Encipherment, Data Encipherment
{quote}

it generates a plain proxy without VOMS information. With LHCb this still works because they have >only LHCb on their servers but with the multi-VO gridpp Dirac it picks the first VO I belong to to run >the jobs if the jobs are submitted without VOMS credentials.

The text was updated successfully, but these errors were encountered:

drmarkwslater · 2015-11-22T09:18:55Z

I can't actually recreate this behaviour outside Ganga - i.e. I can't get dirac-proxy-init to NOT add the VOMS extensions - can anyone else recreate this outside Ganga?

rob-c · 2015-11-22T23:12:36Z

@drmarkwslater We're not using this tool to create destroy proxies.
We use the voms-proxy-init most of the time I think (I've not played around with the vanilla install for a while)
If the dirac-proxy-init does something more sensible than voms-proxy-init or grid-proxy-init currently in the credentials code then surely the best thing to do is to use this as the default tool?
(also this goes more hand in hand with the GridPP way of presenting Ganga+Dirac to be the best way to use the grid)

afortiorama · 2015-11-27T13:43:13Z

Hi,

dirac-proxy-init in .gangarc is suggested in this documentation written by Mark

https://www.gridpp.ac.uk/wiki/Guide_to_Ganga#Installation_and_Configuration

At first I thought the problem was -M missing from the command, but even with that it didn't work. We are heavily relying on that wiki in the UK for smaller or local groups. If it needs to be corrected please correct it.

drmarkwslater · 2015-11-27T14:00:46Z

I put this in as it is what is used by the Dirac docs on the GridPP wiki. This will mean we get the Dirac user group as well as a default voms proxy. For LHCb, this is set to lhcb-proxy-init I believe.

In any case, @afortiorama are you able to get dirac-proxy-init to NOT create the VOMS extensions? I don't seem to be able to and it also picks the correct ones even without -M:

bash-4.1$ dirac-proxy-init -g gridpp_user
Generating proxy...
Enter Certificate password:
Added VOMS attribute /gridpp
Uploading proxy for gridpp_user...
Proxy generated:
subject : /C=UK/O=eScience/OU=Birmingham/L=ParticlePhysics/CN=mark slater/CN=proxy/CN=proxy
issuer : /C=UK/O=eScience/OU=Birmingham/L=ParticlePhysics/CN=mark slater/CN=proxy
identity : /C=UK/O=eScience/OU=Birmingham/L=ParticlePhysics/CN=mark slater
timeleft : 23:53:59
DIRAC group : gridpp_user
path : /tmp/x509up_u34811
username : mark.slater
properties : NormalUser
VOMS : True
VOMS fqan : ['/gridpp']

Proxies uploaded:
DN | Group | Until (GMT)
/C=UK/O=eScience/OU=Birmingham/L=ParticlePhysics/CN=mark slater | na62.vo.gridpp.ac.uk_user | 2016/05/26 13:17
/C=UK/O=eScience/OU=Birmingham/L=ParticlePhysics/CN=mark slater | gridpp_user | 2016/05/26 13:17
bash-4.1$ voms-proxy-info --all
subject : /C=UK/O=eScience/OU=Birmingham/L=ParticlePhysics/CN=mark slater/CN=proxy/CN=proxy
issuer : /C=UK/O=eScience/OU=Birmingham/L=ParticlePhysics/CN=mark slater/CN=proxy
identity : /C=UK/O=eScience/OU=Birmingham/L=ParticlePhysics/CN=mark slater/CN=proxy
type : proxy
strength : 1024 bits
path : /tmp/x509up_u34811
timeleft : 23:53:36
key usage : Digital Signature, Key Encipherment, Data Encipherment
=== VO gridpp extension information ===
VO : gridpp
subject : /C=UK/O=eScience/OU=Birmingham/L=ParticlePhysics/CN=mark slater
issuer : /C=UK/O=eScience/OU=Manchester/L=HEP/CN=voms.gridpp.ac.uk
attribute : /gridpp/Role=NULL/Capability=NULL
timeleft : 23:53:36
uri : voms.gridpp.ac.uk:15000
bash-4.1$ dirac-proxy-init -g na62.vo.gridpp.ac.uk_user
Generating proxy...
Enter Certificate password:
Added VOMS attribute /na62.vo.gridpp.ac.uk
Uploading proxy for na62.vo.gridpp.ac.uk_user...
Proxy generated:
subject : /C=UK/O=eScience/OU=Birmingham/L=ParticlePhysics/CN=mark slater/CN=proxy/CN=proxy
issuer : /C=UK/O=eScience/OU=Birmingham/L=ParticlePhysics/CN=mark slater/CN=proxy
identity : /C=UK/O=eScience/OU=Birmingham/L=ParticlePhysics/CN=mark slater
timeleft : 23:53:59
DIRAC group : na62.vo.gridpp.ac.uk_user
path : /tmp/x509up_u34811
username : mark.slater
properties : NormalUser
VOMS : True
VOMS fqan : ['/na62.vo.gridpp.ac.uk']

Proxies uploaded:
DN | Group | Until (GMT)
/C=UK/O=eScience/OU=Birmingham/L=ParticlePhysics/CN=mark slater | na62.vo.gridpp.ac.uk_user | 2016/05/26 13:17
/C=UK/O=eScience/OU=Birmingham/L=ParticlePhysics/CN=mark slater | gridpp_user | 2016/05/26 13:17
bash-4.1$ voms-proxy-info --all
subject : /C=UK/O=eScience/OU=Birmingham/L=ParticlePhysics/CN=mark slater/CN=proxy/CN=proxy
issuer : /C=UK/O=eScience/OU=Birmingham/L=ParticlePhysics/CN=mark slater/CN=proxy
identity : /C=UK/O=eScience/OU=Birmingham/L=ParticlePhysics/CN=mark slater/CN=proxy
type : proxy
strength : 1024 bits
path : /tmp/x509up_u34811
timeleft : 23:53:37
key usage : Digital Signature, Key Encipherment, Data Encipherment
=== VO na62.vo.gridpp.ac.uk extension information ===
VO : na62.vo.gridpp.ac.uk
subject : /C=UK/O=eScience/OU=Birmingham/L=ParticlePhysics/CN=mark slater
issuer : /C=UK/O=eScience/OU=Imperial/L=Physics/CN=voms03.gridpp.ac.uk
attribute : /na62.vo.gridpp.ac.uk/Role=NULL/Capability=NULL
timeleft : 23:53:37
uri : voms03.gridpp.ac.uk:15501

I'll keep looking to see if I can see any other issues...

drmarkwslater · 2015-11-27T14:12:41Z

Hi Again!

@afortiorama Putting some debugging messages in shows that Ganga is running the command:

dirac-proxy-init -g gridpp_user -M -valid 24:00

and this works for me (as I say, even without the -M). What happens if you run this on the command line? Does it produce the voms extension correctly? Could you maybe send me your Dirac setup script? Maybe theres some differences there...

afortiorama · 2015-11-27T14:16:08Z

Hi Mark,

the problem wasn't with dirac-proxy-init but with what ganga runs. If you read the initial post it tells you what I did.

cheers
alessandra

drmarkwslater · 2015-11-27T14:21:40Z

@afortiorama so looking at the specific output of your Ganga session it looks like it's not actually running dirac-proxy-init at all. In Ganga, it has the following:

Your identity: /C=UK/O=eScience/OU=Manchester/L=HEP/CN=alessandra forti
Enter GRID pass phrase for this identity:

Which is different from the output of dirac-proxy-init:

Generating proxy...
Enter Certificate password:

So could you send me your .gangarc (either attached to this or privately)? I suspect there's some setting that it making Ganga use grid-proxy-init instead.

drmarkwslater · 2015-11-27T15:03:10Z

Found the problem! It seems that python (or at least Ganga) doesn't like:

[group]param=value

instead of:

[group]
param=value

I've updated the docs for the GridPP wiki and will check to see if this is a Python or Ganga limitation.

Thanks!

Mark

alexanderrichards added bug DIRAC labels Nov 20, 2015

drmarkwslater removed the bug label Nov 27, 2015

drmarkwslater self-assigned this Nov 27, 2015

drmarkwslater closed this as completed Nov 27, 2015

drmarkwslater mentioned this issue Apr 28, 2016

LSST Im3Shape support in Ganga #343

Closed

15 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Dirac proxy multi VO problems #45

Dirac proxy multi VO problems #45

alexanderrichards commented Nov 20, 2015

drmarkwslater commented Nov 22, 2015

rob-c commented Nov 22, 2015

afortiorama commented Nov 27, 2015

drmarkwslater commented Nov 27, 2015

drmarkwslater commented Nov 27, 2015

afortiorama commented Nov 27, 2015

drmarkwslater commented Nov 27, 2015

drmarkwslater commented Nov 27, 2015

Dirac proxy multi VO problems #45

Dirac proxy multi VO problems #45

Comments

alexanderrichards commented Nov 20, 2015

drmarkwslater commented Nov 22, 2015

rob-c commented Nov 22, 2015

afortiorama commented Nov 27, 2015

drmarkwslater commented Nov 27, 2015

drmarkwslater commented Nov 27, 2015

afortiorama commented Nov 27, 2015

drmarkwslater commented Nov 27, 2015

drmarkwslater commented Nov 27, 2015