Use HTCondor python API

[mesmith75]

Start on #1916

• Submit
• Kill
• Resubmit
• Query
• peek
• Hold, release + other extras

[egede]

Remember that there is a new directory GangaCore/test/Condor where tests of HTCondor can be placed.

[mesmith75]

The monitoring is currently returning an error:

ERROR    Exception raised executing 'updateMonitoringInformation' in Thread 'Backend Monitor':
Traceback (most recent call last):
  File "/afs/cern.ch/user/m/masmith/cmtuser/GANGA/GANGA_HEAD/install/ganga/ganga/GangaCore/Core/GangaThread/WorkerThreads/WorkerThreadPool.py", line 124, in __worker_thread
    result = item.command_input.function(*these_args, **item.command_input.kwargs)
  File "/afs/cern.ch/user/m/masmith/cmtuser/GANGA/GANGA_HEAD/install/ganga/ganga/GangaCore/Lib/Condor/Condor.py", line 480, in updateMonitoringInformation
    stati = schedd.query(constraint = expr_tree, projection = ["ClusterId", "ProcId", "JobStatus", "RemoteUserCpu","AllRemoteHosts"])
  File "/afs/cern.ch/user/m/masmith/gangatest_python3/lib/python3.6/site-packages/htcondor/_deprecation.py", line 111, in wrapper
    return method(self, *args, **kwargs)
  File "/afs/cern.ch/user/m/masmith/gangatest_python3/lib/python3.6/site-packages/htcondor/_lock.py", line 69, in wrapper
    rv = func(*args, **kwargs)
htcondor.HTCondorIOError: Failed to fetch ads from schedd, errmsg=SECMAN:2007:Failed to end classad message.

I am not sure if this is a fatal issue or something we can hide

[mesmith75]

This is giving an error when trying to access a sandbox:

Traceback (most recent call last):
  File "./Ganga_13_Executable", line 74, in getPackedInputSandbox
    with closing(tarfile.open(tarpath, "r:*")) as tf:
  File "/usr/lib64/python3.6/tarfile.py", line 1573, in open
    return func(name, "r", fileobj, **kwargs)
  File "/usr/lib64/python3.6/tarfile.py", line 1638, in gzopen
    fileobj = gzip.GzipFile(name, mode + "b", compresslevel, fileobj)
  File "/usr/lib64/python3.6/gzip.py", line 163, in __init__
    fileobj = self.myfileobj = builtins.open(filename, mode or 'rb')
PermissionError: [Errno 13] Permission denied: '/afs/cern.ch/work/m/masmith/GangaTest/test/repos/workspace/masmith/LocalXML/13/input/_input_sandbox_13_master.tgz'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "./Ganga_13_Executable", line 146, in <module>
    getPackedInputSandbox( inFile )
  File "./Ganga_13_Executable", line 77, in getPackedInputSandbox
    raise Exception("Error opening tar file: %s" % tarpath)
Exception: Error opening tar file: /afs/cern.ch/work/m/masmith/GangaTest/test/repos/workspace/masmith/LocalXML/13/input/_input_sandbox_13_master.tgz

It looks as if these jobs cannot access the AFS file system

[egede]

Link to CERN helpdesk request

[egede]

So, it seems like it is possible to preserve the kerberos tickets using the Python API as well. Just make sure that the code for that is written such that it doesn't break the Condor backend in places without afs.

[mesmith75]

@egede I think this is ready - all the existing functionality has been updated (there is no python binding for condor_tail yet).

I'll put any other extras in another PR, along with the fix to the outputfiles.