DIRAC 7.3 (v7r3)
Make sure the "Rename of scripts" in the v7r2 wiki has been completed before continuing.
Python3 DIRAC clients are now fully supported, and they should be considered the default way of installing DIRAC clients (see instructions).
SiteDirectors are now by default installing python3 clients.
Python3 servers have been extensively tested. The suggestion is that DIRAC servers should be moved, one at a time, from python2 to python3, while running DIRAC 7.3 (v7r3).
This is the last version of DIRAC supporting python2.
Server bashrc files should be updated to be compatible with both the Python 2 and Python 3 based installations. Installations at CERN (LHCb, CLIC, Certification) are using the following bashrc which is compatible with both:
Click to expand!
# DIRAC bashrc file, used by service and agent run scripts to set environment
# Find which DIRAC installation to use
default_dirac="/opt/dirac/pro"
# If "pro" points to a Python 3 version there will be a Linux-x86_64 folder
system_arch="$(uname -s)-$(uname -m)"
if [ -d "${default_dirac}/${system_arch}" ]; then
default_dirac="${default_dirac}/${system_arch}"
fi
if [[ -n "${DIRAC:-}" ]] && [[ "${default_dirac}" != "${DIRAC:-}" ]]; then
echo DIRAC installation path overridden to ${DIRAC}
fi
export DIRAC=${DIRAC:-${default_dirac}}
unset default_dirac system_arch
# CAs path for SSL verification
export X509_CERT_DIR=${X509_CERT_DIR:-"/etc/grid-security/certificates"}
export X509_VOMS_DIR=${X509_VOMS_DIR:-"/etc/grid-security/vomsdir"}
export X509_VOMSES=${X509_VOMSES:-"/etc/vomses"}
if [[ -d "${DIRAC}/diracos" ]]; then
py3_based=0
. $DIRAC/diracos/diracosrc
else
py3_based=1
. $DIRAC/diracosrc
fi
# Configuration which is only required for legacy installations
if [[ "${py3_based}" -eq 0 ]]; then
export PYTHONUNBUFFERED=yes
export PYTHONOPTIMIZE=x
export DIRACSCRIPTS=$DIRAC/scripts
export TERMINFO=$DIRACOS/usr/share/terminfo:/usr/share/terminfo:/etc/terminfo
export RRD_DEFAULT_FONT=$DIRACOS/usr/share/fonts/DejaVuSansMono-Roman.ttf
# Prepend the PYTHONPATH, the LD_LIBRARY_PATH, and the DYLD_LIBRARY_PATH
( echo $PATH | grep -q $DIRACSCRIPTS ) || export PATH=$DIRACSCRIPTS:$PATH
( echo $PYTHONPATH | grep -q $DIRAC ) || export PYTHONPATH=$DIRAC
# new OpenSSL version require OPENSSL_CONF to point to some accessible location
export OPENSSL_CONF=/tmp
# probably not needed but we'll keep it for Python 2
export SSL_CERT_DIR=${SSL_CERT_DIR:-"/etc/grid-security/certificates"}
fi
# The remainder of this file is custom environment configuration for this specific installation
# IPv6 support
export GLOBUS_IO_IPV6=TRUE
export GLOBUS_FTP_CLIENT_IPV6=TRUE
# Fork support for xrootd
export XRD_RUNFORKHANDLER=1
export DIRAC_GFAL_GRIDFTP_SESSION_REUSE=True
# CONDOR config
export _condor_DELEGATE_JOB_GSI_CREDENTIALS_LIFETIME=0
# If there is an extra_bashrc file, source it
current_dir=$(dirname $(readlink -f "$BASH_SOURCE"));
extra_bashrc=$current_dir/extra_bashrc
if [ -f "$extra_bashrc" ]; then
source $extra_bashrc
fisource /path/to/bashrc in the runsvdir-start file. This should be removed before updating to Python 3. You will have to restart runsvdir for the change to take effect.
If using nginx: Add the /WebApp/StaticResourceLinkDir option to the dirac.cfg so Python 3 based webapps can be installed, the recommended value is /opt/dirac/webRoot/resources/.
Older DIRAC versions used to install components with cfg files at paths like /opt/dirac/pro/etc/ResourceStatus_EmailAgent.cfg. These need to be replaced with the target of the symlink (i.e. /opt/dirac/pro/etc/ → /opt/dirac/etc/).
# Find files with matches
grep -r -E '/opt/dirac/pro/etc/[^ /]+\.cfg' /opt/dirac/runit/*/*/run
# Check the new files look reasonable
grep --files-with-matches -r -E '/opt/dirac/pro/etc/[^ /]+\.cfg' /opt/dirac/runit/*/*/run | xargs -I £ sed -E 's@/opt/dirac/pro/etc/([^ /]+\.cfg)@/opt/dirac/etc/\1@g' £
# Assuming the output from the above command looks reasonable, edit the files in place
grep --files-with-matches -r -E '/opt/dirac/pro/etc/[^ /]+\.cfg' /opt/dirac/runit/*/*/run | xargs -I £ sed -i -E 's@/opt/dirac/pro/etc/([^ /]+\.cfg)@/opt/dirac/etc/\1@g' £After the above changes are made the system administrator tools can be used to update a host to a Python 3 based installation. This can be done by adding DIRAC==v7.3 (edit to specify a different primary extension and/or a specific patch release as required). The DIRAC== prefix is only required for the first update. Future updates can be performed by simply entering a PEP-440 style version such as v7.3.999.
Python 3 based installations not able to install Python 2 based versions of DIRAC however you can roll back by either clicking Revert in the system administrator or by manually editing the pro symlink to point to a different version.
The VMDIRAC extension (https://github.com/DIRACGrid/VMDIRAC) has been merged into DIRAC (and its web page, into WebAppDIRAC): https://dirac.readthedocs.io/en/latest/AdministratorGuide/Systems/WorkloadManagement/VMDIRAC.html
If you have been using before VMDIRAC, you should at a minimum update the DIRAC configuration to take into account that the VMDIRAC extension is not anymore needed.
Following this PR, the following services and agents have been removed:
- service
Framework/SystemLogging - service
Framework/SystemLoggingReport - agent
Framework/ErrorMessageMonitor - agent
Framework/SystemLoggingDBCleaner - agent
Framework/TopErrorMessageReporter
And the SystemLogging DB can be removed: DROP DATABASE SystemLoggingDB
Users are encouraged to use the ES stack for logs aggregation like described in this documentation
Extend Extension column in InstalledComponentsDB to be String(512):
ALTER TABLE `HostLogging` MODIFY COLUMN `Extension` VARCHAR(512)Note that the PathFinder methods API has been modified as follows. The main change in API is that the system name and component name are no longer transmitted by tuple, instead the second argument you can insert the name of the component:
PathFinder.getComponentSection('Framework/ProxyManager')OR
PathFinder.getComponentSection('Framework', 'ProxyManager')| method name | before | now |
|---|---|---|
| divideFullName | entityName | entityName, componentName=None |
| getComponentSection | componentName, componentTuple=False, setup=False, componentCategory="Services" |
system, component=False, setup=False, componentCategory="Services" |
| getSystemSection | serviceName, serviceTuple=False, instance=False, setup=False |
system, instance=False, setup=False |
| getSystemInstance | systemName, setup=False |
system, setup=False |
| getServiceSection | serviceName, serviceTuple=False, setup=False |
system, serviceName=False, setup=False |
| getExecutorSection | executorName, executorTuple=False, setup=False |
system, executorName=None, component=False, setup=False |
| getAgentSection | serviceName, serviceTuple=False, setup=False |
system, agentName=None, component=False, setup=False |
| getDatabaseSection | dbName, dbTuple=False, setup=False |
system, dbName=False, setup=False |
| getSystemURLSection | serviceName, serviceTuple=False, setup=False |
system, setup=False |
| getServiceURL | serviceName, serviceTuple=False, setup=False |
system, service=None, setup=False |
| getServiceFailoverURL | serviceName, serviceTuple=False, setup=False |
system, service=None, setup=False |
| getGatewayURLs | serviceName="" | system="", service=None |
The agent WorkloadManagement/StatesMonitoringAgent has been removed (follow-up of this change in v7r2)
The agent WorkloadManagement/TaskQueuesAgent has to be installed (following this PR)
This PR introduced 3 new, optional HTTPs services: JobManager, JobMonitor, JobStateUpdate. Do consider use them.
Following this PR, RequestTasks and WorkflowTasks are now into distinct module instead of being in TaskManager. If your extension relies on it, you need to fix the import path:
- from DIRAC.TransformationSystem.Client.TaskManager import RequestTasks, WorkflowTasks
+ from DIRAC.TransformationSystem.Client.WorkflowTasks import WorkflowTasks
+ from DIRAC.TransformationSystem.Client.RequestTasks import RequestTasks