Batch Accounting

A set of utilities to collect batch accounting data, store them in a database and publish them for display on an accounting portal. It also includes various tools to perform some batch monitoring, produce statistics, generate reports and draw plots from accounting information.

Description

The different components (common, loccol, cecol, pub, cpuhours) are organised in Python modules sharing the batchacct Python package:

• the common component contains the common.py module providing the routines, constants and DB table schema information shared by the other modules;
• the loccol component provides the acct.py script, a daemon collecting data from accounting files to send them to the DB, and create.py, a tool to create the accounting DB tables and useful indices;
• the cecol component provides the whisk.py script, a daemon collecting data from CREAM CE BLAH files to send them to the DB;
• the pub component provides the join.py script which, when periodically run as a cron job, joins the data collected by the loccol and cecol components before publishing them for display on the accounting portal;
• the cpuhours component contains the cpuhours.py script which provides tools to plot information based on the accounting data stored in the DB.

Accounting Workflow

Both the loccol and cecol components read accounting files generated by schedulers and send the job records to the DB. The pub component reads the job records from the DB, processes them and sends them to a message broker or writes them to files a message broker client can read.

DB Connection

The connection file which must be supplied to the different components contains the credentials necessary to connect to the DB. It should hold the following format:

username/password@dsn

Parsing Accounting Files

The loccol component expects LSF accounting files. It parses them by using a modified PyLSF module. Likewise, the cecol component expects BLAH accounting files as generated by CREAM CEs and parses them by using the parse generator function defined in the common.py module.

You can easily use another parser of yours if you need to read a different type of accounting file. What you need to do is to pass a generator as second argument (recs) to the insert function defined in the common.py module. This generator has to yield a dictionary of job record field name keys mapped to their values.

Typical Setup

The common, loccol, cecol and pub components are typically RPM-packaged by subsequently running make sdist and make rpm.

Unpackaged Setup

The cpuhours.py script isn't provided with any packaging yet, but you can run it -- or any of the other scripts -- in place, without RPM-installing anything. This is possible by simply adding the path to the common.py module to your PYTHONPATH. For instance, if your working directory is that of the cpuhours.py script, you may want to run on command along the lines of:

cpuhours/batchacct% PYTHONPATH=../../common/batchacct:$PYTHONPATH
cpuhours/batchacct% python cpuhours.py -c connectionfile

Examples

• Starting the acct.py daemon to read accounting files and send job records to the DB:

    loccol/batchacct% python acct.py --connfile connectionfile
                                     --acctfile /path/to/accountingfile
                                     --pidfile /var/run/batchacct/loccol.pid
                                     --logfile /var/log/batchacct/loccol.log
  

• Likewise, starting the whisk.py daemon to read CREAM CE BLAH accounting files and send job records to the DB:

    cecol/batchacct% python whisk.py --connfile connectionfile
                                     --acctdir /path/to/accountingdir/
                                     --pidfile /var/run/batchacct/cecol.pid
                                     --logfile /var/log/batchacct/cecol.log
  

• Launching the join.py daemon to read job records from the DB and write them to message files for a message broker client to further report them. Note the --conf option which specifies a file holding reporting settings (see the example in pub/pub) and the --vofile option which specifies a file containing VO-to-group mappings (see the example in pub/vos):

    pub/batchacct% python join.py --acctdbfile connectionfile
                                  --conf /path/to/pubconf
                                  --vofile /path/to/vofile
                                  --ssm /path/to/outgoing/messages/
                                  --logfile /var/log/batchacct/pub.log
  

Online Help

Each of the acct.py, create.py, whisk.py, join.py and cpuhours.py scripts can be passed the -h option to print out a summary of the available options along with a short description.