Skip to content


Switch branches/tags

Name already in use

A tag already exists with the provided branch name. Many Git commands accept both tag and branch names, so creating this branch may cause unexpected behavior. Are you sure you want to create this branch?

Latest commit


Git stats


Failed to load latest commit information.
Latest commit message
Commit time

Elasticsearch Job Accounting Scripts


git clone the repository.


You can either work out of the cloned source directory, or you add the cloned source directory to PYTHONPATH and copy the script to another working directory.

The main script is

usage: [-h] [--filter FILTER] [--formatter FORMATTER]
                     [--es_index ES_INDEX] [--es_host ES_HOST]
                     [--es_user ES_USER] [--es_use_https]
                     [--es_ca_certs ES_CA_CERTS] [--daily] [--weekly]
                     [--monthly] [--start_ts START_TS] [--end_ts END_TS]
                     [--csv_dir CSV_DIR] [--from_addr FROM_ADDR]
                     [--to_addr TO_ADDR] [--cc_addr CC_ADDR]
                     [--bcc_addr BCC_ADDR]

optional arguments:
  -h, --help            show this help message and exit
  --filter FILTER       Filter class, one of [BaseFilter, OsgScheddCpuFilter]
                        (default: BaseFilter)
  --formatter FORMATTER
                        Formatter class, one of [BaseFormatter,
                        OsgScheddCpuFormatter] (default: BaseFormatter)
  --es_index ES_INDEX   Elasticsearch index (default: htcondor-000001)
  --es_host ES_HOST     Elasticsearch hostname[:port] (default: localhost)
  --es_user ES_USER     Elasticsearch username
  --es_use_https        Use HTTPS with Elasticsearch (default:
  --es_ca_certs ES_CA_CERTS
                        Elasticsearch custom CA certs
  --daily               Set report period to last full day
                        (REPORT_PERIOD=daily, default)
  --weekly              Set report period to last full week
  --monthly             Set report peroid to last full month
  --start_ts START_TS   Custom starting timestamp
  --end_ts END_TS       Custom ending timestamp
  --csv_dir CSV_DIR     Output directory for CSVs (default: csv)
  --from_addr FROM_ADDR
                        From: email address (default:
  --to_addr TO_ADDR     To: email address (can be specified multiple times)
  --cc_addr CC_ADDR     CC: email address (can be specified multiple times)
  --bcc_addr BCC_ADDR   BCC: email address (can be specified multiple times)

This is best wrapped in a shell script that sets up the required Python environment. Using an existing Miniconda environment named job_accounting, for example, you could have a script like:


# Set up the Python environment
source ~/miniconda3/bin/activate
conda activate job_accounting
pip install -r requirements.txt

# Send an OSPool email to and
python3 \
    --daily \
    --es_index="osg-schedd-*" --filter=OsgScheddCpuFilter --formatter=OsgScheddCpuFormatter \
    --to_addr="" \


There are base filtering and formatting classes along with child classes related to the OSPool report in accounting/filters and accounting/formatters. Filter classses contain methods that query Elasticsearch for specific fields, and contain methods that use the resulting data to derive output fields (like All CPU Hours). Formatting classes contain methods that take a CSV file and turns the data in the CSV file into nicely formatted HTML.

Changes to the OSPool report should only require changes to the OSPool-specific classes (OsgScheddCpuFilter and OsgScheddCpuFormatter).

If you need to create a new type of report, create new filter and format classes, using the existing classes as reference. Be sure to add any new classes to the in the filters and formatters directories, and update with the new class names if you want them to show up in the --help message.