Home

WMArchive wiki pages

WMArchive is CMS data-service to store FrameWork JobReport (FWJR) documents. It provides a set of tools to store FWJR into short-term (MongoDB) and long-term (HDFS) storages, to convert FWJR from JSON to AVRO data-formats and end-user tools (myspark) to run a spark job to find FWJR on HDFS. Please use side bar to search for specific topic.

Managing the service

WMArchive has two nodes, vocms0181 (pre-production) and vocms0182 (production). To start, stop the service you need only two stpes:

# login to wma account
sudo -i -u wma /bin/bash
# service command (start|stop|status)
/data/vm_manage.sh start

If you do not want to use this wrapper, then you can use the standard commands for managing services in our VMs, like:

(A=/data/cfg/admin; cd /data; $A/InstallDev -s status:wmarchive)
(A=/data/cfg/admin; cd /data; $A/InstallDev -s stop:wmarchive)
(A=/data/cfg/admin; cd /data; $A/InstallDev -s start:wmarchive)

Web interface

WMArchvie web interface is accessible via /wmarchive/web/performance, while its APIs via /wmarchive/web/apis URLs.

Technologies

• Big-data storage backend: Apache Hadoop
• Aggregation pipeline: Apache Spark, MongoDB, Python
• REST web server: WMArchive, Python
• Frontend: JavaScript with Backbone.js, HTML, CSS, Sass
• Visualization: D3.js

Documentation

Preparation:

• Running the WMArchive Server
• Generating sample data

Usage and Implementation:

• Performance data REST endpoint
• Performance UI architecture
• Common tasks

Aggregation and Data:

• Performance data structure
• Aggregation procedure

Dashboard and Explantion:

• Time & CPU
• Host & All
• Notes for Time & CPU Plots

Outlook:

• Pending improvements
• Known issues