Cuneiform Plotting Library

This library visualizes Cuneiform history logs. These logs can be collected by connecting to a common runtime environment (CRE) via localhost:4142/history.json.

Usage

Building the cfplot Command Line Tool

A binary can be created by entering

raco make -j 8 *.rkt
raco exe cfplot.rkt

The resulting file cfplot provides a command line interface to the plotting library.

Using cfplot

cfplot my-history-file.json

History JSON format

A history document is a JSON document with a single field history. The value of the history field is a list. Each entry in this list is an object with two fields: app and delta. The app field describes the task that was executed as it was scheduled to the execution engine. The delta field describes the result of the execution. Below is a history document with one app-delta pair. The ellipses at the end of the history list represent the following entries.

{
  "history": [
    {
      "app": {
        "app_id": "042913bc99e7464b8ab377a36bbe17cda963445664cf5f3d1b70e74f",
        "arg_bind_lst": [
          {
            "arg_name": "idx",
            "value": "8d851bb_idx.tar"
          },
          {
            "arg_name": "fastq",
            "value": "558b7dd_SRR576938.sra.fastq"
          }
        ],
        "lambda": {
          "arg_type_lst": [
            {
              "arg_name": "idx",
              "arg_type": "File",
              "is_list": false
            },
            {
              "arg_name": "fastq",
              "arg_type": "File",
              "is_list": false
            }
          ],
          "lambda_name": "bowtie-align",
          "lang": "Bash",
          "ret_type_lst": [
            {
              "arg_name": "bam",
              "arg_type": "File",
              "is_list": false
            }
          ],
          "script": "\n  bam=$fastq.bam\n  tar xf $idx\n  bowtie -S idx $fastq | samtools view -b - > $bam\n"
        }
      },
      "delta": {
        "app_id": "042913bc99e7464b8ab377a36bbe17cda963445664cf5f3d1b70e74f",
        "result": {
          "node": "cf_worker@default-ubuntu-1804",
          "ret_bind_lst": [
            {
              "arg_name": "bam",
              "value": "b70e74f_558b7dd_SRR576938.sra.fastq.bam"
            }
          ],
          "stat": {
            "run": {
              "duration": "178156775904",
              "t_start": "1545049650778968246"
            },
            "sched": {
              "duration": "178197792686",
              "t_start": "1545049650753821561"
            },
            "stage_in_lst": [
              {
                "duration": "13259112",
                "filename": "8d851bb_idx.tar",
                "size": "13373440",
                "t_start": "1545049650755376933"
              },
              {
                "duration": "9195502",
                "filename": "558b7dd_SRR576938.sra.fastq",
                "size": "1355957118",
                "t_start": "1545049650769217200"
              }
            ],
            "stage_out_lst": [
              {
                "duration": "10451127",
                "filename": "b70e74f_558b7dd_SRR576938.sra.fastq.bam",
                "size": "272845997",
                "t_start": "1545049828936511802"
              }
            ]
          },
          "status": "ok"
        }
      }
    },
    ...
  ]
}

App Field

The app field of a history entry describes an independent task (job, invocation, ...). An app is a closure of the form of a function application consisting of a function and argument bindings. An app object has the following fields:

• app_id a unique string that identifies the application. This can be a hash, a UUID, or any arbitrary identifying string.
• lambda an object describing the foreign function to be executed.
• arg_bind_lst a list of objects that enumerate the argument bindings.

Delta Field

The delta field is says the result of the execution.

• app_id must match the corresponding application
• result contains a ret_bind_list field holding the values produced by the application, a node field identifying the computer the application ran on, and a stat field holding statistics about the execution.

Graph Types

Dependency Graph

Complete variant call dependency graph

Detail of the variant call dependency graph

Worker Allocation Graph

Staging Bandwidth over Time Scatter Plot

Staging Bandwidth Density

Stage-in Bandwidth Density per Compute Node

Stage-out Bandwidth Density per Compute Node

Processing Throughput Scatter Plot

Processing Throughput Density

Processing Throughput Density per Compute Node

Processing Selectivity Scatter Plot

System Requirements

• Racket 7.0 or higher

Resources

• cuneiform-lang.org. Official website of the Cuneiform programming language.

Authors

• Jörgen Brandt (@joergen7) joergen@cuneiform-lang.org

License

Apache 2.0