Skip to content
No description, website, or topics provided.
Jupyter Notebook
Branch: master
Clone or download
Latest commit 6aa3a7a Jul 10, 2019
Permalink
Type Name Latest commit message Commit time
Failed to load latest commit information.
analysis fix analysis license location Jul 10, 2019
.gitattributes Initial import of Trace Data and Analysis Notebook Jul 10, 2019
LICENSE adding license files Jul 10, 2019
README.md Update README.md Jul 10, 2019
trace-data.tar.gz Initial import of Trace Data and Analysis Notebook Jul 10, 2019

README.md

Overview

This repository contains a representative subset of the first-party DNN training workloads on Microsoft's internal Philly clusters. The trace is a sanitized subset of the workload described in "Analysis of Large-Scale Multi-Tenant GPU Clusters for DNN Training Workloads" in ATC’19. We include in this repository a jupyter notebook that highlights the main characteristics of the traces and shows how to parse them (a huge thank you to Keshav Santhanam for putting this together).

We provide the trace as is. If you do use this trace in your research, please make sure to cite our ATC’19 paper (mentioned above).

Trace Details

Main characteristics:

  • Dataset size: 6.6 GB
  • Compressed dataset size: 0.98 GB
  • Number of files: 5 files
  • Duration: Jobs submitted between 2017-08-07 - 2017-12-22
  • Total number of jobs: 117325

Schema:

cluster_job_log

Description: Contains information about each job, including each individual successful scheduling attempt.

Format: JSON

Example entry:

{
    "status": "Pass",
    "vc": "2869ce",
    "jobid": "application_1506638472019_10238",
    "attempts": [
        {
            "start_time": "2017-10-05 14:50:06",
            "end_time": "2017-10-08 21:08:07",
            "detail": [
                {
                    "ip": "m193",
                    "gpus": [
                        "gpu0",
                        "gpu4",
                        "gpu5",
                        "gpu7"
                    ]
                },
                {
                    "ip": "m287",
                    "gpus": [
                        "gpu1",
                        "gpu2",
                        "gpu3",
                        "gpu4"
                    ]
                }
            ]
        },
        {
            "start_time": "2017-10-08 09:24:12",
            "end_time": "2017-10-08 21:08:07",
            "detail": [
                {
                    "ip": "m56",
                    "gpus": [
                        "gpu0",
                        "gpu1",
                        "gpu2",
                        "gpu3",
                        "gpu4",
                        "gpu5",
                        "gpu6",
                        "gpu7"
                    ]
                }
            ]
        }
    ],
    "submitted_time": "2017-10-05 14:49:31",
    "user": "e4f747"
}

List of keys:

  • status: The job's status upon completion. One of Pass, Killed, or Failed.
  • vc: The hash of the virtual cluster the job was run in.
  • jobid: The id of the job.
  • attempts: A list of dicts where each dict has the following keys:
    • start_time: The start time of the attempt.
    • end_time: The end time of the attempt.
    • detail: A list of dicts where each dict has the following keys:
      • ip: The id of the server the attempt was scheduled on.
      • gpus: A list of GPUs used by the attempt.
  • submitted_time: The time the job was submitted to the scheduler.
  • user: A hash of the user id.

Notes:

  • A job may have no recorded scheduling attempts.
  • A scheduling attempt may have no recorded start_time and/or end_time - this could be the result of a logging error.
  • If a job has a None value for its last attempt's end_time, the job was still running at the time the snapshot was taken.

cluster_gpu_util

Description: Provides a per-minute record of each GPU's utilization as reported by nvidia-smi.

Format: CSV

Columns:

time machineId gpu0_util gpu1_util gpu2_util gpu3_util gpu4_util gpu5_util gpu6_util gpu7_util

Example entry:

2017-10-03 00:08:00 PDT,m29,60.8,99.366666667,100.0,63.333333333,100.0,100.0,100.0,100.0,

Notes:

  • Some gpu*_util values may be "NA", indicating the GPU was offline at the time of measurement.

cluster_cpu_util

Description: Provides a per-minute record of each server's CPU utilization.

Format: CSV

Columns:

time machine_id cpu_util

Example entry:

2017-11-27 00:04:00 PST,m29,31.845

Notes:

  • Some cpu_util values may be "NA", indicating the server was offline at the time of measurement.

cluster_mem_util

Description: Provides a per-minute record of each server's memory utilization.

Format: CSV

Columns:

time machine_id mem_total mem_free

Example entry:

2017-10-03 00:06:00 PDT,m29,528272672.0,2030730.6667

Notes:

  • Some mem_total and mem_free values may be "NA", indicating the server was offline at the time of measurement.

cluster_machine_list

Description: Lists the number of GPUs and per-GPU memory available on each server in the cluster.

Format: CSV

Columns:

machineId number of GPUs single GPU mem

Example entry:

m31,8, 24GB
You can’t perform that action at this time.