Skip to content
Permalink
master
Switch branches/tags
Go to file
Tweaked and clarified description.
1 contributor

Users who have contributed to this file

ClusterData 2019 traces

John Wilkes.

The clusterdata-2019 trace dataset provides information about eight different Borg cells for the month of May 2019. It includes the following new information:

  • CPU usage information histograms for each 5 minute period, not just a point sample;
  • information about alloc sets (shared resource reservations used by jobs);
  • job-parent information for master/worker relationships such as MapReduce jobs.

The 2019 traces focus on resource requests and usage, and contain no information about end users, their data, or access patterns to storage systems and other services.

Because of it's size (about 2.4TiB compressed), we are only making the trace data available via Google BigQuery so that sophisticated analyses can be performed without requiring local resources.

The clusterdata-2019 traces are described in this document: Google cluster-usage traces v3. You can find the download and access instructions there, as well as many more details about what is in the traces, and how to interpret them. For additional background information, please refer to the 2015 Borg paper, Large-scale cluster management at Google with Borg.

  • If you haven't already joined our mailing list, please do so now. Important: to avoid spammers, you MUST fill out the "reason" field, or your application will be rejected.

Creative Commons CC-BY license The data and trace documentation are made available under the CC-BY license. By downloading it or using them, you agree to the terms of this license.

Questions?

You can send email to googleclusterdata-discuss@googlegroups.com. The more detailed the request the greater the chance that somebody can help you: screen shots, concrete examples, error messages, and a list of what you already tried are all useful.

Acknowledgements

This trace is the result of a collaboration involving Muhammad Tirmazi, Nan Deng, Md Ehtesam Haque, Zhijing Gene Qin, Steve Hand and Adam Barker.