Skip to content

Experiment data from the paper "Towards Collaborative Optimization of Cluster Configurations for Distributed Dataflow Jobs"

Notifications You must be signed in to change notification settings

dos-group/c3o-experiments

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 

History

8 Commits
 
 
 
 
 
 
 
 
 
 
 
 

Repository files navigation

C3O Experiments

Here are the experiment data for the paper

Towards Collaborative Optimization of Cluster Configurations for Distributed Dataflow Jobs [PDF]

Presented at the BPOD workshop of the IEEE Big Data 2020 conference [Presentation Video].

The individual files contain the experiment data for each job in tsv format.


Paper Abstract:

Analyzing large datasets with distributed dataflow systems requires the use of clusters. Public cloud providers offer a large variety and quantity of resources that can be used for such clusters. However, picking the appropriate resources in both type and number can often be challenging, as the selected configuration needs to match a distributed dataflow job's resource demands and access patterns. A good cluster configuration avoids hardware bottlenecks and maximizes resource utilization, avoiding costly overprovisioning.

We propose a collaborative approach for finding optimal cluster configurations based on sharing and learning from historical runtime data of distributed dataflow jobs. Collaboratively shared data can be utilized to predict runtimes of future job executions through the use of specialized regression models. However, training prediction models on historical runtime data that were produced by different users and in diverse contexts requires the models to take these contexts into account.


Cite Us:

@inproceedings{will2020towards,
    title={{Towards Collaborative Optimization of Cluster Configurations for Distributed Dataflow Jobs}},
    author={Will, Jonathan and Bader, Jonathan and Thamsen, Lauritz},
    booktitle={2020 IEEE International Conference on Big Data},
    year={2020},
    publisher={IEEE},
    isbn={978-1-7281-6251-5},
    pages={2851--2856},
}

About

Experiment data from the paper "Towards Collaborative Optimization of Cluster Configurations for Distributed Dataflow Jobs"

Resources

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published