Skip to content

Commit

Permalink
Update Technical Report (#268)
Browse files Browse the repository at this point in the history
  • Loading branch information
ignite-abd committed Nov 1, 2023
1 parent 3f29a78 commit 8807be5
Show file tree
Hide file tree
Showing 3 changed files with 8 additions and 2 deletions.
2 changes: 1 addition & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -253,6 +253,6 @@ Please cite the repo if you use the code in this repo.
title = {Couler: Unified Machine Learning Workflow Optimization in Cloud},
year = {2023},
publisher = {GitHub},
howpublished = {\url{https://github.com/couler-proj/couler}}.
howpublished = {\url{https://github.com/couler-proj/couler/tree/master/docs/Technical-Report-of-Couler}}.
}
```
Binary file not shown.
8 changes: 7 additions & 1 deletion docs/Technical-Report-of-Couler/README.md
Original file line number Diff line number Diff line change
@@ -1 +1,7 @@
In this technical report, we delve into "Couler: Optimizing Machine Learning Workflows in Cloud", a framework designed to streamline the construction and execution of machine learning workflows. The report is segmented into three comprehensive chapters: Unified Programming Model, Implementation and Running Example.
Machine Learning (ML) has become ubiquitous, fueling data-driven applications across various organizations. Contrary to the traditional perception of ML in research, ML workflows can be complex, resource-intensive, and time-consuming.
Expanding an ML workflow to encompass a wider range of data infrastructure and data types may lead to larger workloads and increased deployment costs.
Currently, numerous workflow engines are available (with over ten being widely recognized). This variety poses a challenge for end-users in terms of mastering different engine APIs. While efforts have primarily focused on optimizing ML Operations (MLOps) for a specific workflow engine, current methods largely overlook workflow optimization across different engines.

In this work, we design and implement Couler, a system designed for unified ML workflow optimization in the cloud.
Our main insight lies in the ability to generate an ML workflow using natural language (NL) descriptions.
We integrate Large Language Models (LLMs) into workflow generation, and provide a unified programming interface for various workflow engines. This approach alleviates the need to understand various workflow engines' APIs. Moreover, Couler enhances workflow computation efficiency by introducing automated caching at multiple stages, enabling large workflow auto-parallelization and automatic hyperparameters tuning. These enhancements minimize redundant computational costs and improve fault tolerance during deep learning workflow training.

0 comments on commit 8807be5

Please sign in to comment.