Skip to content

Commit

Permalink
minor update
Browse files Browse the repository at this point in the history
  • Loading branch information
hosiet committed Aug 28, 2023
1 parent d2f6754 commit 3bec8c1
Show file tree
Hide file tree
Showing 3 changed files with 21 additions and 1 deletion.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
2 changes: 1 addition & 1 deletion config/_default/params.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -69,7 +69,7 @@ features:
- r
- latex
math:
enable: false
enable: true
privacy_pack:
enable: false
repository:
Expand Down
20 changes: 20 additions & 0 deletions content/publication/2023-elastictrainer/index.md
Original file line number Diff line number Diff line change
Expand Up @@ -79,8 +79,28 @@ us to transform to online and fully elastic selection of trainable NN portion.

![Opportunities of training speedup](2023-elastictrainer/elastictrainer-fig2.png)

## Granuliarity of Selection

The optimality of selecting the trainable NN portion is affected by the granularity
of selection, which can be at the level of weights, tensors and layers on most NNs.
ElasticTrainer adopts tensor-level selection, which ensures accuracy and can also
be efficiently executed in existing NN frameworks without extra overhead.

![Granularities of Selection](2023-elastictrainer/elastictrainer-fig4.png)

## System Overview

ElasticTrainer’s design aims to select the optimal
trainable NN portion at runtime, to achieve the desired training
speedup with the maximum training loss reduction.
In the offline stage, ElasticTrainer uses a *Tensor Timing Profiler* to
profile the training times of selected tensors, to provide inputs for
calculating {{< math >}}$ T_{sel}(M) ${{< /math >}}.
In the online stage, the system builds on an accurate yet computationally efficient metric
that evaluates the aggregate importance of selected tensors and the
corresponding reduction of training loss, and such evaluation is
done by *Tensor Importance Evaluator* in ElasticTrainer design.

![ElasticTrainer System Overview](2023-elastictrainer/elastictrainer-system-overview.png)

## Teaser Video
Expand Down

0 comments on commit 3bec8c1

Please sign in to comment.