Introduce VariadicAlias, remove hardcoded alias limits#6106
Introduce VariadicAlias, remove hardcoded alias limits#6106SherlockNoMad merged 4 commits intomasterfrom
Conversation
SherlockNoMad
added
the
training
| } | ||
|
|
||
| const nonstd::optional<std::pair<int, int>>& VariadicAlias() const { | ||
| return variadic_alias_offsets_; |
There was a problem hiding this comment.
variadic_alias_offsets_ [](start = 11, length = 23)
How would it be used to represent Lamb's schema such as [eta, w0, g0, m0, v0, w1, g1, m1, v1] -> [w0, m0, v0, w1, g1, v1]? I think we need an initial bias and a vector to specify the input-to-output mapping in a peridic pattern. For my example here, the vector is [0, -1, 1, 2] where '-1' mean 'g0' is not mapped to output.
There was a problem hiding this comment.
lamb is too special a case, I will leave it out, to make the interface clean and simple.
There was a problem hiding this comment.
Many optimizers may fall into that category, not only Lamb.
wschin
left a comment
wschin
left a comment
There was a problem hiding this comment.
This PR removes the input-output-length limit on several operators, which is good. I hope we can have a long-term solution for all operators with variadic-input/output list, but if this unblocks a production model, I am fine to merge this PR.
* Introduce VariadicAlias, remove hardcoded alias limits * Include optional-lite in winml build Co-authored-by: Sherlock Huang <bahuang@OrtTrainingDev3.af05slrtruoetgaxwwjv5nsq5e.px.internal.cloudapp.net> (cherry picked from commit a53f4dd)
3 participants
Description: Introduce VariadicAlias, remove hardcoded alias range limits