[Relax][Training] Add automatic differentiation pass #103
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
6 tasks
20 tasks
debug finished Change details on and modify document polish test unit test finished reformat changed files Fix problems in pr comments move op and change them static in AD pass fix some problems fix for comments so far update documents formatted update document1 draft 1 revise version 1 version 1 completed and formatted revise on doc and detail polish after refactor remove zeros_tracker_ update doc restructure and add float check for input format refactor done refactor again add test_tuple_ops update comments rebase onto struct info fix log_softmax case fix test eliminator draft fix after rebase refactor draft Revise doc & Epilogue & details remove some irrelevant codes remove collapse_sum_eliminator
normalize has problems to fix
SiriusNEO
force-pushed
the
mlc-dev/2023-01-18-ad_after_tuple_refactor
branch
from
f2364e4
to
da9f2d7
Compare
SiriusNEO
force-pushed
the
mlc-dev/2023-01-18-ad_after_tuple_refactor
branch
from
2eafa15
to
ac98e81
Compare
SiriusNEO
force-pushed
the
mlc-dev/2023-01-18-ad_after_tuple_refactor
branch
from
bb035e4
to
cd34f4d
Compare
Ubospica
force-pushed
the
mlc-dev/2023-01-18-ad_after_tuple_refactor
branch
from
f6d5622
to
79a9e94
Compare
Ubospica
changed the title
Ubospica
force-pushed
the
mlc-dev/2023-01-18-ad_after_tuple_refactor
branch
from
79a9e94
to
ad2495c
Compare
Ubospica
force-pushed
the
mlc-dev/2023-01-18-ad_after_tuple_refactor
branch
from
ad2495c
to
7eb39f8
Compare
SiriusNEO
reviewed
Jan 20, 2023
modify nested_msg.h, util.h add testcases
Ubospica
force-pushed
the
mlc-dev/2023-01-18-ad_after_tuple_refactor
branch
from
9421fff
to
9011549
Compare
Ubospica
force-pushed
the
mlc-dev/2023-01-18-ad_after_tuple_refactor
branch
from
9011549
to
d2854c6
Compare
Ubospica
force-pushed
the
mlc-dev/2023-01-18-ad_after_tuple_refactor
branch
from
3b56962
to
0cdd5dc
Compare
Ubospica
force-pushed
the
mlc-dev/2023-01-18-ad_after_tuple_refactor
branch
2 times, most recently
from
c6c10dc
to
e5f4504
Compare
Ubospica
force-pushed
the
mlc-dev/2023-01-18-ad_after_tuple_refactor
branch
from
e5f4504
to
0389aaf
Compare
SiriusNEO
force-pushed
the
mlc-dev/2023-01-18-ad_after_tuple_refactor
branch
from
04572e8
to
3c0d8fc
Compare
MasterJH5574
approved these changes
Jan 23, 2023
spectrometerHBH
approved these changes
Jan 23, 2023
|
Thanks for the great efforts in pushing this PR! Would you folks mind sending the changes of nested msg together with the test in this PR to tlc-pack when getting time? That would ease our next sync. But no hurry at all - enjoy your holidays :-) |
MasterJH5574
pushed a commit
that referenced
this pull request
Jan 24, 2023
This PR is a small fix patch for #103, containing two small modifications: - In AD PR we introduce `NestedMsgToExpr`, where the signature of `fmapleaf` is `Expr fmapleaf(T)`. And for null nested msg it will directly throws error. Now we change it to `Expr fmapleaf(Optional<T>)` and pass `NullOpt` to `fmapleaf`, which enables user to decide whether to throw an error or return some default value. - In the test of nested msg we forget to change the signature of `fmapleaf` (originally it is `Expr fmapleaf(NestedMsg<T>)`). It still passed due to the type casting between `NestedMsg<T>` and `T` but it needs to be fixed.
MasterJH5574
pushed a commit
that referenced
this pull request
Jan 28, 2023
This is the PR following #55 after source branch moved to personal repo. This PR is based on #98. This PR adds the new automatic differentiation API: - `Gradient(func: GlobalVar, require_grads: Optional[Union[Var, List[Var]]] = None) -> tvm.ir.transform.Pass` - transforms the given funcion in the IRModule, and adds a new function that calculates the gradient with regard to the function's output Now Gradient only supports differentiating a function in the IRModule with one dataflow block with respect to the only return value of the function, which needs to be scalar. This PR writes two files for unit test: - `tests/python/relax/test_transform_gradient.py` only contains `assert_structural_equal` assertions. - `tests/python/relax/test_transform_gradient_numeric.py` contains numeric checks, including manually derived gradients and the numerical differentiation method `check_numerical_grads`. Checkpoints: - [x] Refactor to use CopyWithNewParams and ExprFunctor - [x] Check int64/int32 tensors should not be differentiated (now only check in params) - [x] Rebase & migrate to StructInfo - [x] Refactor about Tuple - [x] Refactor about NestedMsg - [x] Support ops taking in tuple or returning tuple - [x] Eliminating collapse_sum_to (done in #98) Future: - (Not in this PR) Handle undefined gradient in add and return value - Now we handle them as zeros Co-authored-by: SiriusNEO <1713833595@qq.com>
MasterJH5574
pushed a commit
that referenced
this pull request
Jan 28, 2023
This PR is a small fix patch for #103, containing two small modifications: - In AD PR we introduce `NestedMsgToExpr`, where the signature of `fmapleaf` is `Expr fmapleaf(T)`. And for null nested msg it will directly throws error. Now we change it to `Expr fmapleaf(Optional<T>)` and pass `NullOpt` to `fmapleaf`, which enables user to decide whether to throw an error or return some default value. - In the test of nested msg we forget to change the signature of `fmapleaf` (originally it is `Expr fmapleaf(NestedMsg<T>)`). It still passed due to the type casting between `NestedMsg<T>` and `T` but it needs to be fixed.
MasterJH5574
pushed a commit
that referenced
this pull request
Jan 31, 2023
This is the PR following #55 after source branch moved to personal repo. This PR is based on #98. This PR adds the new automatic differentiation API: - `Gradient(func: GlobalVar, require_grads: Optional[Union[Var, List[Var]]] = None) -> tvm.ir.transform.Pass` - transforms the given funcion in the IRModule, and adds a new function that calculates the gradient with regard to the function's output Now Gradient only supports differentiating a function in the IRModule with one dataflow block with respect to the only return value of the function, which needs to be scalar. This PR writes two files for unit test: - `tests/python/relax/test_transform_gradient.py` only contains `assert_structural_equal` assertions. - `tests/python/relax/test_transform_gradient_numeric.py` contains numeric checks, including manually derived gradients and the numerical differentiation method `check_numerical_grads`. Checkpoints: - [x] Refactor to use CopyWithNewParams and ExprFunctor - [x] Check int64/int32 tensors should not be differentiated (now only check in params) - [x] Rebase & migrate to StructInfo - [x] Refactor about Tuple - [x] Refactor about NestedMsg - [x] Support ops taking in tuple or returning tuple - [x] Eliminating collapse_sum_to (done in #98) Future: - (Not in this PR) Handle undefined gradient in add and return value - Now we handle them as zeros Co-authored-by: SiriusNEO <1713833595@qq.com>
MasterJH5574
pushed a commit
that referenced
this pull request
Feb 8, 2023
This is the PR following #55 after source branch moved to personal repo. This PR is based on #98. This PR adds the new automatic differentiation API: - `Gradient(func: GlobalVar, require_grads: Optional[Union[Var, List[Var]]] = None) -> tvm.ir.transform.Pass` - transforms the given funcion in the IRModule, and adds a new function that calculates the gradient with regard to the function's output Now Gradient only supports differentiating a function in the IRModule with one dataflow block with respect to the only return value of the function, which needs to be scalar. This PR writes two files for unit test: - `tests/python/relax/test_transform_gradient.py` only contains `assert_structural_equal` assertions. - `tests/python/relax/test_transform_gradient_numeric.py` contains numeric checks, including manually derived gradients and the numerical differentiation method `check_numerical_grads`. Checkpoints: - [x] Refactor to use CopyWithNewParams and ExprFunctor - [x] Check int64/int32 tensors should not be differentiated (now only check in params) - [x] Rebase & migrate to StructInfo - [x] Refactor about Tuple - [x] Refactor about NestedMsg - [x] Support ops taking in tuple or returning tuple - [x] Eliminating collapse_sum_to (done in #98) Future: - (Not in this PR) Handle undefined gradient in add and return value - Now we handle them as zeros Co-authored-by: SiriusNEO <1713833595@qq.com>
MasterJH5574
pushed a commit
that referenced
this pull request
Feb 12, 2023
This is the PR following #55 after source branch moved to personal repo. This PR is based on #98. This PR adds the new automatic differentiation API: - `Gradient(func: GlobalVar, require_grads: Optional[Union[Var, List[Var]]] = None) -> tvm.ir.transform.Pass` - transforms the given funcion in the IRModule, and adds a new function that calculates the gradient with regard to the function's output Now Gradient only supports differentiating a function in the IRModule with one dataflow block with respect to the only return value of the function, which needs to be scalar. This PR writes two files for unit test: - `tests/python/relax/test_transform_gradient.py` only contains `assert_structural_equal` assertions. - `tests/python/relax/test_transform_gradient_numeric.py` contains numeric checks, including manually derived gradients and the numerical differentiation method `check_numerical_grads`. Checkpoints: - [x] Refactor to use CopyWithNewParams and ExprFunctor - [x] Check int64/int32 tensors should not be differentiated (now only check in params) - [x] Rebase & migrate to StructInfo - [x] Refactor about Tuple - [x] Refactor about NestedMsg - [x] Support ops taking in tuple or returning tuple - [x] Eliminating collapse_sum_to (done in #98) Future: - (Not in this PR) Handle undefined gradient in add and return value - Now we handle them as zeros Co-authored-by: SiriusNEO <1713833595@qq.com>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This is the PR following #55 after source branch moved to personal repo.
This PR is based on #98.
This PR adds the new automatic differentiation API:
Gradient(func: GlobalVar, require_grads: Optional[Union[Var, List[Var]]] = None) -> tvm.ir.transform.PassNow Gradient only supports differentiating a function in the IRModule with one dataflow block with respect to the only return value of the function, which needs to be scalar.
This PR writes two files for unit test:
tests/python/relax/test_transform_gradient.pyonly containsassert_structural_equalassertions.tests/python/relax/test_transform_gradient_numeric.pycontains numeric checks, including manually derived gradients and the numerical differentiation methodcheck_numerical_grads.Checkpoints:
Future: