Fix upsample sbp infer bug and add global test #7884
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
clackhan
commented
Mar 23, 2022
|
|
||
| # backward compute result of oneflow is not same with pytorch | ||
| @autotest(n=1, auto_backward=False, check_graph=False) | ||
| def _test_global_upsample2d_bicubic(test_case, placement, sbp): |
There was a problem hiding this comment.
bicubic模式下，oneflow 后向计算结果与pytorch对不上，不知道是否是在实现上有差异
复现命令：设置```auto_backward=True````,
python test_consistent_upsample.py --verbose --failfast
There was a problem hiding this comment.
这里需要确定是否真的对不上，我们要以Pytorch1.11为准。如果确实对不上，那么就更新到：https://github.com/Oneflow-Inc/OneTeam/issues/1207#issuecomment-1073432125 ，我来debug。
There was a problem hiding this comment.
这里需要确定是否真的对不上，我们要以Pytorch1.11为准。如果确实对不上，那么就更新到：Oneflow-Inc/OneTeam#1207 (comment) ，我来debug。
升级pytorch到1.11后（原本是在1.10下侧的），后向计算结果仍然不一样，已更新在Oneflow-Inc/OneTeam#1207 (comment) 中
There was a problem hiding this comment.
@clackhan 此bug已在https://github.com/Oneflow-Inc/oneflow/pull/7916中修复。
合并 pr7916 后，打开后向测试，直接 Abroted 的了，关闭后向没有问题，可以正常跑，报错信息如下：
python test_consistent_upsample.py --verbose --failfast
test_global_upsample2d_bicubic (__main__.TestGlobalUpsample2d) ... Environment has been initialized, this env init will do nothing.
/home/hanbinbin/anaconda3/envs/oneflow/lib/python3.8/site-packages/torch/_tensor.py:1104: UserWarning: The .grad attribute of a Tensor that is not a leaf Tensor is being accessed. Its .grad attribute won't be populated during autograd.backward(). If you indeed want the .grad field to be populated for a non-leaf Tensor, use .retain_grad() on the non-leaf Tensor. If you access the non-leaf Tensor by mistake, make sure you access the leaf Tensor instead. See github.com/pytorch/pytorch/pull/30531 for more informations. (Triggered internally at aten/src/ATen/core/TensorBody.h:475.)
return self._grad
free(): corrupted unsorted chunks
Aborted (core dumped)…add_var_upsample_global_test
clackhan
commented
Mar 24, 2022
Comment on lines
+79
to
+83
| @unittest.skip( | ||
| "The nearest interpolate operation in pytorch has bug, https://github.com/pytorch/pytorch/issues/65200" | ||
| ) | ||
| @globaltest | ||
| def test_global_upsample2d_nearest(test_case): |
There was a problem hiding this comment.
这个里pytroch中的issue已经close了，但是这测结果还是和pytorch对不上，不知道只oneflow的问题还是pytorch的问题
复现命令：注释@unittest.skip，
python test_consistent_upsample.py --verbose --failfast
There was a problem hiding this comment.
可能和我们CI环境下的PyTorch版本比较旧有关，这里暂时也和Local一样skip吧
There was a problem hiding this comment.
之前我和盛航在升级的过程中碰到了很多问题，还没有来得及一一解决。
There was a problem hiding this comment.
我这边重复测试了一下，发现原因是因为pytorch的cpu和gpu结果在缩放系数不是整数情况下跑出的结果对不上。这个bug我之前确实反馈了，但pytorch不修就直接把我issue关了，这个问题先不管吧。
wyg1997
approved these changes
Apr 6, 2022
|
View latest API docs preview at: https://staging.oneflow.info/docs/Oneflow-Inc/oneflow/pr/7884/ |
|
CI failed when running job: cpu-misc. PR label automerge has been removed |
Speed stats: |
|
View latest API docs preview at: https://staging.oneflow.info/docs/Oneflow-Inc/oneflow/pr/7884/ |
Speed stats: |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
添加var 和 upsample global 测试