Implement python API for AdaBelief optimizer #50157

juntang-zhuang · 2021-01-06T20:06:35Z

Fixes #47571 partially, add python API for AdaBelief optimizer.

facebook-github-bot · 2021-01-06T20:06:50Z

💊 CI failures summary and remediations

As of commit 92938c6 (more details on the Dr. CI page):

8/8 failures possibly* introduced in this PR
- 1/8 non-CircleCI failure(s)

🕵️ 7 new failures recognized by patterns

The following CI failures do not appear to be due to upstream breakages:

pytorch_linux_bionic_py3_8_gcc9_coverage_test1 (1/7)

Step: "Run tests" (full log | diagnosis details | 🔁 rerun)

Jan 06 22:09:03 AssertionError: mypy failed: torch/optim/_multi_tensor/adabelief.py:162: error: Incompatible types in assignment (expression has type "Union[Tuple[Tensor, ...], List[Tensor]]", variable has type "List[Any]")  [assignment]

Jan 06 22:08:42   test_type_hint_examples (__main__.TestTypeHints)
Jan 06 22:09:03 Runs mypy over all the test examples present in ... ok (20.698s)
Jan 06 22:09:03 
Jan 06 22:09:03 ======================================================================
Jan 06 22:09:03 FAIL [77.961s]: test_run_mypy (__main__.TestTypeHints)
Jan 06 22:09:03 Runs mypy over all files specified in mypy.ini
Jan 06 22:09:03 ----------------------------------------------------------------------
Jan 06 22:09:03 Traceback (most recent call last):
Jan 06 22:09:03   File "test_type_hints.py", line 214, in test_run_mypy
Jan 06 22:09:03     self.fail(f"mypy failed: {stdout} {stderr}")
Jan 06 22:09:03 AssertionError: mypy failed: torch/optim/_multi_tensor/adabelief.py:162: error: Incompatible types in assignment (expression has type "Union[Tuple[Tensor, ...], List[Tensor]]", variable has type "List[Any]")  [assignment]
Jan 06 22:09:03 torch/optim/_multi_tensor/adabelief.py:250: error: Argument 1 of "zero_grad" is incompatible with supertype "Optimizer"; supertype defines the argument type as "Optional[bool]"  [override]
Jan 06 22:09:03 torch/optim/_multi_tensor/adabelief.py:251: error: Need type annotation for 'per_device_and_dtype_grads'  [var-annotated]
Jan 06 22:09:03 Found 3 errors in 1 file (checked 1185 source files)
Jan 06 22:09:03  
Jan 06 22:09:03 
Jan 06 22:09:03 ----------------------------------------------------------------------
Jan 06 22:09:03 Ran 4 tests in 115.840s
Jan 06 22:09:03 
Jan 06 22:09:03 FAILED (failures=1)
Jan 06 22:09:03

pytorch_linux_xenial_cuda10_2_cudnn7_py3_gcc7_test1 (2/7)

Step: "Run tests" (full log | diagnosis details | 🔁 rerun)

Jan 06 22:19:50 AssertionError: mypy failed: torch/optim/_multi_tensor/adabelief.py:162: error: Incompatible types in assignment (expression has type "Union[Tuple[Tensor, ...], List[Tensor]]", variable has type "List[Any]")  [assignment]

Jan 06 22:19:29   test_run_mypy (__main__.TestTypeHints) ... FAIL (64.640s)
Jan 06 22:19:31   test_run_mypy_strict (__main__.TestTypeHints) ... ok (2.777s)
Jan 06 22:19:50   test_type_hint_examples (__main__.TestTypeHints) ... ok (18.458s)
Jan 06 22:19:50 
Jan 06 22:19:50 ======================================================================
Jan 06 22:19:50 FAIL [64.640s]: test_run_mypy (__main__.TestTypeHints)
Jan 06 22:19:50 ----------------------------------------------------------------------
Jan 06 22:19:50 Traceback (most recent call last):
Jan 06 22:19:50   File "test_type_hints.py", line 214, in test_run_mypy
Jan 06 22:19:50     self.fail(f"mypy failed: {stdout} {stderr}")
Jan 06 22:19:50 AssertionError: mypy failed: torch/optim/_multi_tensor/adabelief.py:162: error: Incompatible types in assignment (expression has type "Union[Tuple[Tensor, ...], List[Tensor]]", variable has type "List[Any]")  [assignment]
Jan 06 22:19:50 torch/optim/_multi_tensor/adabelief.py:250: error: Argument 1 of "zero_grad" is incompatible with supertype "Optimizer"; supertype defines the argument type as "Optional[bool]"  [override]
Jan 06 22:19:50 torch/optim/_multi_tensor/adabelief.py:251: error: Need type annotation for 'per_device_and_dtype_grads'  [var-annotated]
Jan 06 22:19:50 Found 3 errors in 1 file (checked 1185 source files)
Jan 06 22:19:50  
Jan 06 22:19:50 
Jan 06 22:19:50 ----------------------------------------------------------------------
Jan 06 22:19:50 Ran 4 tests in 97.674s
Jan 06 22:19:50 
Jan 06 22:19:50 FAILED (failures=1)
Jan 06 22:19:50

pytorch_doc_test (3/7)

Step: "Doc test" (full log | diagnosis details | 🔁 rerun)

Jan 06 22:02:18 sccache: error: couldn't connect to server

Jan 06 22:02:18 ++++ eval 'extract_trap_cmd '
Jan 06 22:02:18 +++++ extract_trap_cmd
Jan 06 22:02:18 +++++ printf '%s\n' ''
Jan 06 22:02:18 ++++ printf '%s\n' cleanup
Jan 06 22:02:18 +++ trap -- '
Jan 06 22:02:18 cleanup' EXIT
Jan 06 22:02:18 +++ [[ pytorch-linux-xenial-py3.6-gcc5.4-build != *pytorch-win-* ]]
Jan 06 22:02:18 +++ which sccache
Jan 06 22:02:18 +++ sccache --stop-server
Jan 06 22:02:18 Stopping sccache server...
Jan 06 22:02:18 sccache: error: couldn't connect to server
Jan 06 22:02:18 sccache: caused by: Connection refused (os error 111)
Jan 06 22:02:18 +++ true
Jan 06 22:02:18 +++ rm /var/lib/jenkins/sccache_error.log
Jan 06 22:02:18 +++ [[ pytorch-linux-xenial-py3.6-gcc5.4-build == *rocm* ]]
Jan 06 22:02:18 +++ SCCACHE_ERROR_LOG=/var/lib/jenkins/sccache_error.log
Jan 06 22:02:18 +++ SCCACHE_IDLE_TIMEOUT=1200
Jan 06 22:02:18 +++ RUST_LOG=sccache::server=error
Jan 06 22:02:18 +++ sccache --start-server
Jan 06 22:02:18 sccache: Starting the server...
Jan 06 22:02:19 +++ sccache --zero-stats

pytorch_linux_xenial_py3_6_gcc5_4_test (4/7)

Step: "Run tests" (full log | diagnosis details | 🔁 rerun)

Jan 06 22:06:15 AssertionError: mypy failed: torch/optim/_multi_tensor/adabelief.py:162: error: Incompatible types in assignment (expression has type "Union[Tuple[Tensor, ...], List[Tensor]]", variable has type "List[Any]")  [assignment]

Jan 06 22:05:49   test_run_mypy (__main__.TestTypeHints) ... FAIL (68.339s)
Jan 06 22:05:52   test_run_mypy_strict (__main__.TestTypeHints) ... ok (3.434s)
Jan 06 22:06:15   test_type_hint_examples (__main__.TestTypeHints) ... ok (22.495s)
Jan 06 22:06:15 
Jan 06 22:06:15 ======================================================================
Jan 06 22:06:15 FAIL [68.339s]: test_run_mypy (__main__.TestTypeHints)
Jan 06 22:06:15 ----------------------------------------------------------------------
Jan 06 22:06:15 Traceback (most recent call last):
Jan 06 22:06:15   File "test_type_hints.py", line 214, in test_run_mypy
Jan 06 22:06:15     self.fail(f"mypy failed: {stdout} {stderr}")
Jan 06 22:06:15 AssertionError: mypy failed: torch/optim/_multi_tensor/adabelief.py:162: error: Incompatible types in assignment (expression has type "Union[Tuple[Tensor, ...], List[Tensor]]", variable has type "List[Any]")  [assignment]
Jan 06 22:06:15 torch/optim/_multi_tensor/adabelief.py:250: error: Argument 1 of "zero_grad" is incompatible with supertype "Optimizer"; supertype defines the argument type as "Optional[bool]"  [override]
Jan 06 22:06:15 torch/optim/_multi_tensor/adabelief.py:251: error: Need type annotation for 'per_device_and_dtype_grads'  [var-annotated]
Jan 06 22:06:15 Found 3 errors in 1 file (checked 1189 source files)
Jan 06 22:06:15  
Jan 06 22:06:15 
Jan 06 22:06:15 ----------------------------------------------------------------------
Jan 06 22:06:15 Ran 4 tests in 106.299s
Jan 06 22:06:15 
Jan 06 22:06:15 FAILED (failures=1)
Jan 06 22:06:15

pytorch_python_doc_build (5/7)

Step: "Doc Build and Push" (full log | diagnosis details | 🔁 rerun)

Jan 06 22:15:12 Makefile:38: recipe for target 'html' failed

Jan 06 22:15:11 copying images... [ 96%] _static/img/tensorboard/add_hparam.png
Jan 06 22:15:11 copying images... [100%] scripts/activation_images/ReLU6.png
Jan 06 22:15:11 
Jan 06 22:15:12 copying static files... ... done
Jan 06 22:15:12 copying extra files... done
Jan 06 22:15:12 dumping search index in English (code: en)... done
Jan 06 22:15:12 dumping object inventory... done
Jan 06 22:15:12 build finished with problems, 4 warnings.
Jan 06 22:15:12 /var/lib/jenkins/workspace/docs/src/pytorch-sphinx-theme/pytorch_sphinx_theme/search.html:21: RemovedInSphinx30Warning: To modify script_files in the theme is deprecated. Please insert a <script> tag directly in your theme instead.
Jan 06 22:15:12   {% trans %}Please activate JavaScript to enable the search
Jan 06 22:15:12 Makefile:38: recipe for target 'html' failed
Jan 06 22:15:12 make: *** [html] Error 1


Exited with code exit status 2

pytorch_linux_xenial_py3_clang5_asan_test1 (6/7)

Step: "Run tests" (full log | diagnosis details | 🔁 rerun)

Jan 06 22:04:54 AssertionError: mypy failed: torch/optim/_multi_tensor/adabelief.py:162: error: Incompatible types in assignment (expression has type "Union[Tuple[Tensor, ...], List[Tensor]]", variable has type "List[Any]")  [assignment]

Jan 06 22:04:28   test_run_mypy (__main__.TestTypeHints) ... FAIL (69.953s)
Jan 06 22:04:31   test_run_mypy_strict (__main__.TestTypeHints) ... ok (3.298s)
Jan 06 22:04:54   test_type_hint_examples (__main__.TestTypeHints) ... ok (22.656s)
Jan 06 22:04:54 
Jan 06 22:04:54 ======================================================================
Jan 06 22:04:54 FAIL [69.953s]: test_run_mypy (__main__.TestTypeHints)
Jan 06 22:04:54 ----------------------------------------------------------------------
Jan 06 22:04:54 Traceback (most recent call last):
Jan 06 22:04:54   File "test_type_hints.py", line 214, in test_run_mypy
Jan 06 22:04:54     self.fail(f"mypy failed: {stdout} {stderr}")
Jan 06 22:04:54 AssertionError: mypy failed: torch/optim/_multi_tensor/adabelief.py:162: error: Incompatible types in assignment (expression has type "Union[Tuple[Tensor, ...], List[Tensor]]", variable has type "List[Any]")  [assignment]
Jan 06 22:04:54 torch/optim/_multi_tensor/adabelief.py:250: error: Argument 1 of "zero_grad" is incompatible with supertype "Optimizer"; supertype defines the argument type as "Optional[bool]"  [override]
Jan 06 22:04:54 torch/optim/_multi_tensor/adabelief.py:251: error: Need type annotation for 'per_device_and_dtype_grads'  [var-annotated]
Jan 06 22:04:54 Found 3 errors in 1 file (checked 1185 source files)
Jan 06 22:04:54  
Jan 06 22:04:54 
Jan 06 22:04:54 ----------------------------------------------------------------------
Jan 06 22:04:54 Ran 4 tests in 108.797s
Jan 06 22:04:54 
Jan 06 22:04:54 FAILED (failures=1)
Jan 06 22:04:54

pytorch_linux_bionic_py3_6_clang9_test (7/7)

Step: "Run tests" (full log | diagnosis details | 🔁 rerun)

Jan 06 22:02:47 AssertionError: mypy failed: torch/optim/_multi_tensor/adabelief.py:162: error: Incompatible types in assignment (expression has type "Union[Tuple[Tensor, ...], List[Tensor]]", variable has type "List[Any]")  [assignment]

Jan 06 22:02:25   test_run_mypy (__main__.TestTypeHints) ... FAIL (60.092s)
Jan 06 22:02:28   test_run_mypy_strict (__main__.TestTypeHints) ... ok (2.858s)
Jan 06 22:02:47   test_type_hint_examples (__main__.TestTypeHints) ... ok (19.526s)
Jan 06 22:02:47 
Jan 06 22:02:47 ======================================================================
Jan 06 22:02:47 FAIL [60.092s]: test_run_mypy (__main__.TestTypeHints)
Jan 06 22:02:47 ----------------------------------------------------------------------
Jan 06 22:02:47 Traceback (most recent call last):
Jan 06 22:02:47   File "test_type_hints.py", line 214, in test_run_mypy
Jan 06 22:02:47     self.fail(f"mypy failed: {stdout} {stderr}")
Jan 06 22:02:47 AssertionError: mypy failed: torch/optim/_multi_tensor/adabelief.py:162: error: Incompatible types in assignment (expression has type "Union[Tuple[Tensor, ...], List[Tensor]]", variable has type "List[Any]")  [assignment]
Jan 06 22:02:47 torch/optim/_multi_tensor/adabelief.py:250: error: Argument 1 of "zero_grad" is incompatible with supertype "Optimizer"; supertype defines the argument type as "Optional[bool]"  [override]
Jan 06 22:02:47 torch/optim/_multi_tensor/adabelief.py:251: error: Need type annotation for 'per_device_and_dtype_grads'  [var-annotated]
Jan 06 22:02:47 Found 3 errors in 1 file (checked 1185 source files)
Jan 06 22:02:47  
Jan 06 22:02:47 
Jan 06 22:02:47 ----------------------------------------------------------------------
Jan 06 22:02:47 Ran 4 tests in 93.505s
Jan 06 22:02:47 
Jan 06 22:02:47 FAILED (failures=1)
Jan 06 22:02:47

ci.pytorch.org: 1 failed

Failed: pr/caffe2-pytorch-linux-bionic-rocm3.10-py3.6-test

This comment was automatically generated by Dr. CI (expand for details).

Follow this link to opt-out of these comments for your Pull Requests.

Please report bugs/suggestions to the (internal) Dr. CI Users group.

…pytorch into adabelief_python

glaringlee · 2021-01-06T21:07:44Z

@juntang-zhuang
Hey Juntang, please see the comment from my colleague who is working on python side of pytorch. It seems we have an adoption rule for new features on python side which I am not aware of. Sry about this, but I think this is a good feature as the paper is already there, more and more people will use it and we probably can adopt this into pytorch main branch later I think.

here is the link: #47571

juntang-zhuang · 2021-01-06T21:13:54Z

@glaringlee Thanks a lot for the update. I'll just keep the changes in my local repository so we can consider merging it later.

glaringlee · 2021-01-06T21:16:19Z

@glaringlee Thanks a lot for the update. I'll just keep the changes in my local repository so we can consider merging it later.

Sure, let's revisit this in the middle of the year see whether you have more citations, or if you optimizer is used by a large facility (eg, some big companies?), I think that would be a big plus. let's see.

iramazanli

Thanks for proposing AdaBelief implementation to PyTorch Core Library.

We would want to refer to rulebook for adding optimizers to PyTorch Core:
https://github.com/pytorch/pytorch/wiki/Developer-FAQ#i-have-implemented-an-algorithm--feature-from-a-research-paper-that-is-related-to-optimization-when-is-the-correct-time-to-add-it-to-pytorch-core-

Which implies although AdaBelief is a very promising algorithm, given that it's relatively new algorithm we cannot add it to Core Library yet.

It can be interesting discussion to return back to this issue in the following years.

rebase_master

3b0d30b

facebook-github-bot added the cla signed label Jan 6, 2021

pytorchbot added the open source label Jan 6, 2021

juntang-zhuang added 2 commits January 6, 2021 16:01

rebase_master

d34c537

Merge branch 'adabelief_python' of https://github.com/juntang-zhuang/…

04cca20

…pytorch into adabelief_python

remove_blank_line

92938c6

H-Huang added module: optimizer Related to torch.optim triaged This issue has been looked at a team member, and triaged and prioritized into an appropriate module labels Jan 6, 2021

spandantiwari mentioned this pull request Jan 12, 2021

[ONNX] ONNX dev branch merge 01-06-2021 #50163

Closed

iramazanli self-requested a review August 30, 2021 15:27

iramazanli reviewed Aug 30, 2021

View reviewed changes

iramazanli closed this Aug 30, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Implement python API for AdaBelief optimizer #50157

Implement python API for AdaBelief optimizer #50157

juntang-zhuang commented Jan 6, 2021

facebook-github-bot commented Jan 6, 2021 •

edited

glaringlee commented Jan 6, 2021 •

edited

juntang-zhuang commented Jan 6, 2021

glaringlee commented Jan 6, 2021 •

edited

iramazanli left a comment

Implement python API for AdaBelief optimizer #50157

Implement python API for AdaBelief optimizer #50157

Conversation

juntang-zhuang commented Jan 6, 2021

facebook-github-bot commented Jan 6, 2021 • edited

💊 CI failures summary and remediations

🕵️ 7 new failures recognized by patterns

pytorch_linux_bionic_py3_8_gcc9_coverage_test1 (1/7)

pytorch_linux_xenial_cuda10_2_cudnn7_py3_gcc7_test1 (2/7)

pytorch_doc_test (3/7)

pytorch_linux_xenial_py3_6_gcc5_4_test (4/7)

pytorch_python_doc_build (5/7)

pytorch_linux_xenial_py3_clang5_asan_test1 (6/7)

pytorch_linux_bionic_py3_6_clang9_test (7/7)

ci.pytorch.org: 1 failed

glaringlee commented Jan 6, 2021 • edited

juntang-zhuang commented Jan 6, 2021

glaringlee commented Jan 6, 2021 • edited

iramazanli left a comment

Choose a reason for hiding this comment

facebook-github-bot commented Jan 6, 2021 •

edited

glaringlee commented Jan 6, 2021 •

edited

glaringlee commented Jan 6, 2021 •

edited