Update test_cuda.py and test_torch.py optim tests to use OptimizerInfo and optim_db #123451

janeyx99 · 2024-04-05T16:11:50Z

🚀 The feature, motivation and pitch

Update: all tasks below have in progress PRs currently!

In test_cuda.py and test_torch.py, we have the following optimizer tests that should be migrated to use the new OptimizerInfo infrastructure. (See any of the tests in TestOptimRenewed for how to apply OptimizerInfos.) The basic structure to expect for an updated test will look like:

    @optims([optim for optim in optim_db if ...], dtypes=[torch.float32])
    def test_name(self, device, dtype, optim_info):
        optim_cls = optim_info.optim_cls
        optim_inputs = optim_info.optim_inputs_func(device=device)
        for optim_input in optim_inputs:
            params = ...
            // processing, like updating grads

            optimizer = optim_cls(params, ..., **optim_input.kwargs)

            // the actual test
            ...
            self.assertEquals(...)

test_cuda.py:

test_grad_scaling_autocast_fused_optimizers will be addressed in [optim]fix ut and sgd kernel #124904 and by @SandishKumarHN
test_graph_grad_scaling
test_graph_optims
test_graph_optims_with_explicitly_capturable_param_groups
test_graph_scaling_fused_optimizers

test_torch.py: #125538

tests that call _grad_scaling_autocast_test can be combined into one test
test_params_invalidated_with_grads_invalidated_between_unscale_and_step should be tested on more configurations (like SGD fused!)
there are many other tests that test just one optimizer config that we can leave for now.

Feel free to take up one or a subset of those tests to migrate--edit your username next to the checklist to claim an item to submit PRs for!

Alternatives

No response

Additional context

No response

cc @vincentqb @jbschlosser @albanD @crcrpar

The text was updated successfully, but these errors were encountered:

rithikp06 · 2024-04-05T22:33:10Z

I'd like to work on this

janeyx99 · 2024-04-05T23:21:10Z

@rithikp06 feel free to claim a sub task (e.g., which tests are you migrating?) and open a PR!

rithikp06 · 2024-04-05T23:25:20Z

I'll work on the test_torch.py tests.

ComposeC · 2024-04-07T13:23:49Z

@janeyx99 I'd like to work on the test_grad_scaling_autocast_fused_optimizers function in test_cuda.py.

rithikp06 · 2024-04-08T01:26:03Z

I created a PR that addresses the test_torch.py part:
#123537

FireACNC · 2024-04-08T14:30:55Z

I'd like to work on the test_graph_grad_scaling function in test_cuda.py.

jayanthd04 · 2024-04-08T15:48:43Z

Hi, I'm a first time contributor. I would like to work on test_graph_optims and test_graph_scaling_fused_optimizers. Could i get some pointer on where I could find more information about OptimizerInfo infrastructure and TestOptimRenewed?

dwang3851 · 2024-04-08T17:09:31Z

I'd like to work on the test_graph_optims_with_explicitly_capturable_param_groups function in test_cuda.py

khlaifiabilel · 2024-04-08T22:57:03Z

Greetings 👋 , This is very interesting
I would like to work on the test_graph_optims in the test_cuda.py

janeyx99 · 2024-04-16T03:11:53Z

Hi, I'm a first time contributor. I would like to work on test_graph_optims and test_graph_scaling_fused_optimizers. Could i get some pointer on where I could find more information about OptimizerInfo infrastructure and TestOptimRenewed?

TestOptimRenewed is a class full of tests that use the OptimizerInfo infrastructure: https://github.com/pytorch/pytorch/blob/main/test/test_optim.py#L44

The OptimizerInfo infrastructure is defined in common_optimizers.py, you would want to search for optims_db and OptimizerInfo for what they are. I think looking at an existing test (any of the ones in TestOptimRenewed) would give a good highlevel of how these are used.

julian-8897 · 2024-04-19T21:23:35Z

HI there, I'm a first time contributor here :). I would like to work on the test for _grad_scaling_autocast_test under test_torch.py

PetaBread545 · 2024-04-20T21:23:41Z

Hello, I am also a first time contributor. Seeing that #123537 already addresses the test_py subtasks, I would like to work on the test_cuda subtasks.

dwang3851 · 2024-04-21T03:59:26Z

Hi, I'm a first time contributor working on this issue. Is there any way to approve Github workflows for me on the pr above, I was hoping to make sure those results matched my local results?

omastey · 2024-04-22T18:51:00Z

Hi, I have begun working on this ticket, are there any specs or documentation I could look at?

Support fused_sgd_kernel support for CPU. ## Bench result: 32 core/sockets ICX Test Scripts: https://gist.github.com/zhuhaozhe/688763e17e93e4c5e12f25f676ec90d9 https://gist.github.com/zhuhaozhe/ad9938694bc7fae8b66d376f4dffc6c9 ``` Tensor Size: 262144, Num Tensor 4, Num Threads: 1 _single_tensor_sgd time: 0.2301 seconds _fused_sgd time: 0.0925 seconds Tensor Size: 4194304, Num Tensor 32, Num Threads: 32 _single_tensor_sgd time: 2.6195 seconds _fused_sgd time: 1.7543 seconds ``` ## Test Plan: ``` python test_optim.py -k test_fused_matches_forloop python test_optim.py -k test_fused_large_tensor python test_optim.py -k test_can_load_older_state_dict python test_optim.py -k test_grad_scaling_autocast_fused_optimizers python test_torch.py -k test_grad_scaling_autocast_fused python test_torch.py -k test_params_invalidated_with_grads_invalidated_between_unscale_and_step ``` Looks like we already have some PRs under this issue #123451 to unified the UTs, I did not modified UT in this PR. Co-authored-by: Jane Xu <janeyx@meta.com> Pull Request resolved: #123629 Approved by: https://github.com/jgong5, https://github.com/janeyx99

Rbhu376264 · 2024-04-24T01:48:18Z

Hello There,
Hope this message finds you well!!!
I am a newbie to open-source contribution, and would really like to get some advice how to get started in contributing "test_graph_scaling_fused_optimizers" . How can I get started on this? Any suggestions would be highly appreciated!!!

…re (#123581) This PR targets the issue mentioned in #123451 , and solves the specific task to update`test_graph_grad_scaling` in `test/test_cuda.py` to use the new OptimizerInfo infrastructure. `test_graph_grad_scaling` is moved to a new `TestCase` class called `TestCudaOptims` in order to use `instantiate_device_type_tests`. The test content remained the same. `@onlyCUDA` is applied to the new test; the original use of the wrapper function is also changed to a `@parametrize` decorator for better style. If we think that this migration is successful, we can delete the original test item under `TestCuda`. Currently it is left untouched to avoid any unexpected issues. Local linter passed. ``` $ lintrunner test/test_cuda.py ok No lint issues. ``` Local tests passed. ``` > python .\test\test_cuda.py -k test_graph_grad_scaling Ran 7 tests in 0.458s OK (skipped = 3) ``` Co-authored-by: Jane (Yuan) Xu <31798555+janeyx99@users.noreply.github.com> Pull Request resolved: #123581 Approved by: https://github.com/janeyx99

…re (pytorch#123581) This PR targets the issue mentioned in pytorch#123451 , and solves the specific task to update`test_graph_grad_scaling` in `test/test_cuda.py` to use the new OptimizerInfo infrastructure. `test_graph_grad_scaling` is moved to a new `TestCase` class called `TestCudaOptims` in order to use `instantiate_device_type_tests`. The test content remained the same. `@onlyCUDA` is applied to the new test; the original use of the wrapper function is also changed to a `@parametrize` decorator for better style. If we think that this migration is successful, we can delete the original test item under `TestCuda`. Currently it is left untouched to avoid any unexpected issues. Local linter passed. ``` $ lintrunner test/test_cuda.py ok No lint issues. ``` Local tests passed. ``` > python .\test\test_cuda.py -k test_graph_grad_scaling Ran 7 tests in 0.458s OK (skipped = 3) ``` Co-authored-by: Jane (Yuan) Xu <31798555+janeyx99@users.noreply.github.com> Pull Request resolved: pytorch#123581 Approved by: https://github.com/janeyx99

Support fused_sgd_kernel support for CPU. ## Bench result: 32 core/sockets ICX Test Scripts: https://gist.github.com/zhuhaozhe/688763e17e93e4c5e12f25f676ec90d9 https://gist.github.com/zhuhaozhe/ad9938694bc7fae8b66d376f4dffc6c9 ``` Tensor Size: 262144, Num Tensor 4, Num Threads: 1 _single_tensor_sgd time: 0.2301 seconds _fused_sgd time: 0.0925 seconds Tensor Size: 4194304, Num Tensor 32, Num Threads: 32 _single_tensor_sgd time: 2.6195 seconds _fused_sgd time: 1.7543 seconds ``` ## Test Plan: ``` python test_optim.py -k test_fused_matches_forloop python test_optim.py -k test_fused_large_tensor python test_optim.py -k test_can_load_older_state_dict python test_optim.py -k test_grad_scaling_autocast_fused_optimizers python test_torch.py -k test_grad_scaling_autocast_fused python test_torch.py -k test_params_invalidated_with_grads_invalidated_between_unscale_and_step ``` Looks like we already have some PRs under this issue pytorch#123451 to unified the UTs, I did not modified UT in this PR. Co-authored-by: Jane Xu <janeyx@meta.com> Pull Request resolved: pytorch#123629 Approved by: https://github.com/jgong5, https://github.com/janeyx99

…re (pytorch#123581) This PR targets the issue mentioned in pytorch#123451 , and solves the specific task to update`test_graph_grad_scaling` in `test/test_cuda.py` to use the new OptimizerInfo infrastructure. `test_graph_grad_scaling` is moved to a new `TestCase` class called `TestCudaOptims` in order to use `instantiate_device_type_tests`. The test content remained the same. `@onlyCUDA` is applied to the new test; the original use of the wrapper function is also changed to a `@parametrize` decorator for better style. If we think that this migration is successful, we can delete the original test item under `TestCuda`. Currently it is left untouched to avoid any unexpected issues. Local linter passed. ``` $ lintrunner test/test_cuda.py ok No lint issues. ``` Local tests passed. ``` > python .\test\test_cuda.py -k test_graph_grad_scaling Ran 7 tests in 0.458s OK (skipped = 3) ``` Co-authored-by: Jane (Yuan) Xu <31798555+janeyx99@users.noreply.github.com> Pull Request resolved: pytorch#123581 Approved by: https://github.com/janeyx99

Support fused_sgd_kernel support for CPU. ## Bench result: 32 core/sockets ICX Test Scripts: https://gist.github.com/zhuhaozhe/688763e17e93e4c5e12f25f676ec90d9 https://gist.github.com/zhuhaozhe/ad9938694bc7fae8b66d376f4dffc6c9 ``` Tensor Size: 262144, Num Tensor 4, Num Threads: 1 _single_tensor_sgd time: 0.2301 seconds _fused_sgd time: 0.0925 seconds Tensor Size: 4194304, Num Tensor 32, Num Threads: 32 _single_tensor_sgd time: 2.6195 seconds _fused_sgd time: 1.7543 seconds ``` ## Test Plan: ``` python test_optim.py -k test_fused_matches_forloop python test_optim.py -k test_fused_large_tensor python test_optim.py -k test_can_load_older_state_dict python test_optim.py -k test_grad_scaling_autocast_fused_optimizers python test_torch.py -k test_grad_scaling_autocast_fused python test_torch.py -k test_params_invalidated_with_grads_invalidated_between_unscale_and_step ``` Looks like we already have some PRs under this issue pytorch#123451 to unified the UTs, I did not modified UT in this PR. Co-authored-by: Jane Xu <janeyx@meta.com> Pull Request resolved: pytorch#123629 Approved by: https://github.com/jgong5, https://github.com/janeyx99

…re (pytorch#123581) This PR targets the issue mentioned in pytorch#123451 , and solves the specific task to update`test_graph_grad_scaling` in `test/test_cuda.py` to use the new OptimizerInfo infrastructure. `test_graph_grad_scaling` is moved to a new `TestCase` class called `TestCudaOptims` in order to use `instantiate_device_type_tests`. The test content remained the same. `@onlyCUDA` is applied to the new test; the original use of the wrapper function is also changed to a `@parametrize` decorator for better style. If we think that this migration is successful, we can delete the original test item under `TestCuda`. Currently it is left untouched to avoid any unexpected issues. Local linter passed. ``` $ lintrunner test/test_cuda.py ok No lint issues. ``` Local tests passed. ``` > python .\test\test_cuda.py -k test_graph_grad_scaling Ran 7 tests in 0.458s OK (skipped = 3) ``` Co-authored-by: Jane (Yuan) Xu <31798555+janeyx99@users.noreply.github.com> Pull Request resolved: pytorch#123581 Approved by: https://github.com/janeyx99

Support fused_sgd_kernel support for CPU. ## Bench result: 32 core/sockets ICX Test Scripts: https://gist.github.com/zhuhaozhe/688763e17e93e4c5e12f25f676ec90d9 https://gist.github.com/zhuhaozhe/ad9938694bc7fae8b66d376f4dffc6c9 ``` Tensor Size: 262144, Num Tensor 4, Num Threads: 1 _single_tensor_sgd time: 0.2301 seconds _fused_sgd time: 0.0925 seconds Tensor Size: 4194304, Num Tensor 32, Num Threads: 32 _single_tensor_sgd time: 2.6195 seconds _fused_sgd time: 1.7543 seconds ``` ## Test Plan: ``` python test_optim.py -k test_fused_matches_forloop python test_optim.py -k test_fused_large_tensor python test_optim.py -k test_can_load_older_state_dict python test_optim.py -k test_grad_scaling_autocast_fused_optimizers python test_torch.py -k test_grad_scaling_autocast_fused python test_torch.py -k test_params_invalidated_with_grads_invalidated_between_unscale_and_step ``` Looks like we already have some PRs under this issue pytorch#123451 to unified the UTs, I did not modified UT in this PR. Co-authored-by: Jane Xu <janeyx@meta.com> Pull Request resolved: pytorch#123629 Approved by: https://github.com/jgong5, https://github.com/janeyx99

…re (#123581) This PR targets the issue mentioned in #123451 , and solves the specific task to update`test_graph_grad_scaling` in `test/test_cuda.py` to use the new OptimizerInfo infrastructure. `test_graph_grad_scaling` is moved to a new `TestCase` class called `TestCudaOptims` in order to use `instantiate_device_type_tests`. The test content remained the same. `@onlyCUDA` is applied to the new test; the original use of the wrapper function is also changed to a `@parametrize` decorator for better style. If we think that this migration is successful, we can delete the original test item under `TestCuda`. Currently it is left untouched to avoid any unexpected issues. Local linter passed. ``` $ lintrunner test/test_cuda.py ok No lint issues. ``` Local tests passed. ``` > python .\test\test_cuda.py -k test_graph_grad_scaling Ran 7 tests in 0.458s OK (skipped = 3) ``` Co-authored-by: Jane (Yuan) Xu <31798555+janeyx99@users.noreply.github.com> Pull Request resolved: #123581 Approved by: https://github.com/janeyx99

…h#125538) Fixes pytorch#123451 (only addresses test_torch.py cases) This PR solves the specific task to update `test_grad_scaling_autocast` and `test_params_invalidated_with_grads_invalidated_between_unscale_and_step` in `test/test_torch.py` to use the new OptimizerInfo infrastructure. I have combined tests that call `_grad_scaling_autocast_test` into one called `test_grad_scaling_autocast` and used `_get_optim_inputs_including_global_cliquey_kwargs` to avoid hard-coded configurations. ``` $ lintrunner test/test_cuda.py ok No lint issues. ``` Pull Request resolved: pytorch#125538 Approved by: https://github.com/janeyx99

… use new OptimizerInfo infrastructure (#125127) This PR is meant to address issue #123451, more specifically, the ```test_graph_optims``` and ```test_graph_scaling_fused_optimizers``` functions in ```test_cuda.py``` have been updated so that they now use the new OptimizerInfo infrastructure. Lintrunner passed: ``` $ lintrunner test/test_cuda.py ok No lint issues. ``` Tests passed: ``` >python test_cuda.py -k test_graph_optims Ran 19 tests in 7.463s OK (skipped=9) >python test_cuda.py -k test_graph_scaling_fused_optimizers Ran 6 tests in 2.800s OK (skipped=3) ``` Both the functions have been moved to the newly created TestCase class ```TestCudaOptims```. The test is mostly the same except the ```@optims``` decorator is used at the top of the function to implicitly call the function using each of the optimizers mentioned in the decorator instead of explicitly using a for loop to iterate through each of the optimizers. I was unable to use the ```_get_optim_inputs_including_global_cliquey_kwargs``` to get all kwargs for each of the optimizers since some of the kwargs that are used in the original ```test_graph_optims``` function are not being returned by the new OptimizerInfo infrastructure, more specifically, for the ```torch.optim.rmsprop.RMSprop``` optimizer, the following kwargs are not returned whenever ```_get_optim_inputs_including_global_cliquey_kwargs``` is called: ``` {'foreach': False, 'maximize': True, 'weight_decay': 0} { 'foreach': True, 'maximize': True, 'weight_decay': 0} ``` I ran into the same issue for ```test_graph_scaling_fused_optimizers```, for the ```torch.optim.adamw.AdamW``` optimizer, whenever ```optim_info.optim_inputs_func(device=device)``` was called, the following kwarg was not returned: ``` {'amsgrad': True} ``` Due to this issue, I resorted to using a dictionary to store the kwargs for each of the optimizers, I am aware that this is less than ideal. I was wondering whether I should use the OptimizerInfo infrastructure to get all the kwargs regardless of the fact that it lacks some kwargs. Pull Request resolved: #125127 Approved by: https://github.com/janeyx99

janeyx99 · 2024-05-20T15:04:10Z

not done yet

janeyx99 added module: optimizer Related to torch.optim good first issue triaged This issue has been looked at a team member, and triaged and prioritized into an appropriate module better-engineering Relatively self-contained tasks for better engineering contributors actionable labels Apr 5, 2024

janeyx99 mentioned this issue Apr 5, 2024

[optim] add fused_adam/adamw_kernel support for CPU device #123074

Closed

rithikp06 mentioned this issue Apr 8, 2024

Updated test_torch.py optim tests to use OptimizerInfo and optim_db #123537

Closed

FireACNC mentioned this issue Apr 8, 2024

Updated test_graph_grad_scaling to use new OptimizerInfo infrastructure #123581

Closed

dwang3851 linked a pull request Apr 21, 2024 that will close this issue

Updated test_cuda.py optim tests to use OptimizerInfo #124563

Open

zhuhaozhe mentioned this issue Apr 23, 2024

add fused_sgd_kernel support for CPU device #123629

Closed

This was referenced Apr 24, 2024

test_cuda.py test_grad_scaling_autocast_fused_optimizers migration to OptimizerInfo (IGNORE/DELETE THIS PR) PetaBread545/pytorch#1

Closed

test_cuda.py test_grad_scaling_autocast_fused_optimizers migration to OptimizerInfo #124893

Closed

omastey linked a pull request Apr 26, 2024 that will close this issue

Updating optims and combining torch functions #125071

Open

jayanthd04 mentioned this issue Apr 28, 2024

Updated test_graph_optims and test_graph_scaling_fused_optimizers to use new OptimizerInfo infrastructure #125127

Open

gambiTarun mentioned this issue May 4, 2024

Updated test_torch.py to use new OptimizerInfo infrastructure #125538

Closed

SandishKumarHN mentioned this issue May 15, 2024

Updated test_graph_optims to use new optimizer info infrastructure #126340

Closed

pytorchmergebot closed this as completed in ad67553 May 18, 2024

gambiTarun mentioned this issue May 19, 2024

Test failures with new configs in test_grad_scaling_autocast in test_torch.py #126638

Open

janeyx99 reopened this May 20, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Update test_cuda.py and test_torch.py optim tests to use OptimizerInfo and optim_db #123451

Update test_cuda.py and test_torch.py optim tests to use OptimizerInfo and optim_db #123451

janeyx99 commented Apr 5, 2024 •

edited

rithikp06 commented Apr 5, 2024

janeyx99 commented Apr 5, 2024

rithikp06 commented Apr 5, 2024

ComposeC commented Apr 7, 2024

rithikp06 commented Apr 8, 2024

FireACNC commented Apr 8, 2024

jayanthd04 commented Apr 8, 2024

dwang3851 commented Apr 8, 2024

khlaifiabilel commented Apr 8, 2024

janeyx99 commented Apr 16, 2024

julian-8897 commented Apr 19, 2024

PetaBread545 commented Apr 20, 2024

dwang3851 commented Apr 21, 2024

omastey commented Apr 22, 2024

Rbhu376264 commented Apr 24, 2024

janeyx99 commented May 20, 2024

Update test_cuda.py and test_torch.py optim tests to use OptimizerInfo and optim_db #123451

Update test_cuda.py and test_torch.py optim tests to use OptimizerInfo and optim_db #123451

Comments

janeyx99 commented Apr 5, 2024 • edited

🚀 The feature, motivation and pitch

Alternatives

Additional context

rithikp06 commented Apr 5, 2024

janeyx99 commented Apr 5, 2024

rithikp06 commented Apr 5, 2024

ComposeC commented Apr 7, 2024

rithikp06 commented Apr 8, 2024

FireACNC commented Apr 8, 2024

jayanthd04 commented Apr 8, 2024

dwang3851 commented Apr 8, 2024

khlaifiabilel commented Apr 8, 2024

janeyx99 commented Apr 16, 2024

julian-8897 commented Apr 19, 2024

PetaBread545 commented Apr 20, 2024

dwang3851 commented Apr 21, 2024

omastey commented Apr 22, 2024

Rbhu376264 commented Apr 24, 2024

janeyx99 commented May 20, 2024

janeyx99 commented Apr 5, 2024 •

edited