Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

add NotImplementedError for multi optimizers #22181

Merged
merged 3 commits into from
Jan 11, 2020
Merged

add NotImplementedError for multi optimizers #22181

merged 3 commits into from
Jan 11, 2020

Conversation

liym27
Copy link
Contributor

@liym27 liym27 commented Jan 8, 2020 •
edited
Loading

If optimizer is used in control flow, training on multi-places is not supported now.

This PR adds NotImplementedError.

  • For example:
    Optimizer adam or sgd is used in control flow：
def fn_1(opt, avg_loss):
    opt.minimize(avg_loss)

def fn_2(opt, avg_loss):
    opt.minimize(avg_loss)

x = fluid.layers.data("X", [10], 'float32')
hidden = layers.fc(x, 5)
avg_loss = layers.mean(hidden)

adam = optimizer.Adam(learning_rate=LR)
sgd = optimizer.SGD(learning_rate=LR)

cond = layers.fill_constant([1], 'bool', True)

layers.case([(cond, lambda: fn_1(adam, avg_loss))],
            lambda: fn_2(sgd, avg_loss))

  • if it is run on multi cards with parallel executor, error message is as follows:

NotImplementedError: If optimizer is used in control flow, training on multi-places is not supported now.

@liym27 liym27 force-pushed the add_error_for_PE_multi_cards branch from ce3ed56 to 333568f Compare January 9, 2020 02:33
@liym27 liym27 changed the title add error add NotImplementedError for multi optimizers Jan 9, 2020
@liym27 liym27 force-pushed the add_error_for_PE_multi_cards branch 2 times, most recently from 9b316b4 to 0bb78fa Compare January 9, 2020 05:56
@liym27 liym27 force-pushed the add_error_for_PE_multi_cards branch from 0bb78fa to 11cbd01 Compare January 9, 2020 06:07
@liym27 liym27 requested a review from Aurelius84 January 9, 2020 06:25
sneaxiy
sneaxiy previously approved these changes Jan 9, 2020
View reviewed changes
Aurelius84
Aurelius84 previously approved these changes Jan 9, 2020
View reviewed changes
@liym27 liym27 dismissed stale reviews from Aurelius84 and sneaxiy via 0dd47bb January 10, 2020 01:41
sneaxiy
sneaxiy reviewed Jan 10, 2020
View reviewed changes
python/paddle/fluid/tests/unittests/test_optimizer_in_control_flow.py
@@ -225,5 +227,58 @@ def test_optimzier_in_switch(self):
loss_2))


class TestMultiOptimizersMultiCardsError(unittest.TestCase):
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You should add this unittest here https://github.com/PaddlePaddle/Paddle/blob/develop/python/paddle/fluid/tests/unittests/CMakeLists.txt#L327 so that it can run in 2 gpu cards in CI.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks. Done

@liym27 liym27 force-pushed the add_error_for_PE_multi_cards branch from f2693db to c36b1f3 Compare January 10, 2020 02:51
@liym27 liym27 force-pushed the add_error_for_PE_multi_cards branch from 34c39c8 to 34cce8e Compare January 10, 2020 08:35
@liym27 liym27 merged commit 8de33f4 into PaddlePaddle:develop Jan 11, 2020
@liym27 liym27 mentioned this pull request Jan 12, 2020
Merged
liym27 added a commit to liym27/Paddle that referenced this pull request Jan 12, 2020
@liym27
[cherry-pick]add NotImplementedError for multi optimizers (PaddlePad…
cfcfdee 
…dle#22181)

* add NotImplementedError for multi optimizers used on multi-places . test=develop

* assert error only if num_devices>1. test=develop

* set test_optimizer_in_control_flow in CMakeLists for using multi-GPU.test=develop
test=release/1.7
Xreki added a commit to Xreki/Paddle that referenced this pull request Jan 13, 2020
@Xreki @FlyingQianMM
@wangchaochaohu @liym27 @jiweibo @JepsonWong @phlrain
Fusion group final (#6)
351241a 
* Add the first implememtation of fusion_group op PaddlePaddle#19621 (#3)

* Add the dynamic load of nvrtc, and support runtime compiling of CUDA kernel using nvrtc.
test=develop

* Call CUDA driver api to launch the kernel compiled by nvrtc.
test=develop

* Disable for mac and windows.
test=develop

* Refine the codes to support manually specified num_threads and workload_per_thread.
test=develop

* Refine the CUDA kernel to support large dims.
test=develop

* Add DeviceCodePool to manage all device codes.

* Add the first implementation fusion_group op.

* Add unit-test for fusion_group op.

* Add the check of result.

* Add the check of nvrtc in unit-test.
test=develop

* Add comment to explain the inputs, outputs and features of fusion_group op.
test=develop

* Disable fusion_group op for mac and windows.
test=develop

* Make the compiling of device code return status instead of hanging up.
test=develop

* Add the check of whether there is CUDA driver library, and do not core dump when failing to call the CUDA driver API.

* Unify fusion_group_op's input and output names.
test=develop

* Add the check of CUDA driver library in unittest.
test=develop

* Enable generating code for a given subgraph. PaddlePaddle#21126 (#4)

* Enable generating code for a given subgraph.

* Support sorting the subgraph.

* Remove the rearange of expressions because we use the sorted subgraph directly.

* Enable generating code for a subgraph which is composed of grad ops.

* Use expression information to check the accuracy in unittest.

* Separate load and store from computation expressions.
test=develop

* Improve the loading statements in generated codes.
test=develop

* Remove unused arguments from formal list.
test=develop

* Enable the detection of subgraph of grad ops.

* Generate code for detected subgraph in fusion_group_pass.

* Add an option in BuildStrategy to enable fusion_group_pass and add unittest.
test=develop

* Fix a bug when checking whether the shape of all inputs are the same.

* Add debug information.

* Remove subgraph_detector from inference/analysis to the common framework/ir directory. (#5)

test=develop

* Call subgraph_detector in fusion_group pass.
test=develop

* Disable fusion_group when WITH_GPU is OFF.
test=develop

* Refine all PADDLE_ENFORCE message.
test=develop

* Fix the case that some inputs are not defined in grad ops, and set op_role for fused op.
test=develop

* add backward gradient computation for op argsort (PaddlePaddle#22203)

* add backward gradient computation for op argsort test=developo

* use pre-commit test=develop

* fix the bug of profile update (PaddlePaddle#22207)

* fix the bug of profile update test=develop

*  add NotImplementedError for multi optimizers (PaddlePaddle#22181)

* add NotImplementedError for multi optimizers used on multi-places . test=develop

* assert error only if num_devices>1. test=develop

* set test_optimizer_in_control_flow in CMakeLists for using multi-GPU.test=develop

* support fluid-lite subgraph run resnet test=develop (PaddlePaddle#22191)

- 添加了fluid-lite子图方式运行resnet的单测
- 修改了依赖Lite的git commit id

* fix bug fot test_dygraph_mnist_fp16.py, test=develop (PaddlePaddle#22222)

* Check dygraph weight name (PaddlePaddle#22140)

* add parameter check; test=develop

* change parameter name checker in dygraph guard; test=develop

* fix test layers error; test=develop

* revert some code to develop; test=develop

* fix exampel error; test=develop

* fix comment error; test=develop

* fix comment error; test=develop

* only import used test case and function(PaddlePaddle#22208)

Co-authored-by: FlyingQianMM <245467267@qq.com>
Co-authored-by: wangchaochaohu <wangchao66@baidu.com>
Co-authored-by: liym27 <33742067+liym27@users.noreply.github.com>
Co-authored-by: Wilber <jiweibo1028@outlook.com>
Co-authored-by: zhongpu <2013000149@qq.com>
Co-authored-by: hong <43953930+phlrain@users.noreply.github.com>
Co-authored-by: Zhang Ting <709968123@qq.com>
liym27 added a commit that referenced this pull request Jan 13, 2020
@liym27
[cherry-pick]add NotImplementedError for multi optimizers (#22181) (#…
3acb995 
…22229)

* add NotImplementedError for multi optimizers used on multi-places . test=develop

* assert error only if num_devices>1. test=develop

* set test_optimizer_in_control_flow in CMakeLists for using multi-GPU.test=develop
@liym27 liym27 deleted the add_error_for_PE_multi_cards branch January 17, 2020 06:34
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@sneaxiy sneaxiy sneaxiy approved these changes

@Aurelius84 Aurelius84 Aurelius84 left review comments

@zhhsplendid zhhsplendid Awaiting requested review from zhhsplendid

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

None yet
3 participants
@liym27 @Aurelius84 @sneaxiy