[RLlib] DQN torch version. #7597

sven1977 · 2020-03-13T11:56:30Z

NOTE: Please merge PR #7814 (param-noise completion) and PR #7852 (framework_iterator) first.

Implement a DQN version for PyTorch. This includes:

SimpleQ and rainbow DQN (double, dueling, prioritized-replay, n_step, param-noise, noisy layers) with the exception of the distributional-Q head (will be a follow-up/cleanup PR).
Learning regression tests for CartPole on SimpleQ torch+tf, DQN torch+tf, and DQN+param-noise tf+torch have been added/updated.
All affected regression tests (loss funcs, compilation, param-noise, exploration) have been updated to include torch.
Documentation has been updated to show PyTorch symbol on DQN and APEX.

Addresses and solves issue #4371

I've run scripts/format.sh to lint the changes in this PR.
I've included any doc changes needed for https://ray.readthedocs.io/en/latest/.
I've made sure the tests are passing. Note that there might be a few flaky tests, see the recent failure rates at https://ray-travis-tracker.herokuapp.com/.
Testing Strategy
- Unit tests
- Release tests
- This PR is not tested (please justify below)

…oration_api_parameter_noise

… into exploration_api_parameter_noise # Conflicts: # rllib/agents/dqn/dqn_policy.py # rllib/agents/dqn/simple_q_policy.py # rllib/policy/eager_tf_policy.py # rllib/utils/exploration/exploration.py # rllib/utils/exploration/parameter_noise.py

…oration_api_parameter_noise

AmplabJenkins · 2020-03-13T11:59:34Z

Can one of the admins verify this patch?

AmplabJenkins · 2020-03-13T12:49:27Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/23144/
Test FAILed.

AmplabJenkins · 2020-03-13T16:26:28Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/23162/
Test FAILed.

AmplabJenkins · 2020-03-14T20:45:07Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/23198/
Test FAILed.

AmplabJenkins · 2020-03-15T13:06:47Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/23215/
Test FAILed.

AmplabJenkins · 2020-03-16T13:20:14Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/23229/
Test PASSed.

…oration_api_parameter_noise

AmplabJenkins · 2020-03-16T14:07:17Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/23237/
Test FAILed.

AmplabJenkins · 2020-03-16T14:26:22Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/23235/
Test PASSed.

…oration_api_parameter_noise

AmplabJenkins · 2020-04-04T22:55:41Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/24250/
Test PASSed.

AmplabJenkins · 2020-04-04T23:28:58Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/24251/
Test PASSed.

ericl · 2020-04-05T05:03:25Z

LGTM but some regression tests are failing in travis

sven1977 · 2020-04-05T06:52:12Z

Yeah, it's the fake multi-GPU PPO one. It's broken already in master :( I moved it into test_ppo to not have to use tune.

AmplabJenkins · 2020-04-05T08:00:09Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/24268/
Test PASSed.

AmplabJenkins · 2020-04-05T08:18:55Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/24271/
Test PASSed.

AmplabJenkins · 2020-04-05T10:26:21Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/24272/
Test FAILed.

AmplabJenkins · 2020-04-05T14:39:11Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/24273/
Test PASSed.

AmplabJenkins · 2020-04-05T17:04:08Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/24274/
Test PASSed.

…h_dqn � Conflicts: � .travis.yml � rllib/BUILD � rllib/agents/dqn/tests/test_dqn.py � rllib/tests/run_regression_tests.py

AmplabJenkins · 2020-04-06T10:24:04Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/24293/
Test PASSed.

AmplabJenkins · 2020-04-06T11:46:47Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/24296/
Test PASSed.

AmplabJenkins · 2020-04-06T14:19:30Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/24299/
Test PASSed.

AmplabJenkins · 2020-04-06T14:28:19Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/24300/
Test PASSed.

AmplabJenkins · 2020-04-06T15:27:40Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/24302/
Test PASSed.

sven1977 · 2020-04-06T17:51:06Z

@ericl RLlib tests are ok now. Please merge. Thanks!

sven1977 added 17 commits March 4, 2020 10:38

Fix.

e08a141

Merge branch 'master' of https://github.com/ray-project/ray

309effc

Rollback.

91d1e5b

Merge branch 'master' of https://github.com/ray-project/ray

ea9ad54

Merge branch 'master' of https://github.com/ray-project/ray

d720c20

Merge branch 'master' of https://github.com/ray-project/ray

19a6302

Merge branch 'master' of https://github.com/ray-project/ray

eb7da53

Merge branch 'master' of https://github.com/ray-project/ray

f0212ce

WIP.

d4b11f4

WIP.

ca22b19

Merge branch 'master' of https://github.com/ray-project/ray into expl…

9627c7d

…oration_api_parameter_noise

WIP.

5a93a2a

WIP.

a634340

WIP.

e313966

WIP.

dd6c3b7

Merge branch 'master' of https://github.com/ray-project/ray into expl…

939a821

…oration_api_parameter_noise

Merge branch 'master' of https://github.com/ray-project/ray into expl…

33e2128

…oration_api_parameter_noise

sven1977 added 4 commits March 17, 2020 14:30

Merge branch 'master' of https://github.com/ray-project/ray into expl…

2f76c38

…oration_api_parameter_noise

WIP.

79a0998

WIP.

de2908e

Fix.

e0b3532

ericl approved these changes Apr 5, 2020

View reviewed changes

sven1977 added 2 commits April 5, 2020 08:31

test

8108202

Fix.

2fc9ae5

sven1977 mentioned this pull request Apr 5, 2020

[rllib] [Feature Request] PyTorch version of DQN style algorithms #4371

Closed

Fix.

9856ee4

WIP.

722873c

WIP.

d5fcd6e

sven1977 added 2 commits April 6, 2020 10:48

WIP.

6681a59

Merge branch 'master' of https://github.com/ray-project/ray into torc…

376198e

…h_dqn � Conflicts: � .travis.yml � rllib/BUILD � rllib/agents/dqn/tests/test_dqn.py � rllib/tests/run_regression_tests.py

sven1977 added 3 commits April 6, 2020 14:52

WIP.

0e0a23c

LINT.

d5697de

WIP.

f8d6068

sven1977 added the tests-ok The tagger certifies test failures are unrelated and assumes personal liability. label Apr 6, 2020

ericl merged commit 22ccc43 into ray-project:master Apr 6, 2020

sven1977 deleted the torch_dqn branch August 21, 2020 07:44

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[RLlib] DQN torch version. #7597

[RLlib] DQN torch version. #7597

sven1977 commented Mar 13, 2020 •

edited

AmplabJenkins commented Mar 13, 2020

AmplabJenkins commented Mar 13, 2020

AmplabJenkins commented Mar 13, 2020

AmplabJenkins commented Mar 14, 2020

AmplabJenkins commented Mar 15, 2020

AmplabJenkins commented Mar 16, 2020

AmplabJenkins commented Mar 16, 2020

AmplabJenkins commented Mar 16, 2020

AmplabJenkins commented Apr 4, 2020

AmplabJenkins commented Apr 4, 2020

ericl commented Apr 5, 2020

sven1977 commented Apr 5, 2020

AmplabJenkins commented Apr 5, 2020

AmplabJenkins commented Apr 5, 2020

AmplabJenkins commented Apr 5, 2020

AmplabJenkins commented Apr 5, 2020

AmplabJenkins commented Apr 5, 2020

AmplabJenkins commented Apr 6, 2020

AmplabJenkins commented Apr 6, 2020

AmplabJenkins commented Apr 6, 2020

AmplabJenkins commented Apr 6, 2020

AmplabJenkins commented Apr 6, 2020

sven1977 commented Apr 6, 2020

[RLlib] DQN torch version. #7597

[RLlib] DQN torch version. #7597

Conversation

sven1977 commented Mar 13, 2020 • edited

AmplabJenkins commented Mar 13, 2020

AmplabJenkins commented Mar 13, 2020

AmplabJenkins commented Mar 13, 2020

AmplabJenkins commented Mar 14, 2020

AmplabJenkins commented Mar 15, 2020

AmplabJenkins commented Mar 16, 2020

AmplabJenkins commented Mar 16, 2020

AmplabJenkins commented Mar 16, 2020

AmplabJenkins commented Apr 4, 2020

AmplabJenkins commented Apr 4, 2020

ericl commented Apr 5, 2020

sven1977 commented Apr 5, 2020

AmplabJenkins commented Apr 5, 2020

AmplabJenkins commented Apr 5, 2020

AmplabJenkins commented Apr 5, 2020

AmplabJenkins commented Apr 5, 2020

AmplabJenkins commented Apr 5, 2020

AmplabJenkins commented Apr 6, 2020

AmplabJenkins commented Apr 6, 2020

AmplabJenkins commented Apr 6, 2020

AmplabJenkins commented Apr 6, 2020

AmplabJenkins commented Apr 6, 2020

sven1977 commented Apr 6, 2020

sven1977 commented Mar 13, 2020 •

edited