[RLlib] DQN Rainbow Learner with new API stack (and EnvRunner API). #43196

simonsays1980 · 2024-02-15T13:46:29Z

Why are these changes needed?

We are moving DQN Rainbow over to the new stack using RLModule API and EnvRunner API. This PR introduces the learners for DQN Rainbow on the new stack.

Related issue number

Closes #37777

Checks

I've signed off every commit(by using the -s flag, i.e., git commit -s) in this PR.
I've run scripts/format.sh to lint the changes in this PR.
I've included any doc changes needed for https://docs.ray.io/en/master/.
- I've added any new APIs to the API Reference. For example, if I added a
  method in Tune, I've added it in doc/source/tune/api/ under the
  corresponding .rst file.
I've made sure the tests are passing. Note that there might be a few flaky tests, see the recent failures at https://flakey-tests.ray.io/
Testing Strategy
- Unit tests
- Release tests
- This PR is not tested :(

…tributional loss and simple Q-loss. Double-Q implemented next. Signed-off-by: Simon Zehnder <simon.zehnder@gmail.com>

Signed-off-by: Simon Zehnder <simon.zehnder@gmail.com>

sven1977 · 2024-02-20T13:20:33Z

rllib/algorithms/dqn/dqn_rainbow_learner.py

+from ray.rllib.utils.metrics import LAST_TARGET_UPDATE_TS, NUM_TARGET_UPDATES
+from ray.rllib.utils.typing import ModuleID
+
+# Now, this is double defined: In `SACRLModule` and here. I would keep it here


I think it's fine to leave them here as duplicates for better readability.

Imo, it's really not good to cross-import between different algorithms. Everywhere else, this would be ok to share.

rllib/algorithms/dqn/dqn_rainbow_learner.py

sven1977 · 2024-02-20T13:21:55Z

rllib/algorithms/dqn/dqn_rainbow_learner.py

+            timestep=timestep,
+        )
+
+        # TODO (Sven): APPO uses `config.target_update_frequency`. Can we


👍 Will revisit once we re-visit APPO/IMPALA on the new stack in a few days.

sven1977 · 2024-02-20T13:23:18Z

rllib/algorithms/dqn/torch/dqn_rainbow_torch_learner.py

+    def _update_module_target_networks(
+        self, module_id: ModuleID, config: DQNConfig
+    ) -> None:
+        """Updates the target Q network(s) of a module.


Same here, I don't think we need the docstring at all here. Move all the descriptions below into the super (abstract) method.

To keep the code consistent let's implement the same design into the module definitions:

Parent class holds docstrings

Child class has no docstrings for inherited methods

sven1977 · 2024-02-20T13:23:50Z

rllib/algorithms/sac/torch/sac_torch_learner.py

@@ -182,8 +182,6 @@ def compute_loss_for_module(
        if self.config.twin_q:
            q_twin_selected = fwd_out[QF_TWIN_PREDS]

-        # TODO (simon): Implement twin Q.


rllib/algorithms/sac/torch/sac_torch_learner.py

sven1977 · 2024-02-20T13:24:20Z

rllib/algorithms/sac/torch/sac_torch_learner.py

@@ -323,6 +319,12 @@ def _update_module_target_networks(
        """Updates the target Q network(s) of a module.

        Applies Polyak averaging for the update.
+


Same here, move this description to the super (abstract) method and remove the docstring here entirely.

sven1977

LGTM! Just a few comment-related nits!
Awesome work @simonsays1980 !

…menets. Signed-off-by: Simon Zehnder <simon.zehnder@gmail.com>

Signed-off-by: Simon Zehnder <simon.zehnder@gmail.com>

…to dqn-rainbow-learner Signed-off-by: Simon Zehnder <simon.zehnder@gmail.com>

Implemented 'DQNRainbowLearner' and 'DQNRainbowTorchLearner' with dis…

23efebb

…tributional loss and simple Q-loss. Double-Q implemented next. Signed-off-by: Simon Zehnder <simon.zehnder@gmail.com>

simonsays1980 changed the title ~~DQN Rainbow with new stack and EnvRunner API~~ DQN Rainbow Learner with new stack and EnvRunner API Feb 15, 2024

This was referenced Feb 15, 2024

[RLlib] DQN Rainbow on new API stack (w/ EnvRunner):training_step implementation. #43198

Merged

[RLlib] DQN Rainbow on new API stack: RLModule and Catalog together with TorchNoisyMLP. #43199

Merged

Added double Q-learning to the learner and added some keys.

619df29

Signed-off-by: Simon Zehnder <simon.zehnder@gmail.com>

sven1977 reviewed Feb 20, 2024

View reviewed changes

rllib/algorithms/dqn/dqn_rainbow_learner.py Outdated Show resolved Hide resolved

sven1977 reviewed Feb 20, 2024

View reviewed changes

rllib/algorithms/sac/torch/sac_torch_learner.py Show resolved Hide resolved

sven1977 reviewed Feb 20, 2024

View reviewed changes

sven1977 approved these changes Feb 20, 2024

View reviewed changes

sven1977 added the tests-ok The tagger certifies test failures are unrelated and assumes personal liability. label Feb 20, 2024

sven1977 self-assigned this Feb 20, 2024

sven1977 changed the title ~~DQN Rainbow Learner with new stack and EnvRunner API~~ [RLlib] DQN Rainbow Learner with new API stack and EnvRunner API Feb 20, 2024

sven1977 changed the title ~~[RLlib] DQN Rainbow Learner with new API stack and EnvRunner API~~ [RLlib] DQN Rainbow Learner with new API stack (and EnvRunner API). Feb 20, 2024

sven1977 marked this pull request as ready for review February 20, 2024 13:25

sven1977 requested review from avnishn, ArturNiederfahrenhorst, maxpumperla and kouroshHakha as code owners February 20, 2024 13:25

sven1977 and others added 4 commits February 20, 2024 14:25

Merge branch 'master' into dqn-rainbow-learner

dbd6404

Implemented changes from @sven1977's review and made some code improv…

7515289

…menets. Signed-off-by: Simon Zehnder <simon.zehnder@gmail.com>

Merge branch 'master' into dqn-rainbow-learner

54ab834

Signed-off-by: Simon Zehnder <simon.zehnder@gmail.com>

Merge branch 'dqn-rainbow-learner' of github.com:simonsays1980/ray in…

21fe53e

…to dqn-rainbow-learner Signed-off-by: Simon Zehnder <simon.zehnder@gmail.com>

sven1977 merged commit 359edb9 into ray-project:master Feb 27, 2024
8 of 9 checks passed

simonsays1980 deleted the dqn-rainbow-learner branch February 27, 2024 11:51

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[RLlib] DQN Rainbow Learner with new API stack (and EnvRunner API). #43196

[RLlib] DQN Rainbow Learner with new API stack (and EnvRunner API). #43196

simonsays1980 commented Feb 15, 2024 •

edited

Loading

sven1977 Feb 20, 2024

sven1977 Feb 20, 2024

sven1977 Feb 20, 2024

simonsays1980 Feb 23, 2024

sven1977 Feb 20, 2024

sven1977 Feb 20, 2024

sven1977 left a comment

		@@ -323,6 +319,12 @@ def _update_module_target_networks(
		"""Updates the target Q network(s) of a module.

		Applies Polyak averaging for the update.

[RLlib] DQN Rainbow Learner with new API stack (and EnvRunner API). #43196

[RLlib] DQN Rainbow Learner with new API stack (and EnvRunner API). #43196

Conversation

simonsays1980 commented Feb 15, 2024 • edited Loading

Why are these changes needed?

Related issue number

Closes #37777

Checks

sven1977 Feb 20, 2024

Choose a reason for hiding this comment

sven1977 Feb 20, 2024

Choose a reason for hiding this comment

sven1977 Feb 20, 2024

Choose a reason for hiding this comment

simonsays1980 Feb 23, 2024

Choose a reason for hiding this comment

sven1977 Feb 20, 2024

Choose a reason for hiding this comment

sven1977 Feb 20, 2024

Choose a reason for hiding this comment

sven1977 left a comment

Choose a reason for hiding this comment

simonsays1980 commented Feb 15, 2024 •

edited

Loading