[Doc] Explain DreamerV3 architecture and nomenclature

[vmoens]

Stack from ghstack (oldest at bottom):

• -> [Doc] Explain DreamerV3 architecture and nomenclature #4095
• [Algorithm] Add a reproducible DreamerV3 DMC Walker benchmark #4075
• [Feature] Complete DreamerV3 continuation training semantics #4074
• [BugFix] Match DreamerV3 categorical and KL semantics #4073
• [Feature] Add the DreamerV3 block RSSM core #4072
• [Feature] Add DreamerV3 slow-critic regularization #4068
• [Feature] Normalize DreamerV3 REINFORCE return scale #4067
• [BugFix] Decode DreamerV3 rewards and values end to end #4066
• [BugFix] Correct DreamerV3 symexp two-hot semantics #4065

[pytorch-bot]

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/rl/4095

• 📄 Preview Python docs built from this PR

Note: Links to docs will display an error until the docs builds have been completed.

✅ You can merge normally! (2 Unrelated Failures)

As of commit 83c8169 with merge base d5a62cb ():

BROKEN TRUNK - The following jobs failed but were present on the merge base:

👉 Rebase onto the `viable/strict` branch to avoid these failures

• Unit-tests on Linux / tests-cpu (3.10, bulk) / linux-job (gh) (trunk failure)
  test/rb/test_storages.py::TestStorages::test__rand_given_ndim_recompile
• Unit-tests on Linux / tests-cpu (3.13, bulk) / linux-job (gh) (trunk failure)
  test/rb/test_storages.py::TestStorages::test__rand_given_ndim_recompile

This comment was automatically generated by Dr. CI and updates every 15 minutes.

[vmoens]

Added a dedicated “DreamerV3 in a nutshell” API-reference page.

It now:

• defines the world model, belief/deterministic state, stochastic state, prior, posterior, imagination, critic, slow critic, and continuation terminology;
• explains the real-data RSSM filtering path and the prior-only imagination path;
• summarizes the world-model, actor, and critic objectives and their supported variants;
• clarifies that the slow critic is a Polyak/soft-updated target used for critic regularization, while the online critic remains the imagined lambda-return bootstrap;
• documents the external SoftUpdate call and exact update equation;
• separates the optimizer-agnostic loss API from the runnable example’s three Adam optimizers, warmup, and adaptive gradient clipping; and
• provides an API map linking the DreamerV3 modules, objectives, and scalar-transform helpers.

Validation: documentation-relevant pre-commit hooks pass, the new page passes a warnings-as-errors Sphinx parse, and all referenced TorchRL API targets resolve.

[vmoens]

Addressed the discoverability/placement feedback:

• moved the DreamerV3 overview from the first position to the final position in the objectives navigation;
• linked it contextually from the DreamerV3ModelLoss, DreamerV3ActorLoss, and DreamerV3ValueLoss docstrings;
• added contextual links from the three DreamerV3 RSSM class docstrings; and
• linked it from the world-model API section.

This keeps the page discoverable when readers encounter the relevant API without presenting it as the primary objectives entry.

Validation: documentation-relevant pre-commit hooks pass, an isolated Sphinx warnings-as-errors build resolves all new links, and the two DreamerV3 module doctest suites pass (15 tests).