[RLlib] Learner API Docs #37729

avnishn · 2023-07-24T21:16:19Z

Signed-off-by: Avnish avnishnarayan@gmail.com

Why are these changes needed?

Related issue number

Checks

I've signed off every commit(by using the -s flag, i.e., git commit -s) in this PR.
I've run scripts/format.sh to lint the changes in this PR.
I've included any doc changes needed for https://docs.ray.io/en/master/.
- I've added any new APIs to the API Reference. For example, if I added a
  method in Tune, I've added it in doc/source/tune/api/ under the
  corresponding .rst file.
I've made sure the tests are passing. Note that there might be a few flaky tests, see the recent failures at https://flakey-tests.ray.io/
Testing Strategy
- Unit tests
- Release tests
- This PR is not tested :(

Signed-off-by: Avnish <avnishnarayan@gmail.com>

avnishn · 2023-07-24T21:17:01Z

These docs are WIP right now. I still need to add some of the other sections. I'm almost through getting the doc snippets to run without errors.

Signed-off-by: Avnish <avnishnarayan@gmail.com>

kouroshHakha

Can you make sure that everytime you use RLModule, Learner, and LearnerGroup you are referencing the actual API?

rllib/core/learner/learner_group.py

doc/source/rllib/rllib-learner.rst

angelinalg

Just some nits.

doc/source/rllib/rllib-learner.rst

rickyyx · 2023-07-27T21:09:31Z

update? still targeting at 2.6.2?

avnishn · 2023-07-27T21:43:19Z

yep.

…ner_api_docs

Signed-off-by: Avnish <avnishnarayan@gmail.com>

avnishn · 2023-07-31T18:55:34Z

update? still targeting at 2.6.2?

yes

Signed-off-by: Avnish <avnishnarayan@gmail.com>

ArturNiederfahrenhorst · 2023-07-31T21:28:26Z

doc/source/rllib/rllib-learner.rst

+	:hide:
+    :skipif: True
+
+    from ray.rllib.algorithms.ppo.ppo import PPOConfig


This doc contains a lot of code that is not tested.
There exists a folder "doc/source/rllib/doc_code" that could hold a guide file to hold all this test code that actually gets executed there. You could put code there and grab code snippets from there to ensure these example snippets stay up to date.

I think lets do that in a later iteration at this point since doc picks are due today.

But I agree that if thats the workaround then we'll go that way instead.

ArturNiederfahrenhorst · 2023-07-31T21:30:30Z

doc/source/rllib/rllib-learner.rst

+	config.build()  # test that the algorithm can be built with the given resources
+
+
+.. note::


Can we do this more like here?

But the warning that this feature is in alpha and the supported algos up top

Consolidate the information on how to enable it with the above information in section "Enabling Learner API in RLlib experiments".

avnishn · 2023-07-31T21:31:30Z

I purposely disabled doc testing for code snippets in this PR at the request of @bveeramani

We're having ray rpc issues when running the tests on the CI, but they pass locally.

ArturNiederfahrenhorst · 2023-07-31T21:31:34Z

doc/source/rllib/rllib-learner.rst

+
+Adjust the amount of resources for training using the 
+`num_gpus_per_learner_worker`, `num_cpus_per_learner_worker`, and `num_learner_workers`
+arguments in the `AlgorithmConfig`.


Can we link to the API refs page at least once for these things?

doc/source/rllib/rllib-learner.rst

ArturNiederfahrenhorst · 2023-07-31T22:05:12Z

doc/source/rllib/rllib-learner.rst

+                                framework_hyperparameters=FrameworkHyperparameters(),
+                            )
+
+            learner_group = LearnerGroup(learner_spec)


This would become more useful if it would include the necessary imports so it would become copy-pasteable

Its going to add a lot of cognitive load to the guide. Maybe instead we put all of this in a file later then come back to it. We can do that when we update the examples.

doc/source/rllib/rllib-learner.rst

ArturNiederfahrenhorst · 2023-07-31T22:23:52Z

doc/source/rllib/rllib-learner.rst

+            learner_group.load_state(LEARNER_GROUP_CKPT_DIR)
+
+        Checkpoint the state of all learners in the `LearnerGroup` via `save_state` and
+        `load_state`. This includes all states including neural network weights and any


Can we make clear what the difference is between get/set/save/load state?
For many of these sections, we just start off with a code example without giving context.
I think both sections "Getting and setting state" and "Checkpointing" should begin with something like:

"You can checkpoint the state of a LearnerGroup to a directory by calling LearnerGroup.save_state(). This will call LearnerGroup.get_state() under the hood and therefore includes neural networks weights and optimizer states.
Loading a checkpoint works accordingly. Here is an example of how to save and load the state of a LearnerGroup:

.. testcode:: :skipif: True learner_group.save_state(LEARNER_GROUP_CKPT_DIR) learner_group.load_state(LEARNER_GROUP_CKPT_DIR)

Note that since the state of all of the Learner s is identical, only the states from the first Learner need to be saved.

Signed-off-by: Avnish <avnishnarayan@gmail.com>

ArturNiederfahrenhorst · 2023-07-31T22:27:53Z

doc/source/rllib/rllib-learner.rst

+
+Implementation
+==============
+`Learner` has many APIs for flexible implementation, however the core ones that you need to implement are:


Imho we should clarify what this section deals with.
Is this about implementing a learner from scratch? Or for customization of an existing learner?

The heading should be something like "Customizing Learner" or "Implementing your own Learner" or something like that.

why can't it be both? I feel like this might be semantics.

ArturNiederfahrenhorst · 2023-07-31T22:51:21Z

doc/source/rllib/rllib-learner.rst

+
+Implementation
+==============
+:py:class:`~ray.rllib.core.learner.learner.Learner` has many APIs for flexible implementation, however the core ones that you need to implement are:


The Learner class has a lot of abstract methods (13 or so).
I think we should clarify here that if you would want to only override the ones below, you'll have to subclass a framework specific implementation.

I think what I was going for here is to describe the minimum functions that people need to implement or override for their custom learners, whether that be them overriding them or implementing a brand new one.

Imo this should read:

Writing or customizing Learners

If you want to write your own custom :py:class:~ray.rllib.core.learner.learner.Learner, you will want to subclass a framework-specific Learner. The following is a list of the most important methods that you'll want to implement. Similarely, these methods are likely what you'll want to override if you want to modify an existing Learner like TorchPPOLearner :

(...)

bveeramani · 2023-08-01T02:14:58Z

rllib/core/learner/learner.py

+        .. testcode::
+            :skipif: True


You can just do code-block::. Ratched will only complain for code-block:: python

bveeramani · 2023-08-01T02:16:21Z

doc/source/rllib/rllib-learner.rst

+arguments in the :py:class:`~ray.rllib.algorithms.algorithm_config.AlgorithmConfig`.
+
+.. testcode::
+	:hide:


Looks like this overindented?

bveeramani · 2023-08-01T02:17:28Z

doc/source/rllib/rllib-learner.rst

+
+.. testcode::
+	:hide:
+    :skipif: True


Alternatively, instead of labeling all blocks with :skipif: True, we could add rllib-learner.rst to the exclude list in the BUILD file.

Signed-off-by: Avnish <avnishnarayan@gmail.com>

Signed-off-by: Avnish <avnishnarayan@gmail.com> Signed-off-by: NripeshN <nn2012@hw.ac.uk>

Signed-off-by: Avnish <avnishnarayan@gmail.com> Signed-off-by: harborn <gangsheng.wu@intel.com>

Signed-off-by: Avnish <avnishnarayan@gmail.com>

Signed-off-by: Avnish <avnishnarayan@gmail.com> Signed-off-by: e428265 <arvind.chandramouli@lmco.com>

Signed-off-by: Avnish <avnishnarayan@gmail.com> Signed-off-by: Victor <vctr.y.m@example.com>

avnishn added 2 commits July 24, 2023 12:51

Docs for Learner API

ad56848

Signed-off-by: Avnish <avnishnarayan@gmail.com>

[RLlib] Learner API Doc

f98af7e

Signed-off-by: Avnish <avnishnarayan@gmail.com>

avnishn requested review from sven1977, gjoliver, ArturNiederfahrenhorst, smorad, maxpumperla, kouroshHakha, krfricke and a team as code owners July 24, 2023 21:16

avnishn assigned ArturNiederfahrenhorst and kouroshHakha Jul 24, 2023

Finished docs with working pytest

303a0c8

Signed-off-by: Avnish <avnishnarayan@gmail.com>

avnishn assigned angelinalg Jul 25, 2023

kouroshHakha reviewed Jul 25, 2023

View reviewed changes

angelinalg approved these changes Jul 25, 2023

View reviewed changes

avnishn added 8 commits July 30, 2023 13:06

Merge branch 'master' of https://github.com/ray-project/ray into lear…

2097ddc

…ner_api_docs

Fix broken doctest

2c04e9d

Signed-off-by: Avnish <avnishnarayan@gmail.com>

Try to fix broken doctest

6a71839

Signed-off-by: Avnish <avnishnarayan@gmail.com>

Address comments

fae803e

Signed-off-by: Avnish <avnishnarayan@gmail.com>

Remove changes to learner group

912c2a1

Signed-off-by: Avnish <avnishnarayan@gmail.com>

Update docstrings, add skipif for ray rpc issue

cdc0f5a

Signed-off-by: Avnish <avnishnarayan@gmail.com>

Resolve doc issues

62539e7

Signed-off-by: Avnish <avnishnarayan@gmail.com>

Fix noindex problem

f45e0d6

Signed-off-by: Avnish <avnishnarayan@gmail.com>

Resolve errors in docs

c39c06f

Signed-off-by: Avnish <avnishnarayan@gmail.com>

ArturNiederfahrenhorst reviewed Jul 31, 2023

View reviewed changes

doc/source/rllib/rllib-learner.rst Outdated Show resolved Hide resolved

ArturNiederfahrenhorst reviewed Jul 31, 2023

View reviewed changes

doc/source/rllib/rllib-learner.rst Outdated Show resolved Hide resolved

ArturNiederfahrenhorst reviewed Jul 31, 2023

View reviewed changes

doc/source/rllib/rllib-learner.rst Show resolved Hide resolved

ArturNiederfahrenhorst reviewed Jul 31, 2023

View reviewed changes

doc/source/rllib/rllib-learner.rst Outdated Show resolved Hide resolved

ArturNiederfahrenhorst reviewed Jul 31, 2023

View reviewed changes

doc/source/rllib/rllib-learner.rst Show resolved Hide resolved

ArturNiederfahrenhorst reviewed Jul 31, 2023

View reviewed changes

doc/source/rllib/rllib-learner.rst Outdated Show resolved Hide resolved

ArturNiederfahrenhorst reviewed Jul 31, 2023

View reviewed changes

Update all methods to be hyperlinked

64f1feb

Signed-off-by: Avnish <avnishnarayan@gmail.com>

ArturNiederfahrenhorst reviewed Jul 31, 2023

View reviewed changes

ArturNiederfahrenhorst approved these changes Jul 31, 2023

View reviewed changes

ArturNiederfahrenhorst reviewed Jul 31, 2023

View reviewed changes

bveeramani reviewed Aug 1, 2023

View reviewed changes

kouroshHakha merged commit 7dcad86 into ray-project:master Aug 3, 2023
46 of 51 checks passed

avnishn added a commit to avnishn/ray that referenced this pull request Aug 4, 2023

[RLlib][docs] Learner API Docs (ray-project#37729)

7634efc

Signed-off-by: Avnish <avnishnarayan@gmail.com>

rickyyx pushed a commit that referenced this pull request Aug 7, 2023

[RLlib][docs] Learner API Docs (#37729) (#38137)

9a2ecc3

Signed-off-by: Avnish <avnishnarayan@gmail.com>

NripeshN pushed a commit to NripeshN/ray that referenced this pull request Aug 15, 2023

[RLlib][docs] Learner API Docs (ray-project#37729)

152a9d3

Signed-off-by: Avnish <avnishnarayan@gmail.com> Signed-off-by: NripeshN <nn2012@hw.ac.uk>

harborn pushed a commit to harborn/ray that referenced this pull request Aug 17, 2023

[RLlib][docs] Learner API Docs (ray-project#37729)

4f90b98

Signed-off-by: Avnish <avnishnarayan@gmail.com> Signed-off-by: harborn <gangsheng.wu@intel.com>

harborn pushed a commit to harborn/ray that referenced this pull request Aug 17, 2023

[RLlib][docs] Learner API Docs (ray-project#37729)

261d990

Signed-off-by: Avnish <avnishnarayan@gmail.com>

arvind-chandra pushed a commit to lmco/ray that referenced this pull request Aug 31, 2023

[RLlib][docs] Learner API Docs (ray-project#37729)

152d227

Signed-off-by: Avnish <avnishnarayan@gmail.com> Signed-off-by: e428265 <arvind.chandramouli@lmco.com>

vymao pushed a commit to vymao/ray that referenced this pull request Oct 11, 2023

[RLlib][docs] Learner API Docs (ray-project#37729)

8eb1f5b

Signed-off-by: Avnish <avnishnarayan@gmail.com> Signed-off-by: Victor <vctr.y.m@example.com>

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[RLlib] Learner API Docs #37729

[RLlib] Learner API Docs #37729

avnishn commented Jul 24, 2023

avnishn commented Jul 24, 2023

kouroshHakha left a comment

angelinalg left a comment

rickyyx commented Jul 27, 2023

avnishn commented Jul 27, 2023

avnishn commented Jul 31, 2023

ArturNiederfahrenhorst Jul 31, 2023

avnishn Jul 31, 2023

ArturNiederfahrenhorst Jul 31, 2023

avnishn commented Jul 31, 2023

ArturNiederfahrenhorst Jul 31, 2023

avnishn Jul 31, 2023

avnishn Jul 31, 2023

ArturNiederfahrenhorst Jul 31, 2023 •

edited

Loading

avnishn Jul 31, 2023

ArturNiederfahrenhorst Jul 31, 2023

avnishn Jul 31, 2023

ArturNiederfahrenhorst Jul 31, 2023

avnishn Jul 31, 2023

ArturNiederfahrenhorst Jul 31, 2023

avnishn Jul 31, 2023

ArturNiederfahrenhorst Jul 31, 2023

bveeramani Aug 1, 2023

bveeramani Aug 1, 2023

bveeramani Aug 1, 2023

		config.build() # test that the algorithm can be built with the given resources


		.. note::

[RLlib] Learner API Docs #37729

[RLlib] Learner API Docs #37729

Conversation

avnishn commented Jul 24, 2023

Why are these changes needed?

Related issue number

Checks

avnishn commented Jul 24, 2023

kouroshHakha left a comment

Choose a reason for hiding this comment

angelinalg left a comment

Choose a reason for hiding this comment

rickyyx commented Jul 27, 2023

avnishn commented Jul 27, 2023

avnishn commented Jul 31, 2023

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

avnishn commented Jul 31, 2023

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ArturNiederfahrenhorst Jul 31, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Writing or customizing Learners

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ArturNiederfahrenhorst Jul 31, 2023 •

edited

Loading