[RLlib; RLlib contrib] Move tuned_examples into rllib_contrib and remove CI learning tests for contrib algos.

[sven1977]

This PR:

• "rescues" all tuned_example files for those algos soon to be completely moved into rllib_contrib. A new subdir (tuned_examples) is created inside each rllib_contrib algo that contains the yaml/py files.
• Removed the learning_tests from the CI for those algorithms (which use these tuned_examples files). Note that each rllib_contrib algo already has a learning example run by the CI anyways.
• Removes all compilation test (files and CI definitions) for all rllib_contrib algos. Note that these tests are already run inside the rllib_contrib's own CI.

Why are these changes needed?

Related issue number

Checks

• I've signed off every commit(by using the -s flag, i.e., git commit -s) in this PR.
• I've run scripts/format.sh to lint the changes in this PR.
• I've included any doc changes needed for https://docs.ray.io/en/master/.
  • I've added any new APIs to the API Reference. For example, if I added a
    method in Tune, I've added it in doc/source/tune/api/ under the
    corresponding .rst file.
• I've made sure the tests are passing. Note that there might be a few flaky tests, see the recent failures at https://flakey-tests.ray.io/
• Testing Strategy
  • Unit tests
  • Release tests
  • This PR is not tested :(

[kouroshHakha]

This is a super large and critical change so we have to be careful and systematic about it. Can you use this check list to verify each part?

1. Remove deprecated tests from rllib/BUILD

Algorithm	Done?
A2C	[x]
A3C	[x]
Alpha Star	[x]
Alpha Zero	[x]
APEX DDPG	[x]
APEX DQN	[x]
ARS	[x]
Bandit	[x]
CRR	[x]
DDPG	[x]
DDPO	[x]
DT	[x]
ES	[x]
Leela Chess Zero	[x]
MADDPG	[x]
MAML	[x]
MBMPO	[x]
PG	[x]
QMIX	[x]
R2D2	[x]
SimpleQ	[x]
SlateQ	[x]
TD3	[x]

2. Move tuned_examples/algo_x/.yaml to rllib_contrib/algo_x/tuned_examples/.yaml

Algorithm	Done?
A2C	[x]
A3C	[x]
Alpha Star	[x]
Alpha Zero	[x]
APEX DDPG	[x]
APEX DQN	[x]
ARS	[x]
Bandit	[x]
CRR	[x]
DDPG	[x]
DDPO	[x]
DT	[x]
ES	[x]
Leela Chess Zero	[x]
MADDPG	[x]
MAML	[x]
MBMPO	[x]
PG	[x]
QMIX	[x]
R2D2	[x]
SimpleQ	[x]
SlateQ	[x]
TD3	[x]

3. Remove algorithms/algo_x/tests/* (They should already be inside rllib_contrib/algo_x/tests/*)

Algorithm	Done?
A2C	[x]
A3C	[x]
Alpha Star	[x]
Alpha Zero	[x]
APEX DDPG	[x]
APEX DQN	[x]
ARS	[x]
Bandit	[x]
CRR	[x]
DDPG	[x]
DDPO	[x]
DT	[x]
ES	[x]
Leela Chess Zero	[x]
MADDPG	[x]
MAML	[x]
MBMPO	[x]
PG	[x]
QMIX	[x]
R2D2	[x]
SimpleQ	[x]
SlateQ	[x]
TD3	[x]

4. Add the removed deprecated tests to rllib_contrib CI (maybe we should create one BUILD file for each rllib_contrib algo and use BAZEL command?)

Algorithm	Done?
A2C	[ ]
A3C	[ ]
Alpha Star	[ ]
Alpha Zero	[ ]
APEX DDPG	[ ]
APEX DQN	[ ]
ARS	[ ]
Bandit	[ ]
CRR	[ ]
DDPG	[ ]
DDPO	[ ]
DT	[ ]
ES	[ ]
Leela Chess Zero	[ ]
MADDPG	[ ]
MAML	[ ]
MBMPO	[ ]
PG	[ ]
QMIX	[ ]
R2D2	[ ]
SimpleQ	[ ]
SlateQ	[ ]
TD3	[ ]

[kouroshHakha]

only test_dt.py is copied into dt/tests. The other three are missing from rllib_contrib.

[sven1977]

good catch, will fix.

[sven1977]

done

[kouroshHakha]

only test_simple_q.py is copied into simple_q/tests. The other three are missing from rllib_contrib.

[sven1977]

wait, no there are only two files in the original repo (test_repro_simple_q and test_simple_q) and they are both there already in rllib_contrib being covered by the tests.

[kouroshHakha]

shouldn't we delete the original of all these files from rllib/tuned_examples?

[sven1977]

Yes, we absolutely should.

[sven1977]

Ok, I'll do this now:

• For each contrib algo, there will be a new BUILD file with a) all the old compilation tests and b) all the old learning tests in it
• This file is referenced from the main pipeline.ml.yml file and run by bazel.

E.g. for A2C:

[kouroshHakha]

Can we dry run one of these BAZEL changes on one algorithm and make sure that the release stack works for that one algorithm? Then we can repeat the process for others.

[kouroshHakha]

can we avoid repeating copying run_regression_tests.py for every algorithm?

[kouroshHakha]

can this be something like rllib.tests.run_regression_tests? we need to share this code between algorithms.

[sven1977]

The dry run is already done on A2C. It went ok (all green).
I'll do all the others now to see what it would look like in the end.
I'll also try to somehow make run_release_tests importable from rllib itself, even though this could be tricky as we have rllib pinned to 2.5, so we cannot change anything anymore.

[sven1977]

After latest update:

• All contrib algos' tuned_examples have been moved from rllib into the new respective rllib_contrib/[algo name]/tuned_examples folders
• All contrib algos' tests/test_[algo name].py files have been moved from rllib into the respective rllib_contrib/[algo name]/tests folders
• All contrib algos' learning tests and compilation tests have been removed from rllib's BUILD file and into the respective rllib_contrib/[algo name]/BUILD file.
• in .buildkite/pipeline.ml.yml, all contrib algos now have a couple of extra commands:
• • copy the run_regression_tests.py file from the installed ray[rllib] into a local directory
• • run the rllib_contrib's algos already existing example
• • run the rllib_contrib's algos BUILD file (which includes all learning- and compilation tests) with bazel

[sven1977]

cc: @kouroshHakha , this is ready for final review. Also tests pending ...

[kouroshHakha]

The dry run is already done on A2C. It went ok (all green).

👍

All contrib algos' tuned_examples have been moved from rllib into the new respective rllib_contrib/[algo name]/tuned_examples folders

All checklist items are addressed. However, I don't see the original tuned_example yamls and folders deleted, is that intentional? You want to do deletion in another round?

[kouroshHakha]

approved subject to tests passing. Thanks @sven1977 Great work!!

[kouroshHakha]

This is smart. So this does not create any issues when the dependencies are installed independently within each algorithm, right? We just have to make sure run_regression_tests.py remains backward compatible (which better be the case :)), right?

[sven1977]

Yes, we'll have to make sure this script, which will be "running along" with the evolving RLlib, will remain backward compatible wrt these "old" contrib tuned_examples files. This could potentially become an issue, for example if we decided to get rid of yaml support (and do everything with py config files). Let me think about this some more. Maybe a better alternative would be to copy this file from the actually installed (ray[rllib]) location ...

[sven1977]

I'm getting this script now via a wget from the 2.5.1 github location. So this will always be constant (backward compatible), no matter what we do with the master version of this script.

[kouroshHakha]

should we move examples to the bazel BUILD file?

[sven1977]

Great idea, was thinking about this, too. Will make it cleaner. 👍

[sven1977]

done