[RLlib] [CI] Deflake longer running RLlib learning tests for off policy algorithms. Fix seeding issue in TransformedAction Environments

[avnishn]

Hitting 10k timesteps seems to be not achievable
in the 900 second time limit. This means that the
reward threshold must be met in order for the
experiment to terminate. This algorithm is seed
sensitive on this environment, so I toyed with the reward
threshold on various seeds, and ended up fixing the
reward threshold for a certain seed. This should
give us regression information, while eliminating the
possiblilty that the test flakes.

Why are these changes needed?

Related issue number

Checks

• I've run scripts/format.sh to lint the changes in this PR.
• I've included any doc changes needed for https://docs.ray.io/en/master/.
• I've made sure the tests are passing. Note that there might be a few flaky tests, see the recent failures at https://flakey-tests.ray.io/
• Testing Strategy
  • Unit tests
  • Release tests
  • This PR is not tested :(

[richardliaw]

Hey @avnishn, I'm not understanding why we have to make this change.

We've slowly lowered this reward threshold from -300 to -1000 over the last 7 months; is there a reason why it used to work and now it doesn't?

[avnishn]

So here's my best guess @richardliaw.

At some point, we probably tried to decrease the runtime of these tests, by looking at successful reward curves, and picking a reward threshold that SAC should have hit for this environment **within a certain number of timesteps -- so in this case, -700 within 10k timesteps.

If you remove the 10k timestep stopping criteria, then SAC can achieve a reward of -150 in 30k timesteps. (As of today)** The original regression test had no stopping criteria based on the number of timesteps that we've trained on, so it probably worked, but was slower in its runtime.

I just increase the number of timesteps for the stopping threshold, and increased the reward threshold back to -150. Lets see if the BK runner can run for this many timesteps in the allotted 900s.

[avnishn]

I moved the pendulum sac tests to gpu and that seemed to fix the problem. This is ready to merge.

[gjoliver]

looks reasonable.
a couple of questions.

[gjoliver]

why do you want to force this? we should respect users' input.

[avnishn]

We only inject this environment variable in our tests. I wanted to add gpu to our yaml file for the tuned example, but couldn't because then it wouldn't run on most users laptops. So instead, I look for the env variable, which is injected by gpu runners on buildkite, so that tests on the gpu runner take advantage of the gpu on the runner.

[gjoliver]

oh understand why you need to check the env var this way, I am more suggesting the following:

num_gpus = int(os.environ.get("RLLIB_NUM_GPUS", "0"))
if num_gpus:
    exp["config"]["num_gpus"] = num_gpus

I was only curious about the hardcoded number 1 there. I don't see why we want to force the test to only use 1 GPU.

[avnishn]

Oh silly me I'm dumb. Yeah you're right

[gjoliver]

is this necessary? harder to say where the difference comes from if we change multiple variables in a single PR.
I am hoping using prioritized_replay would help learning, but maybe slowing things down a bit?

[avnishn]

I can change it back, but it's possible that this hurts and doesn't help. I'll remove it and ablate the change, then post the learning curve in the channel.

[gjoliver]

cool, I am really curious to see the learning curve. if we need this, we can keep it no problem, but would love to have a bit better understanding.

[sven1977]

What does the "X" mean?

[avnishn]

The gpu runner buildkite tags for rllib are named gpu_rllib_1 and gpu_rllib_2. I thought c would be a nice placeholder for a number.

[sven1977]

Wait, shouldn't we do the framework setting the same as the non-GPU tests? Meaning, no framework specifier and the buildkite pipeline definitions pick the correct command line options?

I understand we probably need more than one GPU BK job then. Let me know if this is what you tried to avoid. Happy to do it this way for some limited amount of time.

[avnishn]

We could do it the old way, but that would require 3 gpu bk runners instead of 2. I'm not opposed, just didn't want to add bk runners unnecessarily.

I could add more runners though. With 4-5 tests on the runner, the setup for gpu was taking like 30 minutes, with 20 minutes of tests. Perhaps it was a one off, but that lead to a 50 minute test runner, which is probably at the limit of what we want in terms of test time.

[sven1977]

Cool, let's add another runner, then. This shouldn't be a problem.

[avnishn]

Using @krfricke's awesome tool for reproducing ci runners, I discovered that the flakey tests don't actually run any faster on gpu. I'm going to revert some of my changes here, including adding multi-gpu runners for the flakey tests.

I also discovered that there wasn't a performance regression in these tests, but rather that pendulum v1 has a different reward function/different dynamics that make it harder to learn, vs v0, for which the test's stop criteria and hparams were based on.

See my new comment below

[avnishn]

So there was also one other h_param changed since the test was created: n_step: 1 ->3 what this did was it replaced the reward for the current timestep with the summed discounted return over 3 steps. I'm not sure why we did this, but when I ablated the change, changing back from 3 to 1 seemed to bring the learning speed back to the initial speeds from when the test was first created ... when I shared my initial thoughts this morning, I hadn't realized that I had made this change from 3 back to 1 for this hparam, but only caught it when looking at my changes with a diff tool.

[avnishn]

meanwhile @gjoliver running this experiment with prioritized replay vs without prioritized replay seems to not have made a difference.

These are 6 runs, where 3 have prioritized replay, and 3 don't.

Meanwhile changing n_steps 3 -> 1 has made it so that the number of timesteps needed to meet the stopping criteria from ~30k -> ~10k

[avnishn]

I'm having strange problems with the tuned example that runs the SAC on pendulum transformed actions environment.

I've fixed the seed of my experiment, and am using the same hparams otherwise, but am seeing that the results of the experiment are different across runs:

This reminds of some previous issues that we have had.

[gjoliver]

meanwhile @gjoliver running this experiment with prioritized replay vs without prioritized replay seems to not have made a difference.

These are 6 runs, where 3 have prioritized replay, and 3 don't.

Meanwhile changing n_steps 3 -> 1 has made it so that the number of timesteps needed to meet the stopping criteria from ~30k -> ~10k

very cool! then let's keep that diff out of this change.
the n_step difference is quite puzzling, but it's RL, so I am not surprised :)

[avnishn]

Here's a google colab describing some changes that I'm going to make to the TransformedActionSpaceEnv Environments in rllib.examples.

https://colab.research.google.com/drive/1BhoZNuG-9NDHqZ8bN7BgPJ_9i0jWWEnr?usp=sharing

It explains how seeding was not implemented correctly by the TransformedActionSpaceEnv .

This contributed to the flakiness in the tests that use this environment.

[avnishn]

The final extremely flakey tests were the ddpg continuous learning tests. I updated them after running them each for 12 seeds (had to given our seeding issue, and then found a rough lower bound for the reward after a certain number of ts runnable on the CI. It has been lowered on the fake gpu ddpg pendulum test, but that is because I also reduced the number of timesteps to hit the reward threshold.

@richardliaw could you please merge when tests are passing?

[richardliaw]

ping me when tests pass.

…

On Thu, Jan 27, 2022 at 3:02 PM Avnish Narayan ***@***.***> wrote: The final extremely flakey tests were the ddpg continuous learning tests. I updated them after running them each for 12 seeds (had to given our seeding issue, and then found a rough lower bound for the reward after a certain number of ts runnable on the CI. It has been lowered on the fake gpu ddpg pendulum test, but that is because I also reduced the number of timesteps to hit the reward threshold. @richardliaw <https://github.com/richardliaw> could you please merge when tests are passing? — Reply to this email directly, view it on GitHub <#21685 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ABCRZZPCD7QQ66UHNSZ7HCTUYHFJVANCNFSM5MILB3GQ> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[bveeramani]

‼️ ACTION REQUIRED ‼️

We've switched our code formatter from YAPF to Black (see #21311).

To prevent issues with merging your code, here's what you'll need to do:

1. Install Black

pip install -I black==21.12b0

2. Format changed files with Black

curl -o format-changed.sh https://gist.githubusercontent.com/bveeramani/42ef0e9e387b755a8a735b084af976f2/raw/7631276790765d555c423b8db2b679fd957b984a/format-changed.sh
chmod +x ./format-changed.sh
./format-changed.sh
rm format-changed.sh

3. Commit your changes.

git add --all
git commit -m "Format Python code with Black"

4. Merge master into your branch.

git pull upstream master

5. Resolve merge conflicts (if necessary).

After running these steps, you'll have the updated format.sh.

[avnishn]

@sven1977 I ended up adding a new test runner for tf2 eager long learning continuous off policy tests. In that target I lower the reward threshold for tf2 eager pendulum sac and ddpg tests, since on tf2 eager they run at half the speed of their tf1 and torch counterparts. Could you PTAL and merge if you're ok with this?

[gjoliver]

What?? Is this a performance regression?
We should get to the bottom of this instead of trying to get the tests to pass.

[avnishn]

What?? Is this a performance regression?

We should get to the bottom of this instead of trying to get the tests to pass.

I think we've been aware for a while that tf2 run in eager execution mode is 1.7 times slower than tf2 graph mode and tf1.

But I could be wrong. I'll let @sven1977 comment on that.

[sven1977]

Looks great! Thanks for fixing and stabilizing these!

[sven1977]

Yeah, n_step=1 better than n_step=3 is weird. But we did add thorough n-step tests lately and I don't think we have a bug there anymore (as we used to). But still something to keep on our radar.

[sven1977]

Just waiting for tests ...