[RLlib] IMPALA (any tf algo using entropy coeff) slowdown observed

[sven1977]

Since ray 1.2, IMPALA (and any other tf algorithm that uses entropy_coeff) has slowed down due to a bug.

• The bug is caused by a tf-(static graph)-op being added to the graph each time we call on_global_var_update (defined inside the EntropyCoeffSchedule (tf) mixin class).

To reproduce:

rllib train -f rllib/tuned_examples/impala/pong-impala-fast.yaml

Runs much slower than in ray<=1.1.

What is the problem?

Ray version and other system information (Python version, TensorFlow version, OS):

Reproduction (REQUIRED)

Please provide a short code snippet (less than 50 lines if possible) that can be copy-pasted to reproduce the issue. The snippet should have no external library dependencies (i.e., use fake or mock data / environments):

If the code snippet cannot be run by itself, the issue will be closed with "needs-repro-script".

• I have verified my script runs in a clean environment and reproduces the issue.
• I have verified the issue also occurs with the latest wheels.