[rllib] move evaluation to trainer.step() such that the result is properly logged

[Maltimore]

Move evaluation to trainer.step() such that the result is properly logged even when training with trainer.train() instead of tune.run()

Why are these changes needed?

When training with trainer.train() instead of tune.run(), the results of the evaluation (evaluation metrics) are not written to disk (e. g. to the progress.csv).

This moves the evaluation code to trainer.step(), which ensures that the evaluation metrics are included in progress.csv and other output files.

Related issue number

Checks

• I've run scripts/format.sh to lint the changes in this PR. --> yes but this errors with a flake8 error unrelated to the code
• I've included any doc changes needed for https://docs.ray.io/en/master/. --> no doc changes needed
• I've made sure the tests are passing. Note that there might be a few flaky tests, see the recent failures at https://flakey-tests.ray.io/
• Testing Strategy
  • Unit tests
  • Release tests
  • This PR is not tested :(

I only tested this PR in my own experiments.

[sven1977]

Looks fine for me.
@Maltimore Could you fix the broken eval test case and run the ci/travis/format.sh (LINT) before pushing?
Thanks!

[Maltimore]

I ran ci/travis/format.sh again and it works now (previously it failed with a flake8 error).

Can you tell me how to run the unit tests for rllib? Or, where can I find the information on how to run the unit tests?
edit: I figured out how to do the eval test now.

[Maltimore]

@sven1977
could you take another look? I simplified the logic in the if condition a bit (I hope I didn't mess it up, maybe if you could check the if condition as well that would be good). The evaluation test case runs through now.

[sven1977]

Thanks for this fix @Maltimore !
Looks good to me.