New optimizer_args tf interface + test optimizer type

[irisliucy]

This PR addresses issue #1062

1. removes optimizer_args pattern from tf algorithm constructors in tf/algos , tf/optimizers and tf/regressors,
2. make members of algorithm class _private if haven't done so,
3. test tf optimizer interface

The tf optimizer interface garage/tf/algos/_utils.py follows the compatibility of TensorFlow Core v2.2.0 and is similar to torch's optimizer interface but accepts an optimizer type only. The algorithm internally constructs an optimizer using a new utility function _make_optimizer.

[ryanjulian]

See my comments

[ryanjulian]

Awesome work!

[naeioi]

Good job! The semantic of make_optimizer() is a bit off from the torch version. It should take parameters from both user and defaults. Please take a look at how torch implements it.

[irisliucy]

I push the unified tf and PyTorch optimizer interface not long ago. There are pickling tests i.e. garage/tf/algos/test_dqn.py failing and I'm looking into the reason why. Any inputs will be appreciated.

A sample of the error in garage/tf/algos/test_dqn.py:
TypeError: can't pickle _thread.RLock objects

Meanwhile, I'll address the rest of the comments.

[ryanjulian]

Be careful with merge commits! They really mess with our CI (which only assumes rebases).

Only use git pull --rebase and never use the "Update branch" button on GitHub.

[irisliucy]

I resolved the python pickle issues on the tf algorithm when using make_optimizer(). After looking into the issue, it may be cause when python cannot pickle lambda expressions in tf.optimizer. I found that by wrapping the make_optimizer() under named functions/ tf.name_scope, as well as using the local variables such as optimizer works. The code will look something like this:

with tf.name_scope(‘init_optimizer’):
            optimizer = make_optimizer(self._optimizer, **self._optimizer_args)
            train_optimizer = optimizer.minimize(
                    …
             )

[ryanjulian]

Hmm -- can cloudpickle not pickle these lambdas? Perhaps the tests are still using pickle, but we switched to cloudpickle for the implementations?

[codecov]

Codecov Report

Merging #1496 into master will increase coverage by 0.00%.
The diff coverage is 93.82%.

@@           Coverage Diff           @@
##           master    #1496   +/-   ##
=======================================
  Coverage   90.81%   90.81%           
=======================================
  Files         211      211           
  Lines       10334    10346   +12     
  Branches     1262     1261    -1     
=======================================
+ Hits         9385     9396   +11     
  Misses        699      699           
- Partials      250      251    +1     

Impacted Files	Coverage Δ
src/garage/torch/_utils.py	100.00% <ø> (ø)
src/garage/tf/regressors/gaussian_mlp_regressor.py	96.66% <75.00%> (ø)
src/garage/tf/algos/ddpg.py	89.61% <87.50%> (+0.20%)	⬆️
src/garage/tf/optimizers/first_order_optimizer.py	75.58% <90.00%> (+1.19%)	⬆️
src/garage/tf/algos/dqn.py	93.47% <94.11%> (+0.07%)	⬆️
src/garage/tf/algos/td3.py	92.85% <95.74%> (+0.12%)	⬆️
src/garage/_functions.py	100.00% <100.00%> (ø)
src/garage/tf/algos/npo.py	95.91% <100.00%> (+0.01%)	⬆️
src/garage/tf/algos/reps.py	96.56% <100.00%> (+0.01%)	⬆️
...rc/garage/tf/regressors/bernoulli_mlp_regressor.py	93.47% <100.00%> (-0.21%)	⬇️
... and 11 more

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update a2b4bfd...bdbf0fa. Read the comment docs.

[naeioi]

This looks very good to me. Please address my tiny comments before submit.