Support non-linearity function and module for PyTorch Module

[yonghyuc]

Throw an error because of using non-linear functions in torch.nn.
Change into using non-linear funcitons in torch.nn.functional or torch

Previously, MultiHeadedMLPModule assumes that we are using non-linear function in torch.nn.
So we make a computational graph inside of __init__ and pass the type of the non-linear function to ModuleList. (layer.add_module('non_linearity', hidden_nonlinearity))

However, for consistency with other modules, I changed the way we use the non-linear functions.
(Similar with MLPModule, ) make linear layers only in __init__ and call the non-linear functions in the forward function. (x = self._hidden_nonlinearity(x))

So now, we need to pass non-linear functions in the torch.nn.functional (ex. torch.tanh or torch.nn.functional.relu) instead of in the torch.nn (ex torch.nn.Tanh, torch.nn.ReLU)

[codecov]

Codecov Report

Merging #928 into master will increase coverage by 0.02%.
The diff coverage is 97.72%.

@@            Coverage Diff            @@
##           master    #928      +/-   ##
=========================================
+ Coverage   85.27%   85.3%   +0.02%     
=========================================
  Files         160     160              
  Lines        7674    7661      -13     
  Branches      958     955       -3     
=========================================
- Hits         6544    6535       -9     
  Misses        937     937              
+ Partials      193     189       -4

Impacted Files	Coverage Δ
src/garage/torch/modules/mlp_module.py	100% <100%> (+5.88%)	⬆️
...rc/garage/torch/modules/multi_headed_mlp_module.py	98.36% <97.29%> (+15.02%)	⬆️
src/garage/envs/grid_world_env.py	84.12% <0%> (-4.77%)	⬇️
src/garage/misc/tensor_utils.py	88.23% <0%> (-2.36%)	⬇️

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update ebea1dd...987dbaf. Read the comment docs.

[ryanjulian]

I'm a little confused about this PR.

The title suggests that it's fixing a non-linearity in multi_headed_mlp_module (perhaps the default was wrong).

The content seems to rewrite how heads in this module are created, and also adds some features (layer norms, selectable output nonlinearities).

Can you add some context behind the change and explain what it does?

[yonghyuc]

I'm a little confused about this PR.

The title suggests that it's fixing a non-linearity in multi_headed_mlp_module (perhaps the default was wrong).

The content seems to rewrite how heads in this module are created, and also adds some features (layer norms, selectable output nonlinearities).

Can you add some context behind the change and explain what it does?

I updated the description above.
Basically, everything is same except how we handle the non-linear functions of hidden and output

[avnishn]

@yonghyuc the changes look to be making the format of this module more standard with what Linda wrote for other torch modules originally. I want to know if there is a merit to adding out activation functions to the nn.modulelist as opposed to using the torch.functional non linearity functions directly in the forward method of the module.

One benefit that I see is that if a user wants to quickly verify the structure of a module by printing it out (e.g. a policy) if we use nn.modules for the activation functions and add them to the nn.modulelist, then the nonlinearities will appear in the printed output, but if we use the torch.functional activiation functions, they won't appear when we print out the module. I'm wondering if you, @lywong92, and @krzentner can chime in on this design decision.

[avnishn]

just to be clear, what I'm trying to avoid is having to manually call the non linearities. I think they should just be defined as computations in the nn.ModuleList

[krzentner]

We should have removed the need for this loop, right?

[avnishn]

LGTM, we'll add support for both nn.nonlinearities and torch.functional nonlinearities in a future pr.

[krzentner]

You should be able to replace this function with return self._layers(input_values).

[yonghyuc]

self.layers is ModuleList, so I think we need to use for loop here.
(I checked this documentation)
Or we can use Sequential to compute the layers like that.

[krzentner]

Perhaps we should actually globally allow not documenting __methods__? It's fairly obvious what __repr__, __getstate__, and __setstate__ do.

In the mean time, yeah. Documenting looks cleaner than locally disabling the rule.

[krzentner]

We should have removed the need for this loop, right?

[ryanjulian]

LGTM -- I removed ready-to-merge so you can look at my last comments, but feel free to add it back when you're ready.

[ryanjulian]

@Mergifyio backport release-2019.10

[mergify]

Command backport release-2019.10: pending

Waiting for the pull request to get merged

Hey, we reacted but our real name is @Mergifyio

[ryanjulian]

@Mergifyio backport release-2019.10

[mergify]

Command backport release-2019.10: success

Backports have been created
The following pull requests have been created:

• #1038 Support non-linearity function and module for PyTorch Module (bp #928)

Hey, we reacted but our real name is @Mergifyio