Epsilon-scaled clipped versions of Softmax and Sigmoid as utilities

[rohitsingh0812]

Blocking #107

Epsilon-scaled clipped versions of Softmax and Sigmoid as utilities that can be used to avoid numerical unstability with Pytorch.nn Softmax and Sigmoid operations

Tested

Added simple tests for both new classes.

[fritzo]

Please fix lint errors:

./pyro/nn/clipped_nn.py:2:1: F401 'torch' imported but unused
./pyro/nn/clipped_nn.py:5:1: E302 expected 2 blank lines, found 1
./pyro/nn/__init__.py:5:1: F401 '.clipped_nn.ClippedSigmoid' imported but unused
./pyro/nn/__init__.py:5:1: F401 '.clipped_nn.ClippedSoftmax' imported but unused
./pyro/nn/__init__.py:5:55: W292 no newline at end of file

[fritzo]

I think this should really be

n = rval.shape(self.dim)
return (rval * (1.0 - n * self.epsilon)) + self.epsilon

so that the clipped value is still normalized and so gradients still sum to zero.

[neerajprad]

Where does the dim here come from? Is it an attribute of nn.Softmax?

[fritzo]

yes: http://pytorch.org/docs/master/nn.html#torch.nn.Softmax

[neerajprad]

@fritzo - that'll be there in the next release, but its not in the current release. Maybe we should add a comment to that effect?

[fritzo]

so maybe getattr(self, 'dim', -1)

[neerajprad]

that should work!

[fritzo]

Should be size rather than shape

[fritzo]

(Note that Tensor.shape is not available on PyTorch 0.2 release, whereas Tensor.size() is)

[neerajprad]

Can we also have some simple tests for the clipped versions of sigmoid and softmax?

[fritzo]

nit: Outdent comment and wrap code in `` to satisfy sphinx:

    """
    A wrapper around `nn.Softmax` that scales its output
    from `[0,1]` to `[epsilon,1-epsilon]`.
    """

Also fix below comment. Note that this is important since we've had sphinx errors due to code not wrapped in ticks.

[fritzo]

Looks good. This just needs some simple tests like

def test_clipped_softmax():
    epsilon = 1e-5
    clipped_softmax = ClippedSoftmax(epsilon, dim=0)
    ps = Variable(torch.Tensor([0, 1, 2]))
    softmax_ps = clipped_softmax(ps)
    assert (softmax_ps.data >= epsilon).all(), (softmax_ps, epsilon)
    assert_equal(softmax_ps.data.sum(), 1.0)

[fritzo]

@rohitsingh0812 I've added some tests. Could you PTAL?

[jpchen]

merging due to peer pressure