How to clip the parameters of model?

[L-M-Sherlock]

Due to some constraint, I need to limit the range of each parameter. Here is an implementation in PyTorch:

class WeightClipper(object):

    def __init__(self, frequency=5):
        self.frequency = frequency

    def __call__(self, module):
        # filter the variables to get the ones you want
        if hasattr(module, 'weight'):
            w = module.weight.data
            w = w.clamp(-1,1)


model = Net()
clipper = WeightClipper()
model.apply(clipper)

Source: https://discuss.pytorch.org/t/set-constraints-on-parameters-or-layers

I try to clip the parameters of model during optimization with this code:

    def weight_clip(self):
        w = self.w.numpy()
        grad = self.w.grad
        w[0] = w[0].clip(1, 10)
        w[1] = w[1].clip(0.1, 5)
        w[2] = w[2].clip(0.1, 5)
        w[3] = w[3].clip(0, 0.5)
        w[4] = w[4].clip(0, 2)
        w[5] = w[5].clip(0.1, 0.8)
        w[6] = w[6].clip(0.01, 1.5)
        w[7] = w[7].clip(0.5, 5)
        w[8] = w[8].clip(0.01, 0.2)
        w[9] = w[9].clip(0.01, 0.9)
        w[10] = w[10].clip(0.01, 2)
        w[11] = w[11].clip(0, 1)
        w[12] = w[12].clip(1, 10)
        self.w = Tensor(w)
        self.w.grad = grad

But I get this error:

Traceback (most recent call last):
  File "/Users/jarrettye/Codes/open-spaced-repetition/fsrs-optimizer-tiny/script.py", line 192, in <module>
    optim.step()
  File "/Users/jarrettye/Codes/open-spaced-repetition/tinygrad/tinygrad/nn/optim.py", line 56, in step
    assert t.grad is not None
AssertionError

[wozeparrot]

You should be able to just call Tensor.clip:

tinygrad/tinygrad/tensor.py

Line 534 in 52db7d7

def clip(self, min_, max_): return self.maximum(min_).minimum(max_)

in the forward pass

[L-M-Sherlock]

Is it possible to pass a list of min values and max values to clip the tensor?

[wozeparrot]

No, you can only specify a single min/max, and clip the whole tensor to those bounds.

[L-M-Sherlock]

Fine. Maybe I need to split the current tensor into 17 tensors.

[wozeparrot]

you can do something like:

# t is your tensor
w = t.chunk(17, 0)

w[0] = w[0].clip(1, 10)
w[1] = w[1].clip(0.1, 5)
w[2] = w[2].clip(0.1, 5)
w[3] = w[3].clip(0, 0.5)
w[4] = w[4].clip(0, 2)
w[5] = w[5].clip(0.1, 0.8)
w[6] = w[6].clip(0.01, 1.5)
w[7] = w[7].clip(0.5, 5)
w[8] = w[8].clip(0.01, 0.2)
w[9] = w[9].clip(0.01, 0.9)
w[10] = w[10].clip(0.01, 2)
w[11] = w[11].clip(0, 1)
w[12] = w[12].clip(1, 10)

t = Tensor.cat(*w)

which will keep the gradient and let you clip individual elements as well.

[L-M-Sherlock]

Perfect! That's what I need. Thank you!

[L-M-Sherlock]

@wozeparrot, sorry for bothering you. But I meet another error when I clip the model's weights with your code:

Traceback (most recent call last):
  File "/Users/jarrettye/Codes/open-spaced-repetition/fsrs-optimizer-tiny/seq2seq.py", line 188, in <module>
    avg_loss.backward()
  File "/Users/jarrettye/Codes/open-spaced-repetition/tinygrad/tinygrad/tensor.py", line 224, in backward
    for t0 in reversed(self.deepwalk()):
  File "/Users/jarrettye/Codes/open-spaced-repetition/tinygrad/tinygrad/tensor.py", line 215, in deepwalk
    return _deepwalk(self, set(), [])
  File "/Users/jarrettye/Codes/open-spaced-repetition/tinygrad/tinygrad/tensor.py", line 212, in _deepwalk
    if i not in visited: _deepwalk(i, visited, nodes)
  File "/Users/jarrettye/Codes/open-spaced-repetition/tinygrad/tinygrad/tensor.py", line 212, in _deepwalk
    if i not in visited: _deepwalk(i, visited, nodes)
  File "/Users/jarrettye/Codes/open-spaced-repetition/tinygrad/tinygrad/tensor.py", line 212, in _deepwalk
    if i not in visited: _deepwalk(i, visited, nodes)
  [Previous line repeated 48 more times]
  File "/Users/jarrettye/Codes/open-spaced-repetition/tinygrad/tinygrad/tensor.py", line 210, in _deepwalk
    if node._ctx:
AttributeError: _ctx

[L-M-Sherlock]

I think the t = Tensor.cat(*w) would block the gradient backprop?

[wozeparrot]

cat doesn't block backprop, do you have the full model code I could look at?

[L-M-Sherlock]

https://github.com/open-spaced-repetition/fsrs-optimizer-tiny/blob/main/seq2seq.py

[wozeparrot]

the numpy usage here (https://github.com/open-spaced-repetition/fsrs-optimizer-tiny/blob/7b199bec90c3fb3d6093af0201b5d1049275960a/seq2seq.py#L41-L46) is probably blocking the gradient, if the gradient doesn't have to flow through this part you can add requires_grad=False to the Tensor creation.

[L-M-Sherlock]

I added requires_grad=False to the Tensor creation. But AttributeError: _ctx still exists.