Multiple variable contraint

[NEGU93]

Click to expand!

Issue Type

Feature Request

Source

source

Tensorflow Version

2.4.1

Custom Code

No

OS Platform and Distribution

Linux Ubuntu

Mobile device

No response

Python version

3.8.8

Bazel version

No response

GCC/Compiler version

No response

CUDA/cuDNN version

No response

GPU model and memory

No response

Current Behaviour?

Based on this [Stackoverflow question](https://stackoverflow.com/questions/68353001/keras-weight-contraints-including-both-bias-and-kernel-matrix-at-the-same-time).

It would be nice to have the possiblilty to apply contraints to multiple variables at the same time. For example, one could apply a maxnorm contraint to a Dense layer that takes into account weigths and biases all together.

Standalone code to reproduce the issue

There are many UX options here. One possibility (but less scalable for other use cases) is to add the parameter `kernel_bias_contraint` to the Dense layer.
A better one would be to let `tf.keras.constraints.Constraint` to receive multiple variables.
Both options are non exclusive.

Relevant log output

No response

[sushreebarsa]

@NEGU93 Could you elaborate about your feature and please specify the use cases for this feature. Thank you!

[NEGU93]

Sure.

Example 1:
The example of StackOverflow, and, what I meant in the phrase:

For example, one could apply a maxnorm contraint to a Dense layer that takes into account weigths and biases all together.

Is that maybe a constraint on the norm for weights and biases separately are not enough. Maybe the bias max norm should be lower if the norm of the weigths is already big.

Example 2

My use case (so, another example), is that I do a Complex-Valued layer by using real and imaginary parts separately and simulating a complex operation. I want to impose contraint on the norm of the absolute value, which is not possible if I don't know both values at the same time when computing the norm.

Basically, the constraint is applied by knowing or using another variable's value. I would therefore like for the constraint to receive and change two variables at the same time. in the weight/bias example the constraint will receive both self.w and self.b and perform maxnorm on the concatenation of both values. In the complex-valued case, it will compute the absolute value, perform maxnorm and change real and imag parts to achieve the needed normalized absolute values.

I hope this is clearer.

Regards,

[rchao]

Hello @NEGU93 , thanks for the request. Would you mind providing usage code snippets of the apis you're proposing so it's clearer what the request is here, for the following two options?

There are many UX options here. One possibility (but less scalable for other use cases) is to add the parameter kernel_bias_contraint to the Dense layer.

A better one would be to let tf.keras.constraints.Constraint to receive multiple variables.
Both options are non exclusive.

[rchao]

Giving it some thoughts, does it appear that you'd need both 1) the Dense layer to take kernel_bias_contraint and 2) tf.keras.constraints.Constraint to receive multiple (2 in this case) variables to achieve what you're looking for?

Also, assuming this gets an okay from Keras team to proceed, would you be interested in contributing?

[NEGU93]

Wouldn't the second option easily allow for the implementation of the first one? I think about this pseudo-code:

def __call__(self, w, b):        # the multi variable contraint
    w = weight_contraint(w, b: Optional)      # maybe I need b to compute  the restraint? why not.
    b = bias_contraint(b)        # same, Optionally I could also give w as const a parameter.
    return w, b

A more generic implementation will use *args and/or **kwargs to allow any number of variables.

In any case, is more the second one that I am particuarly interested. As you noted, what I need is to be able to receive multiple variables.

I am willing to explain as much as possible and even have a short video call to discuss the feature if needed.

Regarding the code I would love to contribute but I make no promises as I am at the end of my PhD. I need to finish my manuscript and I will also soon start looking for a permantent job somewhere in the world. I am sorry!

PS: Another example on why this feature might be useful.
I use TF as backend for my Complex-Valued TensorFlow (kind of).
There is a feature request I received that will be possible if this feature I ask is implemented.

[rchao]

Thanks for the quick response @NEGU93 - from what you described it appears using a custom layer can achieve this. Is there a reason why you'd like an additional type of constraint than say, and subclass of Dense?

[NEGU93]

My custom layer cannot achieve this yet. I am not sure how can I do it.
If it's possible, how can I do it? Here an example of what I need (or how I imagined the solution):

class AmplitudeConstraint(tf...Constraint):
    def __call__(self, real_w, imag_w):        # the multi variable constraint
        real_w, imag_w = amplitude_constraint(real_w, imag_w)  
        return real_w, imag_w

In the example, I set a constraint on the amplitude value, also a constraint on phase would be possible. It depends on what the user of my library wants.

The current solution is a constraint on both variables separately but they don't have information from each other (needed to know the amplitude or phase).

[n-j-p]

@NEGU93 Could you elaborate about your feature and please specify the use cases for this feature. Thank you!

Hi. I stumbled upon this feature request after finding the stack overflow page referenced above.

I am trying to fit a (one neuron) simpleRNN layer, and wanted to constrain the absolute sum of the recurrent and kernel weights to be less than 1. If my understanding is correct, this would ensure stationary (i.e. decaying) predictions/projections for a linear activation function.

I can do this by including a regularization parameter, but it would be good to do this with a constraint. As far as I am aware, this is currently impossible/difficult as mentioned above since constraints work on recurrent and kernels separately.

[moralapablo]

Hi. Are there any updates regarding this issue?

The original StackOverflow post that is referenced in this issue contains a solution with a custom layer. However this is not practical for many applications like adding the constraint to an existing layer without having to recreate the whole network, or generalizing this kind of constraints to several layer types.

I think that a general way of constraining both kernel and bias at the same time is useful as there are mathecatical situations where this constraints have to affect to the whole vector of weights at one neuron, that is, the weights form the kernel and its associated bias.

Furthermore, having to use the custom solution that joins the kernel and bias as a single matrix supposes that when extracting the weights from the whole network will not follow standard behaviour, as now kernel and bias will be in a single matrix, while in a usual dense layer they will be separated. This makes it difficult to generalize the use of these custom layers.