ENH: Gamma Scaling Experiments

[nray]

Context
In Kosta’s papers they represent a single and two layer NN as a random object and study its bias and variance. In particular, they introduce a scaled normalization of each layer’s output before it is fed to the next layer as in the reference Normalization effects on deep neural networks, Equation 1.

We are interested in studying if such scaling has any effect on RVFL type of networks w.r.t their performance . Do we get any guidance from numerical studies if gamma scaling has any effect on RVFL's generalization properties ?

In rest of the issue, I am using the term gamma to mean gamma scaled normalization.

General questions:

1. Is there a universal gamma or a range that leads to consistent performance across multiple datasets ?
   1. This is unlikely and conflicts with “No-free lunch theorem”.
   2. Kosta’s response seemed to agree that this is not “free-lunch”
2. Are there specific properties of the dataset that are correlated with the gamma, so that we can utilize a-priori if such scaling would be useful for improving RVFL accuracy ?
   1. Kosta’s provided an expression for variance $\sigma(NN)=CN^{2/\gamma-1}e^{-At}$ , where $N-> \infty$, $t$ is running time, $C(X, w, a)$ is constant and function of input $X$, initialization $w$ and activation $a$, $A$ is a positive definite matrix, and A’s eigenvalues depend on $X, w, a$. But, the functional form of $C$ and $A$ in terms of $X, w, a$ is not clear.
   2. Also, how variance $\sigma(NN)$ is related to accuracy of the function approximation is something I do not fully understand.

Experiment assumptions
Activation function: non-polynomials and slowly increasing i.e., $tanh, sigmoid$
Gamma $\gamma$ : $[1/2, 1]$
Initialization weight function : normal

Experiment 1: Study the effect of $\gamma$ on solution accuracy for RVFL with direct solve, is there one specific value that gives the best approximation ?

Requirement : We need to scale the output of each layer before feeding to the next layer as described in the
reference.

How to code this up in GFDL library code:

1. Introduce an extra parameter to our GFDL base class as in here.
2. Scale the design matrix for each layer after the activation function has been applied as in here.

Note: This design does not scale the input to first layer or the direct links.

Experiment 2 : Confirmation of experiment 1 with iterative solve.

Kosta’s response to why an iterative solver is needed:

Direct solution is great, but it is also hard to analyze analytically. If the solvers converge, it makes sense to analyze the limit for large enough N. If it does not converge, it could mean different things, the algorithm may hit a local minimum, may be artifact of numerical algorithm. Hope is it should converge.

Constraint: We need to specify the learning rate explicitly, so the learning rate has to be a hyper-parameter or at the very least exposed to the solver api even if it is constant.

Why Ridge solver in current library cannot be re-purposed ?

1. Currently, "reg_alpha=None" always leads to the direct solver path.
2. Even if we manage to call the Ridge solver when "reg_alpha=None", the learning rate for sgd solver used by Ridge is a heuristic and not exposed through their api.

How to code this up ?

1. Implement standalone solvers (current approach, but I am considering the second option to avoid maintaining such solvers on our own)
2. Use other SGD implementations like torch.optim or scikit learning SGDRegressor, etc.

Current design for exposing iterative solvers for the ordinary least-squares formulation is to add a solver type to "fit" method args as in here along with kwargs to process solver specific arguments.

[tylerjereddy]

Kosta’s response seemed to agree that this is not “free-lunch”

It would be good to clarify this in simple terms that clearly describe the scenarios where it may/may not work, ideally as some kind of code we could use on a design matrix/model combo, which also helps bridge the gap between mathematicians and programmers. I feel like I'm still clueless here, basically.

But, the functional form of $C$ and $A$ in terms of $X, w, a$ is not clear.

Ok, I really have no idea based on this whether gamma scaling is genuinely useful to consider across a broad range of design matrices--can we recast this in terms I/a simple programmer can understand?

Also, how variance $\sigma(NN)$ is related to accuracy of the function approximation is something I do not fully understand.

It seems like there is more confusion than clarity here at the moment, so I'm a bit hesitant to go deeper until it is really clear that we need this and it makes sense.

We need to scale the output of each layer before feeding to the next layer as described in the
reference.

I think the above points should be clarified first, but after that it would be good if @eiviani-lanl could confirm that we definitely can't do this without a library code change.

Even if we manage to call the Ridge solver when "reg_alpha=None", the learning rate for sgd solver used by Ridge is a heuristic and not exposed through their api.

As you mention farther down, do we really need to be rolling our own SGD? sklearn has SGDRegressor with learning_rate, eta0, and verbose settings. If that suffices, it saves a fair bit of review/maintenance burden. If not, would be good to clarify why.

[tylerjereddy]

One even crazier thought on this point:

We need to scale the output of each layer before feeding to the next layer as described in the
reference.

Is it possible to achieve the same outcome as this scaling by synthetic manipulation of the weights? In other words, can you achieve a network with equivalent behavior to whatever custom gamma scaling you want by manipulating the weights to pass information between layers in a manner equivalent to what happens via scaling?

I think that's the same question as--instead of providing $\gamma$ could you in principle accept an equivalent effective distribution of weights? That may be far more confusing than just providing $\gamma$, but I am mostly just thinking about the design and requirements on the API. For example, by providing $\gamma$ are we really just providing an extra route for manipulation of weights beyond the weight initialization itself?

[tylerjereddy]

Is it possible to achieve the same outcome as this scaling by synthetic manipulation of the weights?

The email chain with Kostas and Navamita/Tyler on March 29th suggests that the answer to this is "yes." An example is Xavier (Glorot) initialization of the random weights with $\gamma = 0$ being equivalent in model performance to normally distributed random weights with $\gamma = 0.5$ on some of the *MNIST dataset experiments with GFDLClassifier--both are just manipulating the weights/terms equivalently it seems.

Although we may still want to allow $\gamma$ scaling to empower straightforward comparison with our collaborators doing such scaling from a theoretical standpoint, it does give me some pause to have multiple different ways to manipulate what is ultimately just the random weights/terms of the NN matrix.

Some specific considerations might be:

1. Do we want to allow $\gamma$ scaling given that we can achieve the same outcome with custom weight initialization?
2. For random matrix theory-based explorations, wouldn't we typically want to provide the entire random weight matrix so that its spectrum (or other values) are know to us directly, vs. the indirect control from $\gamma$ scaling?
3. The BU team mentioned scaling different layers of the NN differently, so if we were to allow $\gamma$ as a parameter, should it be an array-like rather than a float, so there is the option of specifying per layer? This was also noted in WIP, ENH: Add support for gamma scaling and iterative solvers. #70.

[tylerjereddy]

Shahyad may try to send a write-up on their approach to gamma scaling I think.

The correct extension of gamma scaling to non-rectangular networks wasn't clear to me since only the number of neurons and not their distribution between layers currently seems to be factored in? I'm still quite confused here. It also sounds like applying the gamma scaling to the direct solve happens after "beta" is produced, which seems to be different from my proposal of scaling the initial random weights...

[tylerjereddy]

In a meeting with Kostas on April 15/2026, it was helpfully clarified that gamma scaling can't provably outperform non-gamma-scaled networks on test performance, even in the single layer/infinite neuron/learned scenario that Kostas has previously published.

However, the idea is that gamma scaling may enable achievement of "solid" performance on validation/test in less (hyperopt) trials than non-gamma-scaled case using some rules of thumb. Demonstrating this practical value vs. our more brute force hyperopt might be one way to integrate into a paper focused on numerical experiments/test performance (i.e., gamma scaled case gets to 85 % test accuracy in 10 trials instead of 30 trials without scaling).