-
Notifications
You must be signed in to change notification settings - Fork 5
Regularization FALCON
Regularization is a technique to include additional constrains into an optimization problem. This is done by adding to the cost function, a function of the parameters themselves. For example, Tikhonov regularization, also called L2 or ridge regression or weight decay considers the sum of the squares of the parameters values. In general, regularizations materialize prior knowledge about the parameter distributions, and so are very Bayesian in nature. For example, the Tikhonov penality induces smaller parameter weights, something that makes machine learning models usually more generalizable and moderates overfitting. By increasing incrementally the importance of this additional term in the objective function while controlling for the total goodness of fit, it is possible to choose a threshold to determine the 'best' model. Some regularization schemes are now available for FALCON:
This introduces a partial norm on all parameters. This will induce a simplification in the topology of the network, in the sense that stronger regularization will force the optimization to choose between additive edges to the same nodes and will pick the most important of competitive interactions. It will also have a pruning effect on the negative edges.
This introduces a simple L1 norm on all parameters. Because of the Bayesian structure in FALCON (the sum of activating edges must be equal to 1) this is not useful for activating edges. This will however prune out less important inhibiting edges.
This introduces a simple L2 norm on all parameters. Because of the Bayesian structure in FALCON (the sum of activating edges must be equal to 1) this is not useful for activating edges, actually pushing competitive edges to have more equal contributions. It is not clear what biological assumption this materializes. This will however decrease the weight of less important inhibiting edges.
This penalizes the sum of the absolute differences to the average parameter value for groups of parameters. It will make the parameter values for the same edge more equal across contexts, where contexts are for example cell lines or patients or treatments. By screening regularization strengths, this can determine which edges are probably context specific and which ones are probably not given the data. However in most cases we think this is not the best option when the number of contexts is more than 2. See LCluster below.
This penalizes the sum of cumulative differences across contexts, when the contexts have a natural order, for example timepoints, spatial location, increasing amount of genetic damage etc. This will 'smooth' parameter values and assume that differences between two successive contexts should be minimized. By screening regularization strengths, this can determine which edges are probably context specific (timepoint a change occurs for example) and which ones are probably not given the data. In spirit, this is similar to using convolutions for neural networks, and in general materializes the assumption of local signal correlation.
Combines L1/2 with LGroups.
Combines L1/2 with LGroups with LSmooth. Not well tested.
This is an original contribution of ours (see De Landtsheer et al., 2018, Frontiers on Physiology, xxx). By summing the local deviations to the expected density for all n-dimensional rectangles in the parameter space, we obtain a penality which we demonstrated is useful to group contexts together when there is a high number of them, and that therefore a certain level of heterogeneity is expected. By screening regularization strengths, this can determine which edges are probably context specific and which ones are probably not given the data.
Combines L1/2 with LCluster. Not well tested.
See the driver script Driver_Regularization.m. In order to conduct a regularized optimization on a certain network model over a series of contexts, it is necessary to perform the optimization on all context-specific models simultaneously, as the regularization function is computed from the homologous parameters between the contexts. In other words, we are going to create a super-model with identical input nodes, but parallel signal processing in topologically identical but parametrically free networks and fit each system on its own output nodes.
For example, if we have the conceptual network A -> B -> C, an experimental setup with presence or absence of A, and measurements of C over 3 cell lines, we can construct the following network: A -> B1 -> C1, A -> B2 -> C2, A -> B3 -> C3. This network can the be contextualized on the dataset [C1, C2. C3], and the values of the context-specific edges will only depend on the datapoints of that particular context. This is the same as performing three independent optimizations. However, if we have prior knowledge that the edges between the context should have a certain structure (for example, they should be similar) then we can add a function of the context-specific parameters (k1, k2, k3) to the objective function. This setting-up is done with the FalconMakeGlobalModel() function.
- The variable *estim.Reg* should contain a string and defines the type of regularization applied.
- The variable *estim.RegMatrix* should contain for global models (when there are several contexts) a N-by-P matrix with parameter ID number for N edges and P contexts. For example, a 5-edge network across 3 contexts would basically have the following matrix: [1 2 3; 4 5 6; 7 8 9; 10 11 12; 13 14 15]. It is automatically generated but can be customized before optimization, for example to exclude some edges from regularization or group parameters even further. In the above example we could want to have [1 2 3 4 5 6; 7 8 9 10 11 12] and not care about the last edge.
- the variable *estim.Lambda* should contain a vector with the values of the lambda parameter, or strength of regularization. We recommend screening large ranges at first and increase sampling as much as computationally possible, for example choose values between 2^-20 to 1, with half-log increments (40 simulations).
- For mixed regularizations, we need two or three vectors of lambda values (depending on the case), as we will perform a grid search in two or three dimensions.
After the optimizations are over, the one thing to look for is the most appropriate balance between goodness-of-fit (Mean Squared Error) and model size (number of non-null parameters). Different metrics can be used, we usually prefer the BIC, however AIC often leads to the same conclusion about model topology. We consider 0.01 as the threshold for a parameter to be different from zero.