scnn
requires Python 3.6
or newer. The package is available through pip
,
pip install scnn
Alternatively, you can use the most recent build on GitHub,
git clone https://github.com/pilancilab/scnn
python -m pip install .
scnn
reformulates training a neural network as a convex program. The easiest way to solve this program is to call optimize <scnn.optimize.optimize>
on a training set with a few additional configuration parameters:
from scnn.optimize import optimize
from scnn.regularizers import NeuronGL1
model, metrics = optimize(formulation="relu",
max_neurons=500,
X_train=X,
y_train=y,
regularizer=NeuronGL1(0.001),
device="cpu")
# training accuracy
train_acc = np.sum(np.sign(model(X_train)) == y_train) / len(y_train)
The formulation
parameter specifies what kind of neural network to train. We currently support convex reformulations for two-layer models with the following activation functions:
"relu"
: the standard rectified linear unit (ReLU),
ϕ(x, w) = max {⟨w, x⟩, 0}
"gated_relu"
: the Gated ReLU activation, which uses a fixed gate vector g ∈ ℝd when computing the activation,
ϕg(x, w) = 1(⟨g, x⟩ > 0)⟨w, x⟩
By default, optimize <scnn.optimize.optimize>
returns a neuron sparse solution. The maximum size of the hidden layer is controlled by max_neurons
, while the degree of sparsity is tuned by the regularization strength, lam
. GPU acceleration is supported and specified using the device
parameter.
The scnn
module provides an object-oriented interface for greater control over the problem parameters and solver settings. The following example trains the same ReLU model using the object-oriented interface:
from scnn.optimize import optimize_model, sample_gate_vectors
from scnn.models import ConvexReLU
from scnn.solvers import RFISTA
from scnn.metrics import Metrics
# create convex reformulation
max_neurons = 500
G = sample_gate_vectors(np.random.default_rng(123), d, max_neurons)
model = ConvexReLU(G)
# specify regularizer and solver
regularizer = NeuronGL1(lam=0.001)
solver = AL(model, tol=1e-6)
# choose metrics to collect during training
metrics = Metrics(model_loss=True,
train_accuracy=True,
test_accuracy=True,
neuron_sparsity=True)
# train model!
model, metrics = optimize_model(model,
solver,
metrics,
X_train,
y_train,
X_test,
y_test,
regularizer,
device="cpu")
# training accuracy
train_acc = np.sum(np.sign(model(X_train)) == y_train) / len(y_train)
Instead of specifying the formulation to solve, we directly instantiate a ConvexReLU
model by passing it a matrix of gate vectors. The number of gate vectors is analogous to choice of max_neurons
in the optimize <scnn.optimize.optimize>
function --- see Models
for more details. We also the solver to use (AL <scnn.solvers.AL>
), a sparsity-inducing regularizer (NeuronGL1 <scnn.regularizers.NeuronGL1>
) and the metrics to collect during optimization (Metrics <scnn.metrics.Metrics>
).
See the Documentation
for further details on the models, regularizers, and solvers supported by scnn
. Or, get hands-on experience training neural networks with convex optimization using the Examples
.