Skip to content
Functional gradient boosting based on residual network perception
Branch: master
Clone or download
Fetching latest commit…
Cannot retrieve the latest commit at this time.
Permalink
Type Name Latest commit message Commit time
Failed to load latest commit information.
examples snapshot Feb 19, 2019
resfgb new fw method Feb 19, 2019
.gitignore Added ResFGB Feb 19, 2018
LICENSE Create LICENSE Feb 24, 2018
README.md Added a link to the paper Feb 27, 2018

README.md

ResFGB

This is a Theano(>=1.0.0) implementation of "Functional gradient boosting based on residual network perception".

ResFGB is a functional gradient boosting method for learning a resnet-like deep neural network for non-linear classification problems. The model is composed of a linear classifier such as logistic regression and support vector machine, and a feature extraction. In each iteration, these components are trained by alternate optimization, that is, a linear classifier is trained to classify obtained samples through a feature extraction and this extraction map is updated by stacking a resnet-type layer to move samples along the direction of increasing the linear separability. We finally obtain a highly non-linear classifier forming a residual network.

Usage

A simple pseudocode is provided below.

Note: (X,Y): training data, (Xv,Yv): validation data, (Xt,Yt): test data. These are numpy arrays. n_data: the number of training data, input_dim: dimension of the input space, n_class: the number of classes. A label set should be an integer sequence starting with zero.

from resfgb.models import ResFGB, get_hyperparams

hparams = get_hyperparams( n_data, input_dim, n_class )
model = ResFGB( **hparams )
best_iters,_ ,_ = model.fit( X, Y, Xv, Yv, use_best_iter=True )

train_loss, train_acc = model.evaluate( X, Y )
print( 'train_loss: {0}, train_acc: {1}'.format(train_loss, train_acc) )

test_loss, test_acc  = model.evaluate( Xt,  Yt )
print( 'test_loss : {0}, test_acc : {1}'.format(test_loss, test_acc) )

See examples/sample_resfgb.py for more detail.

Hyperparameters

Hyperparameters of ResFGB are mainly divided three types: the first is for learning a linear classifier, the second is for learning a multi-layer network as a resblock, and the other is for the functional gradient method.

The hyperparameters are listed below. 'Default' is a value set by the function resfgb.models.get_hyperparams. input_dim and n_class stand for the dimension of the input space and the number of classes, respectively.

For the linear model

  • shape[default=(input_dim, n_class)]
    • Shape of the linear model, which should not be changed.
  • wr[default=1/n_data]
    • L2-regularization parameter.
  • bias[default=True]
    • Flag for whether to include bias term or not.
  • eta[default=1e-2]
    • Learning rate for Nesterov's momentum method.
  • momentum[default=0.9]
    • Momentum parameter for Nesterov's momentum method.
  • minibatch_size[default=100]
    • Minibatch size to compute stochastic gradients.
  • max_epoch[default=100]
    • The number of epochs for learning a linear model.
  • tune_eta[default=True]
    • Flag for whether to tune learning rate or not.
  • scale[default=1.0]
    • Positive number by which a tuned learning rate is multiplied.
  • eval_iters[default=1000]
    • The number of iterations in a trial for tuning learning rate.
  • early_stop[default=10]
    • When the training loss does not improve while this number of epochs, the training is stopped.

For the resblock

  • shape[default=(input_dim,100,100,100,100,input_dim)]
    • Shape of the multi-layer perceptron. Dimensions of the input and last layer should set to input_dim.
  • wr[default=1/n_data]
    • L2-regularization parameter.
  • eta[default=1e-2]
    • Learning rate for Nesterov's momentum method.
  • momentum[default=0.9]
    • Momentum parameter for Nesterov's momentum method.
  • minibatch_size[default=100]
    • Minibatch size to compute stochastic gradients.
  • max_epoch[default=50]
    • The number of epochs for learning a linear model.
  • tune_eta[default=True]
    • Flag for whether to tune learning rate or not.
  • scale[default=1.0]
    • Positive number by which a tuned learning rate is multiplied.
  • eval_iters[default=1000]
    • The number of iterations in a trial for tuning learning rate.
  • early_stop[default=10]
    • When the training loss does not improve while this number of epochs, the training is stopped.

For the functional gradient method

  • model_type[default='logistic']
    • Type of the linear model: 'logistic' or 'smooth_hinge'.
  • model_hparams[default=model_hparams]
    • Dictionary of the hyperparameter for the linear model.
  • resblock_hparams[default=resblock_hparams]
    • Dictionary of the hyperparameter for the resblock.
  • fg_eta[default=1e-1]
    • Learning rate used in the functional gradient method.
  • max_iters[default=30
    • The number of iterations of the functional gradient method, which corresponds to the depth of an obtained network.
  • seed[default=1]
    • Random seed used in the method.
You can’t perform that action at this time.