see the detailed description of Signum in the original paper at: https://arxiv.org/abs/1802.04434
Arguments:
params (iterable): iterable of parameters to optimize or dicts defining
parameter groups
lr (float): learning rate
momentum (float, optional): momentum factor (default: 0.9)
weight_decay (float, optional): weight decay (default: 0)
Example:
>>> optimizer = signum.Signum(model.parameters(), lr=0.1, momentum=0.9)
>>> optimizer.zero_grad()
>>> loss_fn(model(input), target).backward()
>>> optimizer.step()
Note:
The optimizer updates the weight by:
momentum = beta * momentum + (1-beta)*rescaled_grad
weight = (1 - lr * weight_decay) * weight - lr * sign(momentum)
Considering the specific case of Momentum, the update Signum can be written as
If do not consider Momentum, the update Sigsgd can be written as
Description of example:
Using pre-trained resnet-18 model and supporting the datasets of Caltech101, VOC2012 and CVPR Indoor.
Directory:
train.py: train and validate the dataset
signum.py: contain the signum optimizer
dataset_info.py: load datasets information from datasets/ to train.py
train_info.py: load training information to train.py
datasets/: save the data pre-process for datasets