Permalink
Switch branches/tags
Nothing to show
Find file Copy path
Fetching contributors…
Cannot retrieve contributors at this time
1270 lines (1051 sloc) 80.4 KB

Parameters

This Section explains which parameters to tune for each algorithm. Almost all algorithms have in common the following:

Parameter Explanation
seed Int value to replicate randomized processes
bags(new) Int value to specify number of times to run a model with different seeds
verbose If True it prints stuff regarding the progress of an algorithm
threads Int value to apply parallelism. Not always applicable, but can facilitate speed’s performance
usescale If True it use maximum absolute scaling. It is useful for linear algorithms
copy If True, it makes a hard copy of the data.

Classifiers

Classifier Models are described first.

DecisionTreeClassifier

DecisionTreeClassifier threads:50 max_tree_size:-1 rounding=10 offset:0.0001 feature_subselection:1.0 cut_off_subsample:1.0 max_depth:6 max_features:0.9 min_leaf:5.0 min_split:10 Objective:ENTROPY row_subsample:0.95 seed:1 threads:1 bags:1 verbose:false
Parameter Explanation
max_depth Maximum depth of the tree (double). This is important.
Objective The objective to optimise in split. It may be “ENTROPY “, “GINI” or “AUC”. ENTROPY (default) almost always performs best. This is important.
row_subsample Proportion of observations to consider (double). This is important.
max_features Proportion of columns (features) to consider in each level (double). This is important.
cut_off_subsample Proportion of best cut offs to consider. This controls how Extremely Randomized the tree will be (double).
feature_subselection Proportion of columns (features) to consider for the whole tree (double).
min_leaf Minimum weighted sum to keep after splitting node (double).
min_split Minimum weighted sum to split a node (double).
rounding Digits of rounding to prevent overfitting. It could help in certain situations(double).
max_tree_size Maximum number of nodes allowed (int)
offset Adds a constant when calculating the objective in a split. It prevents overfitting (double).

The rest of the parameters may be unstable and better left as is.

RandomForestClassifier

RandomForestClassifier bootsrap:false max_tree_size:-1 cut_off_subsample:1.0 feature_subselection:1.0 rounding:6 estimators:100 offset:0.00001 max_depth:6 max_features:0.4 min_leaf:2.0 min_split:5.0 Objective:ENTROPY row_subsample:0.95 seed:1 threads:1 bags:1 verbose:false
Parameter Explanation
estimators Number of trees to build. In most situations after 100 it does not improve dramatically more (int) .
max_depth maximum depth of the tree (double). This is important.
Objective The objective to optimise in split. It may be “ENTROPY “, “GINI” or “AUC”. ENTROPY (default) almost always performs best. This is important.
row_subsample Proportion of observations to consider (double). This is important.
max_features Proportion of columns (features) to consider in each level (double). This is important.
cut_off_subsample Proportion of best cut offs to consider. This controls how Extremely Randomized the tree will be (double).
feature_subselection Proportion of columns (features) to consider for the whole tree (double).
min_leaf Minimum weighted sum to keep after splitting node (double).
min_split Minimum weighted sum to split a node (double).
rounding Digits of rounding to prevent overfitting. It could help in certain situations(double).
max_tree_size Maximum number of nodes allowed (int)
offset Adds a constant when calculating the objective in a split. It prevents overfitting (double).

The rest of the parameters may be left as is.

AdaboostRandomForestClassifier

AdaboostRandomForestClassifier bootsrap:false trees:1 offset:0.00001 max_tree_size:-1 cut_off_subsample:1.0 weight_thresold:0.95 estimators:100 max_depth:6 max_features:0.5 min_leaf:2.0 min_split:5.0 Objective:ENTROPY row_subsample:0.9 seed:1 threads:1 bags:1 verbose:false
Parameter Explanation
estimators Number of Random Forests to build. In most situations after 100 it does not improve dramatically more (int) .
trees Number of trees in each Forest. The default is 1 which basically connotes a adatreeclassifier (int).
weight_thresold Affects the weight (importance) of each new estimator via setting this initial threshold. This may be regarded as a shrinkage parameter. Needs to be between 0 and 1 (double). This is important.
max_depth Maximum depth of the tree (double). This is important.
Objective The objective to optimise in split. It may be “ENTROPY “, “GINI” or “AUC”. ENTROPY (default) almost always performs best. This is important.
row_subsample Proportion of observations to consider (double). This is important.
max_features Proportion of columns (features) to consider in each level (double). This is important.
cut_off_subsample Proportion of best cut offs to consider. This controls how Extremely Randomized the tree will be (double).
feature_subselection Proportion of columns (features) to consider for the whole tree (double).
min_leaf Minimum weighted sum to keep after splitting node (double).
min_split Minimum weighted sum to split a node (double).
rounding Digits of rounding to prevent overfitting. It could help in certain situations(double).
max_tree_size Maximum number of nodes allowed (int)
offset Adds a constant when calculating the objective in a split. It prevents overfitting (double).

The rest of the parameters may be left as is.

GradientBoostingForestClassifier

GradientBoostingForestClassifier rounding:6 estimators:1000 shrinkage:0.1 offset:0.00001 max_tree_size:-1 cut_off_subsample:1.0 max_depth:8 max_features:0.4 min_leaf:4.0 min_split:8.0 Objective:RMSE row_subsample:0.7 seed:1 threads:1 bags:1 verbose:false
Parameter Explanation
estimators Number of Random Forests to build. In most situations after 100 it does not improve dramatically more (int) .
trees Number of trees in each Forest. The default is 1 which basically connotes a adatreeclassifier (int).
shrinkage Penalty applied to each estimator . Smaller values prevent overfitting. Needs to be between 0 and 1 (double). There is also a fairly linear negative correlation between estimators and shrinkage. This is important.
max_depth Maximum depth of the tree (double). This is important.
Objective The objective to optimise inside the split. It may be “RMSE“ or “MAE”. Bear in mind the underlying estimators are regressors.
row_subsample Proportion of observations to consider (double). This is important.
max_features Proportion of columns (features) to consider in each level (double). This is important.
cut_off_subsample Proportions of best cut offs to consider. This controls how Extremely Randomized the tree will be. Very low value means only a few cut-offs are explored (double).
feature_subselection Proportions of columns (features) to consider for the whole tree (double).
min_leaf Minimum weighted sum to keep after splitting node (double).
min_split Minimum weighted sum to split a node (double).
rounding Digits of rounding to prevent overfitting. It could help in certain situations (double).
max_tree_size Maximum number of nodes allowed (int) .
offset Adds a constant when calculating the objective in a split. It prevents overfitting (double).

The rest of the parameters may be left as is.

LogisticRegression

LogisticRegression Type:Liblinear C:1.0 l1C:1.0 learn_rate:0.1 shuffle:true RegularizationType:L2 UseConstant:true usescale:True maxim_Iteration:200 seed:1 threads:1 bags:1 verbose:false
Parameter Explanation
C Regularization value, the more, the stronger the regularization(double). This is important.
l1C L1 Regularization C value for FTRL Type (double).
Type Can be one of “Liblinear”, “Routine”, “SGD”, “FTRL”. Default is Liblinear. SGD and FTRL use adagrad. Routine is based on Matrix multiplications and the Newton-Raphson method.
RegularizationType Can be either "L2" or “L1”. Default is “L2”. “L1” is only supported via Liblnear and FTRL. This is important.
learn_rate For SGD and FTRL (double).
UseConstant If true it uses an intercept.
maxim_Iteration Maximum number of iterations (int) .
shuffle True to train on random rows.

LSVC

LSVC Type:Liblinear usescale:True C:1.0 RegularizationType:L2 shuffle:true UseConstant:true l1C:1.0 maxim_Iteration:100 seed:1 threads:1 bags:1 verbose:false
Parameter Explanation
C Regularization value, the more, the stronger the regularization(double). This is important.
l1C L1 Regularization C value for FTRL Type (double).
Type Can be one of “Liblinear”, “SGD”, “FTRL”. Default is Liblinear. SGD and FTRL use adagrad.
RegularizationType Can be either "L2" or “L1”. Default is “L2”. “L1” is only supported via Liblnear and FTRL. This is important.
learn_rate For SGD and FTRL (double).
UseConstant If true it uses an intercept.
maxim_Iteration Maximum number of iterations (int) .
shuffle True to train on random rows.

LibFmClassifier

LibFmClassifier maxim_Iteration:50 C:0.001 C2:0.001 shuffle:true lfeatures:2 UseConstant:true usescale:True init_values:0.1 learn_rate:0.1 smooth:0.01 seed:1 threads:1 bags:1 verbose:false

Based on Steffen Rendle’s [libfm] (http://www.libfm.org/)

Parameter Explanation
C Regularization value, the more, the stronger the regularization (double). This is important.
C2 Regularization value for the latent features (double). This is important.
Lfeatures Number of latent features to use. Defaults to 4 (int). This is important.
init_values Initialise values of the latent features with random values between [0,init_values) (double). This is important.
learn_rate For SGD (double). This is important.
maxim_Iteration Maximum number of iterations (int) . This is important.
Type Only “SGD”.
UseConstant If true it uses an intercept.
shuffle True to train on random rows.

Softmaxnnclassifier

softmaxnnclassifier usescale:True maxim_Iteration:50 UseConstant:true C:0.000001 shuffle:true tolerance:0.01 learn_rate:0.01 smooth:0.1 h1:20 h2:20 connection_nonlinearity:Relu init_values:0.02 seed:1 threads:1 bags:1 verbose:false

This is a neural network with 2 hidden layers. It is heavily based on the equivalent one in the kaggler python package.

Parameter Explanation
C Regularization value, the more, the stronger the regularization (double). This is important.
h1 Number of the 1st level hidden units (int). This is important.
h2 Number of the 2nd level hidden units (int). This is important.
init_values Initialise values of hidden units with random values between [0,init_values) (double). This is important.
smooth Value to divide gradients and aid convergence (double). This is important.
connection_nonlinearity Can be one of “Relu”,”Linear”,”Sigmoid”,”Tanh”. Commonly Relu performs best. This is important.
learn_rate For SGD (double). This is important.
maxim_Iteration Maximum number of iterations (int) . This is important.
Type Only “SGD”.
UseConstant If true it uses an intercept.
shuffle True to train on random rows.

NaiveBayesClassifier

NaiveBayesClassifier usescale:True Shrinkage:0.1 seed:1 threads:1 verbose:false
Parameter Explanation
Shrinkage Can be seen as a form of a penalty to avoid really big product’s failures.

XgboostClassifier

The original parameters can be found here

XgboostClassifier booster:gbtree num_round:1000 eta:0.005 max_leaves:0 gamma:1. max_depth:5 min_child_weight:1.0 subsample:0.9 colsample_bytree:0.7 colsample_bylevel:1.0 lambda:1.0 alpha:1.0 seed:1 threads:1 bags:1 verbose:false
Parameter Explanation
scale_pos_weight used for imbalanced classes(double)
num_round Number of estimators to build (int) . This is important.
max_leaves Maximum leaves in a tree (int).
eta Penalty applied to each estimator. Needs to be between 0 and 1 (double). This is important.
max_depth Maximum depth of the tree (int). This is important.
subsample Proportion of observations to consider (double). This is important.
colsample_bylevel Proportion of columns (features) to consider in each level (double).
colsample_bytree Proportion of columns (features) to consider in each Tree (double) This is important.
max_delta_step controls optimization step (double).
gamma controls minimum change requirements in loss to allow for a split (double).
booster 'gbtree' or 'gblinear'.
alpha controls overfitting (double).
lambda controls overfitting (double).

LightgbmClassifier

The original parameters can be found here

LightgbmClassifier boosting:gbdt num_leaves:14 num_iterations:100 scale_pos_weight:1.0 skip_drop:0.5 uniform_drop:false xgboost_dart_mode:false two_round:false top_rate:0.1 sigmoid:1.0 is_unbalance:false max_bin:255 poission_max_delta_step:0.7 min_sum_hessian_in_leaf:0.0001 other_rate:0.1 min_data_in_bin:5 max_drop:50 drop_rate:0.1 categorical_feature:0,1,2 learning_rate:0.1 threads:1 max_depth:5 feature_fraction:0.5 min_data_in_leaf:10 min_gain_to_split:20 bagging_fraction:0.9 lambda_l1:0.1 lambda_l2:0.1 bagging_freq:1 bin_construct_sample_cnt:100000 seed:1 threads:1 bags:1 verbose:false
Parameter Explanation
learning_rate weight of each estimator. This is important
bagging_fraction Proportions of rows consider. This is important
num_iterations Number of trees to build. This is important
max_depth maximum depth of the tree. This is important
feature_fraction Proportions of columns (features) to consider within a tree. This is important
bagging_freq Every how many iters it will perform bagging.
bin_construct_sample_cnt Sample number of rows to create histograms.
boosting Type of boosting. Could be 'gbdt','dart' or 'goss' .
categorical_feature comma separated features to be treated as categorical
drop_rate dropout rate in dart boosting
is_unbalance true to oversample weak classes in binary classification
lambda_l1 L1 regularization
lambda_l2 L2 regularization
max_bin max number of bin that feature values will bucket in.
max_drop max number of dropped trees on one iteration (in dart).
min_data_in_bin min number of data inside one bin, use this to avoid one-data-one-bin (may prevent over-fitting).
min_data_in_leaf Minimum number of data in a leaf.
min_gain_to_split Minimum gain to split a node
min_sum_hessian_in_leaf Minimum sum hessian in one leaf
num_leaves maximum number of leaves.
other_rate only used in boosting goss, the retain ratio of small gradient data.
poission_max_delta_step safeguard optimisation.
scale_pos_weight scale weight for binary class.
sigmoid parameter for sigmoid function.
skip_drop probability of skipping drop (in dart).
top_rate used in boosting goss, the retain ratio of large gradient data.
two_round if true it saves memory but takes more time.
uniform_drop Specify whether to use uniform dropout.
boolean xgboost_dart_mode true use xgboost dart mode or not.

SklearnAdaBoostClassifier

The original parameters can be found here

SklearnAdaBoostClassifier algorithm:SAMME.R learning_rate:0.7 n_estimators:100 threads:1 usedense:false seed:1 verbose:false
Parameter Explanation
learning_rate Learning rate shrinks the contribution of each classifier by learning_rate. This is important
n_estimators Number of trees to build. This is important
algorithm Could be SAMME or SAMME.R
use_dense True to Use dense data.

SklearnDecisionTreeClassifier

The original parameters can be found here

SklearnDecisionTreeClassifier criterion:entropy max_leaf_nodes:0 min_impurity_split:0.0000001 threads:1 max_features:0.5 max_depth:5 min_samples_leaf:1 min_samples_split:2 use_dense:false min_weight_fraction_leaf:0.0 new_tree_gain_ratio:1.0 bootsrap:false seed:1 verbose:false
Parameter Explanation
max_depth maximum depth of the tree. This is important
max_features Proportions of columns (features) to consider. This is important
max_leaf_nodes maximum number of nodes allowed.
use_dense True to Use dense data. If your data is in dense format, do select true as all files get loaded as sparse by default in python-based modules.
min_impurity_split Threshold for early stopping in tree growth.
criterion Criterion to determine the split could gini or entropy
min_samples_leaf Minimum cases to keep a splitted node
min_samples_split Minimum cases to split a node
min_weight_fraction_leaf The minimum weighted fraction of the sum total of weights (of all the input samples) required to be at a leaf node.

SklearnExtraTreesClassifier

The original parameters can be found here

SklearnExtraTreesClassifier criterion:entropy max_leaf_nodes:0 n_estimators:100 min_impurity_split:0.0000001 threads:1 max_features:0.5 max_depth:5 min_samples_leaf:1 min_samples_split:2 use_dense:false min_weight_fraction_leaf:0.0 new_tree_gain_ratio:1.0 bootsrap:false seed:1 verbose:false 		  
Parameter Explanation
n_estimators Toral number of trees to build. This is important
max_depth maximum depth of the tree. This is important
max_features Proportions of columns (features) to consider. This is important
use_dense True to Use dense data. If your data is in dense format, do select true as all files get loaded as sparse by default in python-based modules.
max_leaf_nodes maximum number of nodes allowed.
min_impurity_split Threshold for early stopping in tree growth.
bootsrap true use bootsrap or not.
criterion Criterion to determine the split could gini or entropy
min_samples_leaf Minimum cases to keep a splitted node
min_samples_split Minimum cases to split a node
min_weight_fraction_leaf The minimum weighted fraction of the sum total of weights (of all the input samples) required to be at a leaf node.

SklearnRandomForestClassifier

The original parameters can be found here

SklearnRandomForestClassifier criterion:entropy max_leaf_nodes:0 n_estimators:100 min_impurity_split:0.0000001 threads:1 max_features:0.5 max_depth:5 min_samples_leaf:1 min_samples_split:2 use_dense:false min_weight_fraction_leaf:0.0 new_tree_gain_ratio:1.0 bootsrap:false seed:1 verbose:false 		  
Parameter Explanation
n_estimators Toral number of trees to build. This is important
max_depth maximum depth of the tree. This is important
max_features Proportions of columns (features) to consider. This is important
use_dense True to Use dense data. If your data is in dense format, do select true as all files get loaded as sparse by default in python-based modules.
max_leaf_nodes maximum number of nodes allowed.
min_impurity_split Threshold for early stopping in tree growth.
bootsrap true use bootsrap or not.
criterion Criterion to determine the split could gini or entropy
min_samples_leaf Minimum cases to keep a splitted node
min_samples_split Minimum cases to split a node
min_weight_fraction_leaf The minimum weighted fraction of the sum total of weights (of all the input samples) required to be at a leaf node.

SklearnMLPClassifier

The original parameters can be found here

SklearnMLPClassifier standardize:true use_log1p:true shuffle:true learning_rate:adaptive momentum:0.9 optimizer:sgd use_dense:false alpha:0.000001 hidden:50,50 activation:relu epsilon:0.00000001 max_iter:50 learning_rate_init:0.01 batch_size:8 tol:0.0001 validation_split:0.2 seed:1 verbose:false
Parameter Explanation
hidden Toral Comma-separated integers defining the number of hidden layers along with teh hidden units. This is important
epochs Maximum number of iterations. This is important
activation Activation function for the hidden layer coudl be identity, logistic, tanh, relu. This is important
alpha L2 regularization on the weights. This is important
learning_rate_init The (initial) learning rate used. This is important
learning_rate Could be adaptive ,constant or invscaling.
batch_size Number of cases(samples) in a batch.
optimizer could adam, bfgs or sgd.
tol Tolerance to determine the end of the optimization.
epsilon Value for numerical stability in adam.
momentum Only applicable for optimizer=sgd. Nesterov's is on by default.
shuffle true Enable shuffling of training data (on each epoc).
standardize true to standardize dense data (e.g use_dense=true) or to use absmaxscaling on sparse data (use_dense=false).
use_log1p converts the data matrix to log plus 1.
validation_split Split percentage to use for early stopping. Best epoc is determived via cv after 2 consecutive worse loss estimateds.
use_dense True to Use dense data. If your data is in dense format, do select true as all files get loaded as sparse by default in python-based modules.

SklearnSGDClassifier

The original parameters can be found here

SklearnSGDClassifier standardize:true use_log1p:true shuffle:true learning_rate:adaptive momentum:0.9 optimizer:sgd use_dense:false alpha:0.000001 hidden:50,50 activation:relu epsilon:0.00000001 max_iter:50 learning_rate_init:0.01 batch_size:8 tol:0.0001 validation_split:0.2 seed:1 verbose:false
Parameter Explanation
n_iter Maximum number of iterations. This is important
alpha Regularization on the weights. This is important
eta0 The (initial) learning rate used. This is important
learning_rate Could be optimal, constant or invscaling.
loss could be log or modified_huber.
epsilon For huber, determines the threshold at which it becomes less important to get the prediction exactly right.
l1_ratio The Elastic Net mixing parameter, with 0 <= l1_ratio <= 1. l1_ratio=0 corresponds to L2 penalty, l1_ratio=1 to L1.
penalty The penalty (aka regularization term) to be used. could be l2, l1, or elasticnet .
power_t The exponent for inverse scaling learning rate [default 0.5].
shuffle true Enable shuffling of training data (on each iteration).
standardize true to standardize dense data (e.g use_dense=true) or to use absmaxscaling on sparse data (use_dense=false).
use_log1p converts the data matrix to log plus 1.
use_dense True to Use dense data. If your data is in dense format, do select true as all files get loaded as sparse by default in python-based modules.

SklearnknnClassifier

The original parameters can be found here

SklearnknnClassifier seed:1 usedense:true use_scale:false distance:cityblock metric:uniform n_neighbors:3 thread:1 verbose:false
Parameter Explanation
n_neighbors Number of neighbors to use by default for k_neighbors queries. This is important
distance It must be one of euclidean, cosine, manhattan or cityblock
metric Weight function used in prediction. Possible values: uniform or distance.
use_scale true to use absmaxscaling.
use_dense True to Use dense data. If your data is in dense format, do select true as all files get loaded as sparse by default in python-based modules.

SklearnsvmClassifier

The original parameters can be found here

SklearnsvmClassifier seed:1 usedense:false use_scale:false max_iter:-1 kernel:rbf degree:3 C:1.0 tol:0.0001 coef0:0.0 gamma:0.0 verbose:False
Parameter Explanation
max_iter Maximum number of iterations. This is important
kernel Kernel type could be linear, poly, rbf or sigmoid. This is important
C The Penalty parameter C of the error term. This is important
tol Tolerance to determine the end of the optimization.
degree Degree of the polynomial kernel function (poly).
gamma Kernel coefficient for rbf, poly and sigmoid.
coef Independent term in kernel function.It is only significant in poly and sigmoid.
use_scale true to use absmaxscaling.
use_dense True to Use dense data. If your data is in dense format, do select true as all files get loaded as sparse by default in python-based modules.

KerasnnClassifier

The original parameters can be found sparsely in keras' documentation

KerasnnClassifier loss:categorical_crossentropy standardize:true use_log1p:true shuffle:true batch_normalization:true weight_init:lecun_uniform momentum:0.9 optimizer:adam use_dense:true l2:0.000001,0.000001 hidden:50,50 activation:relu,relu droupouts:0.1,0.1 epochs:20 lr:0.01 batch_size:8 stopping_rounds:10 validation_split:0.2 seed:1 verbose:false 
Parameter Explanation
hidden Toral Comma-separated integers defining the number of hidden layers along with teh hidden units. This is important
droupouts Toral Comma-separated floats defining the dropout in each layer (defined by hidden). This is important
l2 Toral Comma-separated floats defining the l2 regularization term on the weights in each layer (defined by hidden). This is important
activation Toral Comma-separated strings defining the activation in each hidden layer. This is important
lr The learning rate used. This is important
epochs Maximum number of iterations. This is important
batch_normalization true to add a batch normlization to the layers. This is important
batch_size Number of cases(samples) in a batch. This is important
weight_init The distribution from which initial weights are to be drawn. Has to be RandomNormal, RandomUniform, TruncatedNormal, VarianceScaling, Orthogonal, Identity, lecun_uniform, glorot_normal,glorot_uniform, he_normal, lecun_normal, he_uniform, he_normal.
optimizer Has to be adam, adagrad, nadam, adadelta or sgd.
loss Has to be categorical_crossentropy, categorical_hinge, logcosh, Kullback–Leibler divergence.
momentum Only applicable for optimizer=sgd. Nesterov's is on by default.
shuffle true Enable shuffling of training data (on each epoc).
standardize true to standardize dense data (e.g use_dense=true) or to use absmaxscaling on sparse data (use_dense=false).
use_log1p converts the data matrix to log plus 1.
validation_split Split percentage to use for early stopping. Best epoc is determived via cv after 2 consecutive worse loss estimateds.
stopping_rounds Split percentage to use for early stopping. Best epoc is determived via cv after 2 consecutive worse loss estimateds.
use_dense True to Use dense data. If your data is in dense format, do select true as all files get loaded as sparse by default in python-based modules.

PythonGenericClassifier

The user can run his/her own python script as long as it is placed in lib/python/ and named after PythonGenericClassifier[INDEX]. Index will be a hyper parameter. Look for PythonGenericClassifier0.py in lib/python/ for an example.

PythonGenericClassifier index:0 seed:1 verbose:False 
Parameter Explanation
index this is the index specifying which PythonGenericClassifier[index].py script to run. This is important

FRGFClassifier

(Some of) the original parameters of fast_rgf can be found here

FRGFClassifier dtree_loss:LOGISTIC max_nodes:16 ntrees:100 eta:0.1 threads:1 sparse_max_features:1000 max_level:4 dense_max_buckets:250 min_sample:1 min_occurrences:1 opt:rgf sparse_lamL2:1.0 min_bucket_weights:250 new_tree_gain_ratio:1.0 stepsize:0.1 lamL2:1.0 lamL1:1.0 sparse_max_buckets:250 verbose:False
Parameter Explanation
ntrees Toral number of trees to build. This is important
max_level maximum depth of the tree. This is important
lamL2 L2 regularization on the weights. This is important
new_tree_gain_ratio new tree is created when leaf-nodes gain < this value * estimated gain of creating new three. This is important
lamL1 L1 regularization on the weights.
stepsize Step size of epsilon-greedy boosting (inactive for rgf).
min_occurrences minimum number of occurrences for a feature to be selected.
min_sample minimum samples in node.
max_nodes maximum number of nodes.
loss Type of loss. could be LS, MODLS (modified least squares loss), or LOGISTIC.
opt optimization method for training forest. Could be rgf or epsilon-greedy.
sparse_lamL2 L2 regularization parameter for sparse data.
min_bucket_weights Minimum sum of data weights for each discretized value.
dense_max_buckets Maximum bins for dense data.
sparse_max_features You may try a different value in [1000,10000000] for fetaures allowed.
dense_max_buckets Maximum bins for dense data.

H2OGbmClassifier

H2OGbmClassifier ntrees:100 learn_rate:0.01 nbins:255 balance_classes:false max_depth:4 col_sample_rate_per_tree:0.5 col_sample_rate:1.0 sample_rate:0.9 min_rows:1 seed:1 threads:1 bags:1 verbose:false
Parameter Explanation
col_sample_rate Proportions of columns (features) to consider at each level of a given tree. This is important
learn_rate weight on each estimator. This is important
max_depth maximum depth of the tree. This is important
ntrees Number of trees to build This is important
sample_rate Proportions of rows consider This is important
col_sample_rate_per_tree Proportions of columns (features) to consider within a tree.
balance_classes whether to oversample the minority classes to balance the class distribution.
min_rows minimum number of cases in a node.
nbins The number of bins for the histogram to build.

H2ODeepLearningClassifier

H2ODeepLearningClassifier activation:Rectifier input_dropout_ratio:0.1 shuffle:true tandardize:false weight_init:UniformAdaptive sample_rate:1.0 l1:0 l2:0.00001 max_w2:1.0 mini_batch_size:1 fast_mode:false adaptive_rate:true rho:0.9 epsilon:1e-8 balance_classes:false epochs:10 dropouts:0.5,0.5 hidden:100,50 col_sample_rate:1.0 sample_rate:0.9 min_rows:1 seed:1 threads:1 bags:1 verbose:false
Parameter Explanation
activation activation functions. Has to be between 'Rectifier', 'Tanh', 'ExpRectifier' or 'Maxout'
adaptive_rate true to use The implemented adaptive learning rate algorithm (ADADELTA) which automatically combines the benefits of learning rate annealing and momentum training to avoid slow convergence.
rho The first of two hyper parameters for ADADELTA. It is like momentum. This is important
epsilon The second of two hyper parameters for ADADELTA. This is important
balance_classes Specify whether to oversample the minority classes to balance the class distribution.
dropouts dropout ratios for each hidden layer,comma separated .Has to match in length the 'hidden' parameter. This is important
epochs Number of iterations to train the DL model. This is important
fast_mode True for faster convergence (but potential loss in accuracy)
hidden Number of hidden neurons, comma separated.The length connotes the number of hidden layers too. This is important
input_dropout_ratio dropout from to the input layer
l1 regularization on the weights.
l2 regularization on the weights. This is important
max_w2 A maximum on the sum of the squared incoming weights into any one neuron.
mini_batch_size minimum number of cases in batch.
momentum_ramp The momentum_ramp parameter controls the amount of learning for which momentum increases (assuming momentum_stable is larger than momentum_start).
momentum_stable The momentum_stable parameter controls the final momentum value reached after momentum_ramp training samples.
momentum_start The momentum_start parameter controls the amount of momentum at the beginning of training.
nesterov_accelerated_gradient True to enable Nesterov accelerated gradient descent method.
rate When adaptive learning rate is disabled, the magnitude of the weight updates are determined by the user specified learning rate (potentially annealed), and are a function of the difference between the predicted value and the target value.
rate_annealing Learning rate annealing reduces the learning rate to “freeze” into local minima in the optimization landscape.
rate_decay The learning rate decay parameter controls the change of learning rate across layers.
sample_rate Proportions of rows consider in each epoc.
shuffle true to enable shuffling of training data (on each node).
tandardize true to standardize the input data.
weight_init The distribution from which initial weights are to be drawn. Has to be 'UniformAdaptive', 'Uniform' or 'Normal'

H2ODrfClassifier

H2ODrfClassifier ntrees:100 nbins:255 balance_classes:false max_depth:4 col_sample_rate_per_tree:0.5 sample_rate:0.9 min_rows:1 seed:1 threads:1 bags:1 verbose:false
Parameter Explanation
max_depth maximum depth of the tree. This is important
ntrees Number of trees to build. This is important
sample_rate Proportions of rows consider This is important
col_sample_rate_per_tree Proportions of columns (features) to consider within a tree.
balance_classes whether to oversample the minority classes to balance the class distribution.
min_rows minimum number of cases in a node.
nbins The number of bins for the histogram to build.

H2OGlmClassifier

H2OGlmClassifier alpha:0 lambda:0.00001 balance_classes:false standardize:false max_iterations:50 beta_epsilon:0.00001 bjective_epsilon:0.00001 seed:1 threads:1 bags:1 verbose:false
Parameter Explanation
alpha Proportion of l1/l2. 0 = Ridge, 1=Lasso
lambda Regularization parameter. This is important
max_iterations Number of iterations to build the model. This is important
beta_epsilon tolerance of the coefficients
bjective_epsilon tolerance of the objective function
balance_classes true to Specify whether to oversample the minority classes to balance the class distribution.
standardize true to standardize input features or not

OriginalLibFMClassifier

Wraps the original implementation of ibFM, made from Steffen Rendle. The reason this implementation is made, is because internal results show that it has better performance (as in accuracy) than StackNet's internal implementation and has more training methods than just sgd. This implementation may not include all libFM features plus it actually uses a version of it that had a bug(!) on purpose. You can find more information about why this was chosen in the following python wrapper for libFM. It basically had this bug that was allowing you to get the parameters of the trained models for all training methods. These parameters are now extracted once a model has been trained and the scoring uses only these parameters (e.g. not the libFM executable). Don't forget to acknowledge libFM if you publish results produced with this software. Also take note of its licence GNU. More information can be found on libFM's repo on github.

OriginalLibFMClassifier type:als lfeatures:4 learn_rate:0.01 maxim_Iteration:10 init_values:0.01 c:1 c2:1 threads:1 usescale:false seed:1 verbose:true bags:1 
Parameter Explanation
Type Type of algorithm to use. It has to be sgd, als, mcmc. Default is mcmc.
C Regularization value, the more, the stronger the regularization. This is important
C2 Regularization value for the latent features. This is important
lfeatures Number of latent features to use. This is important
init_values Initialise values of the latent features with values between[0,init_values). This is important
maxim_Iteration aximum number of iterations. This is important
learn_rate learn_rate for SGD; default=0.1. This is important

VowpaLWabbitClassifier

Wrapper for vowpal wabbit. It does not contain all features, but a fraction.

VowpaLWabbitClassifier use_ftrl:True learning_rate:0.8 decay_learning_rate:0.97 nn:40 bit_precision:18 ftrl_alpha:0.1 ftrl_beta:0.1 power_t:0.9 initial_t:0.9 l1:0.01 l2:0.01 passes:10 threads:1 make2way:false make3way:false use_dropout:true use_meanfield:false seed:1 verbose:true bags:1  
Parameter Explanation
passes Number of training Passes. This is important
bit_precision number of bits in the feature table.
decay_learning_rate Decay factor for learning_rate between passes.
nn Number of hidden units to use in a sigmoidal feedforward network with nn hidden units
initial_t Initial t value. Affects learning rate's updates
power_t t power value. Affects learning rate's updates
ftrl_alpha ftrl alpha parameter when using ftrl This is important
ftrl_beta ftrl beta stability patameter when using ftrl This is important
learning_rate learning Rate for gradient-based updates
l1 L1 regularization
l2 L2 regularization This is important
use_ftrl true to use the ftrl optimization option (instead of adaptive). It is on by default.
make2way if true it creates all possible 2-way interactions of all features
make3way if true it creates all possible 3-way interactions of all features
use_dropout when nn>0, train or test sigmoidal feedforward network using dropout.
use_meanfield when nn>0, train or test sigmoidal feedforward network using mean field.

libffmClassifier

Wraps Libffm. Note this method either requires the user to manually add comma separated indices that form a field or they need to use some self-made heuristics. This is controlled by parameter opt.

libffmClassifier factor:6 iteration:16 learn_rate:0.1 opt:order lambda:0.0001 threads:1 use_norm:false seed:1 verbose:true bags:1  
Parameter Explanation
factor number of latent factors. This is important
iteration number of iterations. This is important
learn_rate learning rate. This is important
lambda regularization parameter. This is important
use_norm true to allow instance-wise normalization. This is important
opt method for determining the factors. The best way (but not the default) is to provide a list with comma separated indices. Consider this String '1,4,7,123,546'. This would mean that the 0 column is a field on its own, {1,2,3} form another field, {4,5,6} another. {7,8...122} form another field and so on. Another possible value is 'no_order' (default). This looks at the proportion of zeros in neighbouring columns to determine if they form a field. The last possible value is 'order'. This calculates frequencies of non-zero values for all columns and then orders them based on frequency. Columns that have a few missing values form their own fields. Weaker columns (frequency-wise) are joined together to form fields.

Regressors

DecisionTreeRegressor

DecisionTreeRegressor threads:50 max_tree_size:-1 rounding=10 offset:0.0001 feature_subselection:1.0 cut_off_subsample:1.0 max_depth:6 max_features:0.9 min_leaf:5.0 min_split:10 Objective:RMSE row_subsample:0.95 seed:1 threads:1 bags:1 verbose:false
Parameter Explanation
max_depth Maximum depth of the tree (double). This is important.
Objective The objective to optimise in split. It may be “RMSE “ or “MAE”.
row_subsample Proportion of observations to consider (double). This is important.
max_features Proportion of columns (features) to consider in each level (double). This is important.
cut_off_subsample Proportion of best cut offs to consider. This controls how Extremely Randomized the tree will be (double).
feature_subselection Proportion of columns (features) to consider for the whole tree (double).
min_leaf Minimum weighted sum to keep after splitting node (double).
min_split Minimum weighted sum to split a node (double).
rounding Digits of rounding to prevent overfitting. It could help in certain situations(double).
max_tree_size Maximum number of nodes allowed (int)
offset Adds a constant when calculating the objective in a split. It prevents overfitting (double).

The rest of the parameters may be unstable and better left as is.

RandomForestRegressor

RandomForestRegressor bootsrap:false max_tree_size:-1 cut_off_subsample:1.0 feature_subselection:1.0 rounding:6 estimators:100 offset:0.00001 max_depth:6 max_features:0.4 min_leaf:2.0 min_split:5.0 Objective:RMSE row_subsample:0.95 seed:1 threads:1 bags:1 verbose:false
Parameter Explanation
estimators Number of trees to build. In most situations after 100 it does not improve dramatically more (int) .
max_depth Maximum depth of the tree (double). This is important.
Objective The objective to optimise in split. It may be “RMSE “ or “MAE”.
row_subsample Proportion of observations to consider (double). This is important.
max_features Proportion of columns (features) to consider in each level (double). This is important.
cut_off_subsample Proportion of best cut offs to consider. This controls how Extremely Randomized the tree will be (double).
feature_subselection Proportion of columns (features) to consider for the whole tree (double).
min_leaf Minimum weighted sum to keep after splitting node (double).
min_split Minimum weighted sum to split a node (double).
rounding Digits of rounding to prevent overfitting. It could help in certain situations(double).
max_tree_size Maximum number of nodes allowed (int)
offset Adds a constant when calculating the objective in a split. It prevents overfitting (double).

The rest of the parameters may be left as is.

AdaboostRandomForestRegressor

AdaboostRandomForestRegressor bootsrap:false trees:1 offset:0.00001 max_tree_size:-1 cut_off_subsample:1.0 weight_thresold:0.95 estimators:100 max_depth:6 max_features:0.5 min_leaf:2.0 min_split:5.0 Objective:RMSE row_subsample:0.9 seed:1 threads:1 bags:1 verbose:false
Parameter Explanation
estimators Number of Random Forests to build. In most situations after 100 it does not improve dramatically more (int) .
trees Number of trees in each Forest. The default is 1 which basically connotes a adatreeregressor (int).
weight_thresold Affects the weight (importance) of each new estimator via setting this initial threshold. This may be regarded as a shrinkage parameter. Needs to be positive (double). This is important.
max_depth Maximum depth of the tree (double). This is important.
Objective The objective to optimise in split. It may be “RMSE “ or “MAE”.
row_subsample Proportion of observations to consider (double). This is important.
max_features Proportion of columns (features) to consider in each level (double). This is important.
cut_off_subsample Proportion of best cut offs to consider. This controls how Extremely Randomized the tree will be (double).
feature_subselection Proportion of columns (features) to consider for the whole tree (double).
min_leaf Minimum weighted sum to keep after splitting node (double).
min_split Minimum weighted sum to split a node (double).
rounding Digits of rounding to prevent overfitting. It could help in certain situations(double).
max_tree_size Maximum number of nodes allowed (int)
offset Adds a constant when calculating the objective in a split. It prevents overfitting (double).

The rest of the parameters may be left as is.

GradientBoostingForestRegressor

GradientBoostingForestRegressor rounding:6 estimators:1000 shrinkage:0.1 offset:0.00001 max_tree_size:-1 cut_off_subsample:1.0 max_depth:8 max_features:0.4 min_leaf:4.0 min_split:8.0 Objective:RMSE row_subsample:0.7 seed:1 threads:1 bags:1 verbose:false
Parameter Explanation
estimators Number of Random Forests to build. In most situations after 100 it does not improve dramatically more (int) .
trees Number of trees in each Forest. The default is 1 which basically connotes a adatreeclassifier (int).
shrinkage Penalty applied to each estimator . Smaller values prevent overfitting. Needs to be between 0 and 1 (double). There is also a fairly linear negative correlation between estimators and shrinkage. This is important.
max_depth Maximum depth of the tree (double). This is important.
Objective The objective to optimise inside the split. It may be “RMSE“ or “MAE”.
row_subsample Proportion of observations to consider (double). This is important.
max_features Proportion of columns (features) to consider in each level (double). This is important.
cut_off_subsample Proportions of best cut offs to consider. This controls how Extremely Randomized the tree will be. Very low value means only a few cut-offs are explored (double).
feature_subselection Proportions of columns (features) to consider for the whole tree (double).
min_leaf Minimum weighted sum to keep after splitting node (double).
min_split Minimum weighted sum to split a node (double).
rounding Digits of rounding to prevent overfitting. It could help in certain situations (double).
max_tree_size Maximum number of nodes allowed (int) .
offset Adds a constant when calculating the objective in a split. It prevents overfitting (double).

The rest of the parameters may be left as is.

LinearRegression

LinearRegression Type:Routine C:1.0 l1C:1.0 learn_rate:0.1 Objective:RMSE tau:0.5 shuffle:true RegularizationType:L2 UseConstant:true usescale:True maxim_Iteration:200 seed:1 threads:1 bags:1 verbose:false
Parameter Explanation
C Regularization value, the more, the stronger the regularization(double). A value here basically triggers a Ridge regression. This is important.
l1C L1 Regularization C value for FTRL Type (double).
Type Can be one of “Routine”, “SGD” or “FTRL”. SGD and FTRL use adagrad. Routine is the Ordinary Least Squares method which is solved with matrix multiplications.
Objective Can be one of “RMSE”, “MAE” or ”QUANTILE”.
tau Tau value for QUANTILE (double).
learn_rate For SGD and FTRL (double).
UseConstant If true it uses an intercept.
maxim_Iteration Maximum number of iterations (int) .
shuffle True to train on random rows.

LSVR

LSVR Type:Liblinear usescale:True C:1.0 learn_rate:0.1 smooth:0.1 RegularizationType:L2 Objective:L2 shuffle:true UseConstant:true l1C:1.0 maxim_Iteration:100 seed:1 threads:1 bags:1 verbose:false
Parameter Explanation
C Regularization value, the more, the stronger the regularization(double). This is important.
l1C L1 Regularization C value for FTRL Type (double).
Type Can be one of “Liblinear”, “SGD”, “FTRL”. Default is Liblinear. SGD and FTRL use adagrad.
Objective Can be either “L1” or “L2” for normal hinge loss and quadratic loss respectively.
learn_rate For SGD and FTRL (double).
smooth value to aid convergence .
UseConstant If true it uses an intercept.
maxim_Iteration Maximum number of iterations (int) .
shuffle True to train on random rows.

LibFmRegressor

Based on Steffen Rendle’s [libfm] (http://www.libfm.org/)

LibFmRegressor maxim_Iteration:50 C:0.001 Objective:“RMSE” tau:0.5 C2:0.001 shuffle:true lfeatures:2 UseConstant:true usescale:True init_values:0.1 learn_rate:0.1 smooth:0.01 seed:1 threads:1 bags:1 verbose:false
Parameter Explanation
C Regularization value, the more, the stronger the regularization (double). This is important.
C2 Regularization value for the latent features (double). This is important.
Lfeatures Number of latent features to use. Defaults to 4 (int). This is important.
init_values Initialise values of the latent features with random values between [0,init_values) (double). This is important.
learn_rate For SGD (double). This is important.
maxim_Iteration Maximum number of iterations (int) . This is important.
Objective Can be one of “RMSE”, “MAE” or ”QUANTILE”.
tau Tau value for QUANTILE (double).
Type Only “SGD”.
UseConstant If true it uses an intercept.
shuffle True to train on random rows.

Multinnregressor

This is a neural network with 2 hidden layers. It is heavily based on the equivalent one in the kaggler python package.

multinnregressor usescale:True maxim_Iteration:50 Objective:RMSE tau:0.5 UseConstant:true C:0.000001 shuffle:true tolerance:0.01 learn_rate:0.01 smooth:0.1 h1:20 h2:20 connection_nonlinearity:Relu init_values:0.02 seed:1 threads:1 bags:1 verbose:false
Parameter Explanation
C Regularization value, the more, the stronger the regularization (double). This is important.
h1 Number of the 1st level hidden units (int). This is important.
h2 Number of the 2nd level hidden units (int). This is important.
init_values Initialise values of hidden units with random values between [0,init_values) (double). This is important.
smooth Value to divide gradients and aid convergence (double). This is important.
connection_nonlinearity Can be one of “Relu”,”Linear”,”Sigmoid”,”Tanh”. Commonly Relu performs best. This is important.
learn_rate For SGD (double). This is important.
maxim_Iteration Maximum number of iterations (int). This is important.
Objective Can be one of “RMSE”, “MAE” or ”QUANTILE”.
tau Tau value for QUANTILE (double).
UseConstant If true it uses an intercept.
shuffle True to train on random rows.

XgboostRegressor

The original parameters can be found here

XgboostRegressor booster:gbtree num_round:1000 eta:0.005 max_leaves:0 gamma:1. max_depth:5 min_child_weight:1.0 subsample:0.9 colsample_bytree:0.7 colsample_bylevel:1.0 lambda:1.0 alpha:1.0 seed:1 threads:1 bags:1 verbose:false
Parameter Explanation
num_round Number of estimators to build (int) .
max_leaves Maximum leaves in a tree (int).
eta Penalty applied to each estimator. Needs to be between 0 and 1 (double). This is important.
max_depth Maximum depth of the tree (int). This is important.
Objective Can be one of ['reg:linear','count:poisson','reg:gamma' ,'rank:pairwise','reg:tweedie']. Note that rank:pairwise is not a regressor but its output was more convenient for a regerssion method.
subsample Proportion of observations to consider (double). This is important.
colsample_bylevel Proportion of columns (features) to consider in each level (double).
colsample_bytree Proportion of columns (features) to consider in each Tree (double) This is important.
max_delta_step controls optimization step (double).
gamma controls minimum change requirements in loss to allow for a split (double).
booster 'gbtree' or 'gblinear'.
alpha controls overfitting (double).
lambda controls overfitting (double).

LightgbmRegressor

The original parameters can be found here

LightgbmRegressor boosting:gbdt objective:regression huber_delta:0.1 fair_c:0.1 num_leaves:14 num_iterations:100 scale_pos_weight:1.0 skip_drop:0.5 uniform_drop:false xgboost_dart_mode:false two_round:false top_rate:0.1 sigmoid:1.0 is_unbalance:false max_bin:255 poission_max_delta_step:0.7 min_sum_hessian_in_leaf:0.0001 other_rate:0.1 min_data_in_bin:5 max_drop:50 drop_rate:0.1 categorical_feature:0,1,2 learning_rate:0.1 threads:1 max_depth:5 feature_fraction:0.5 min_data_in_leaf:10 min_gain_to_split:20 bagging_fraction:0.9 lambda_l1:0.1 lambda_l2:0.1 bagging_freq:1 bin_construct_sample_cnt:100000 seed:1 threads:1 bags:1 verbose:false
Parameter Explanation
learning_rate weight of each estimator. This is important
bagging_fraction Proportions of rows consider. This is important
num_iterations Number of trees to build. This is important
max_depth maximum depth of the tree. This is important
feature_fraction Proportions of columns (features) to consider within a tree. This is important
objective has to be 'regression','regression_l1','fair' ,'huber','poisson'
huber_delta parameter for Huber loss. Will be used in regression task.
fair_c parameter for Fair loss. Will be used in regression task.
bagging_freq Every how many iters it will perform bagging.
bin_construct_sample_cnt Sample number of rows to create histograms.
boosting Type of boosting. Could be 'gbdt','dart' or 'goss' .
categorical_feature comma separated features to be treated as categorical
drop_rate dropout rate in dart boosting
is_unbalance true to oversample weak classes in binary classification
lambda_l1 L1 regularization
lambda_l2 L2 regularization
max_bin max number of bin that feature values will bucket in.
max_drop max number of dropped trees on one iteration (in dart).
min_data_in_bin min number of data inside one bin, use this to avoid one-data-one-bin (may prevent over-fitting).
min_data_in_leaf Minimum number of data in a leaf.
min_gain_to_split Minimum gain to split a node
min_sum_hessian_in_leaf Minimum sum hessian in one leaf
num_leaves maximum number of leaves.
other_rate only used in boosting goss, the retain ratio of small gradient data.
poission_max_delta_step safeguard optimisation.
scale_pos_weight scale weight for binary class.
sigmoid parameter for sigmoid function.
skip_drop probability of skipping drop (in dart).
top_rate used in boosting goss, the retain ratio of large gradient data.
two_round if true it saves memory but takes more time.
uniform_drop Specify whether to use uniform dropout.
boolean xgboost_dart_mode true use xgboost dart mode or not.

H2OGbmRegressor

H2OGbmRegressor ntrees:100 tweedie_power:1.2 quantile_alpha:0.1 objective:auto learn_rate:0.01 nbins:255 balance_classes:false max_depth:4 col_sample_rate_per_tree:0.5 col_sample_rate:1.0 sample_rate:0.9 min_rows:1 seed:1 threads:1 bags:1 verbose:false
Parameter Explanation
col_sample_rate Proportions of columns (features) to consider at each level of a given tree. This is important
learn_rate weight on each estimator. This is important
max_depth maximum depth of the tree. This is important
ntrees Number of trees to build This is important
sample_rate Proportions of rows consider This is important
col_sample_rate_per_tree Proportions of columns (features) to consider within a tree.
balance_classes whether to oversample the minority classes to balance the class distribution.
min_rows minimum number of cases in a node.
nbins The number of bins for the histogram to build.
tweedie_power Only applicable if Tweedie is specified for distribution) Specify the Tweedie power. The range is from 1 to 2. For a normal distribution, enter 0. For Poisson distribution, enter 1. For a gamma distribution, enter 2. For a compound Poisson-gamma distribution, enter a value greater than 1 but less than 2.
quantile_alpha Only applicable if Quantile is specified for distribution) Specify the quantile to be used for Quantile Regression.
objective The objective has to be one of [auto, gamma gaussian huber laplace poisson quantile tweedie].

H2ODeepLearningRegressor

H2ODeepLearningRegressor activation:Rectifier tweedie_power:1.2 quantile_alpha:0.1 objective:auto loss:Automatic input_dropout_ratio:0.1 shuffle:true tandardize:false weight_init:UniformAdaptive sample_rate:1.0 l1:0 l2:0.00001 max_w2:1.0 mini_batch_size:1 fast_mode:false adaptive_rate:true rho:0.9 epsilon:1e-8 balance_classes:false epochs:10 dropouts:0.5,0.5 hidden:100,50 col_sample_rate:1.0 sample_rate:0.9 min_rows:1 seed:1 threads:1 bags:1 verbose:false
Parameter Explanation
activation activation functions. Has to be between 'Rectifier', 'Tanh', 'ExpRectifier' or 'Maxout'
adaptive_rate true to use The implemented adaptive learning rate algorithm (ADADELTA) which automatically combines the benefits of learning rate annealing and momentum training to avoid slow convergence.
rho The first of two hyper parameters for ADADELTA. It is like momentum. This is important
epsilon The second of two hyper parameters for ADADELTA. This is important
balance_classes Specify whether to oversample the minority classes to balance the class distribution.
dropouts dropout ratios for each hidden layer,comma separated .Has to match in length the 'hidden' parameter. This is important
epochs Number of iterations to train the DL model. This is important
fast_mode True for faster convergence (but potential loss in accuracy)
hidden Number of hidden neurons, comma separated.The length connotes the number of hidden layers too. This is important
input_dropout_ratio dropout from to the input layer
l1 regularization on the weights.
l2 regularization on the weights. This is important
max_w2 A maximum on the sum of the squared incoming weights into any one neuron.
mini_batch_size minimum number of cases in batch.
momentum_ramp The momentum_ramp parameter controls the amount of learning for which momentum increases (assuming momentum_stable is larger than momentum_start).
momentum_stable The momentum_stable parameter controls the final momentum value reached after momentum_ramp training samples.
momentum_start The momentum_start parameter controls the amount of momentum at the beginning of training.
nesterov_accelerated_gradient True to enable Nesterov accelerated gradient descent method.
rate When adaptive learning rate is disabled, the magnitude of the weight updates are determined by the user specified learning rate (potentially annealed), and are a function of the difference between the predicted value and the target value.
rate_annealing Learning rate annealing reduces the learning rate to “freeze” into local minima in the optimization landscape.
rate_decay The learning rate decay parameter controls the change of learning rate across layers.
sample_rate Proportions of rows consider in each epoc.
shuffle true to enable shuffling of training data (on each node).
tandardize true to standardize the input data.
weight_init The distribution from which initial weights are to be drawn. Has to be 'UniformAdaptive', 'Uniform' or 'Normal'
tweedie_power Only applicable if Tweedie is specified for distribution) Specify the Tweedie power. The range is from 1 to 2. For a normal distribution, enter 0. For Poisson distribution, enter 1. For a gamma distribution, enter 2. For a compound Poisson-gamma distribution, enter a value greater than 1 but less than 2.
quantile_alpha Only applicable if Quantile is specified for distribution) Specify the quantile to be used for Quantile Regression.
objective The objective has to be of [auto, gamma ,gaussian ,huber ,laplace ,poisson ,quantile ,tweedie].
loss The loss has to be one of [Automatic ,Absolute, Huber, Quadratic or Quantile]

H2ODrfRegressor

H2ODrfRegressor ntrees:100 nbins:255 tweedie_power:1.2 quantile_alpha:0.1 objective:auto balance_classes:false max_depth:4 col_sample_rate_per_tree:0.5 sample_rate:0.9 min_rows:1 seed:1 threads:1 bags:1 verbose:false
Parameter Explanation
max_depth maximum depth of the tree. This is important
ntrees Number of trees to build. This is important
sample_rate Proportions of rows consider This is important
col_sample_rate_per_tree Proportions of columns (features) to consider within a tree.
balance_classes whether to oversample the minority classes to balance the class distribution.
min_rows minimum number of cases in a node.
nbins The number of bins for the histogram to build.
tweedie_power Only applicable if Tweedie is specified for distribution) Specify the Tweedie power. The range is from 1 to 2. For a normal distribution, enter 0. For Poisson distribution, enter 1. For a gamma distribution, enter 2. For a compound Poisson-gamma distribution, enter a value greater than 1 but less than 2.
quantile_alpha Only applicable if Quantile is specified for distribution) Specify the quantile to be used for Quantile Regression.
objective The objective has to be one of [auto, ,gamma ,gaussian ,huber ,laplace ,poisson ,quantile ,tweedie].

H2OGlmRegressor

H2OGlmRegressor alpha:0 lambda:0.00001 balance_classes:false standardize:false max_iterations:50 beta_epsilon:0.00001 bjective_epsilon:0.00001 seed:1 threads:1 bags:1 verbose:false
Parameter Explanation
alpha Proportion of l1/l2. 0 = Ridge, 1=Lasso
lambda Regularization parameter. This is important
max_iterations Number of iterations to build the model. This is important
beta_epsilon tolerance of the coefficients
bjective_epsilon tolerance of the objective function
balance_classes true to Specify whether to oversample the minority classes to balance the class distribution.
standardize true to standardize input features or not
tweedie_power Only applicable if Tweedie is specified for distribution) Specify the Tweedie power. The range is from 1 to 2. For a normal distribution, enter 0. For Poisson distribution, enter 1. For a gamma distribution, enter 2. For a compound Poisson-gamma distribution, enter a value greater than 1 but less than 2.
quantile_alpha Only applicable if Quantile is specified for distribution) Specify the quantile to be used for Quantile Regression.
family The family has to be one of [auto, gamma ,gaussian ,poisson ,tweedie]
link The link has to be one of [auto, log ,identity ,inverse ,tweedie]

SklearnAdaBoostRegressor

The original parameters can be found here

SklearnAdaBoostRegressor algorithm:square learning_rate:0.7 n_estimators:100 threads:1 usedense:false seed:1 verbose:false
Parameter Explanation
learning_rate Learning rate shrinks the contribution of each classifier by learning_rate. This is important
n_estimators Number of trees to build. This is important
algorithm Could be square, linear or exponential.
use_dense True to Use dense data. If your data is in dense format, do select true as all files get loaded as sparse by default in python-based modules.

SklearnDecisionTreeRegressor

The original parameters can be found here

SklearnDecisionTreeRegressor criterion:mse max_leaf_nodes:0 min_impurity_split:0.0000001 threads:1 max_features:0.5 max_depth:5 min_samples_leaf:1 min_samples_split:2 use_dense:false min_weight_fraction_leaf:0.0 new_tree_gain_ratio:1.0 bootsrap:false seed:1 verbose:false
Parameter Explanation
max_depth maximum depth of the tree. This is important
max_features Proportions of columns (features) to consider. This is important
max_leaf_nodes maximum number of nodes allowed.
use_dense True to Use dense data. If your data is in dense format, do select true as all files get loaded as sparse by default in python-based modules.
min_impurity_split Threshold for early stopping in tree growth.
criterion Criterion to determine the split could be mse or mae
min_samples_leaf Minimum cases to keep a splitted node
min_samples_split Minimum cases to split a node
min_weight_fraction_leaf The minimum weighted fraction of the sum total of weights (of all the input samples) required to be at a leaf node.

SklearnExtraTreesRegressor

The original parameters can be found here

SklearnExtraTreesRegressor criterion:mse max_leaf_nodes:0 n_estimators:100 min_impurity_split:0.0000001 threads:1 max_features:0.5 max_depth:5 min_samples_leaf:1 min_samples_split:2 use_dense:false min_weight_fraction_leaf:0.0 new_tree_gain_ratio:1.0 bootsrap:false seed:1 verbose:false 		  
Parameter Explanation
n_estimators Toral number of trees to build. This is important
max_depth maximum depth of the tree. This is important
max_features Proportions of columns (features) to consider. This is important
use_dense True to Use dense data. If your data is in dense format, do select true as all files get loaded as sparse by default in python-based modules.
max_leaf_nodes maximum number of nodes allowed.
min_impurity_split Threshold for early stopping in tree growth.
bootsrap true use bootsrap or not.
criterion Criterion to determine the split could mse or mae
min_samples_leaf Minimum cases to keep a splitted node
min_samples_split Minimum cases to split a node
min_weight_fraction_leaf The minimum weighted fraction of the sum total of weights (of all the input samples) required to be at a leaf node.

SklearnRandomForestRegressor

The original parameters can be found here

SklearnRandomForestRegressor criterion:mse max_leaf_nodes:0 n_estimators:100 min_impurity_split:0.0000001 threads:1 max_features:0.5 max_depth:5 min_samples_leaf:1 min_samples_split:2 use_dense:false min_weight_fraction_leaf:0.0 new_tree_gain_ratio:1.0 bootsrap:false seed:1 verbose:false 		  
Parameter Explanation
n_estimators Toral number of trees to build. This is important
max_depth maximum depth of the tree. This is important
max_features Proportions of columns (features) to consider. This is important
use_dense True to Use dense data. If your data is in dense format, do select true as all files get loaded as sparse by default in python-based modules.
max_leaf_nodes maximum number of nodes allowed.
min_impurity_split Threshold for early stopping in tree growth.
bootsrap true use bootsrap or not.
criterion Criterion to determine the split could mse or mae
min_samples_leaf Minimum cases to keep a splitted node
min_samples_split Minimum cases to split a node
min_weight_fraction_leaf The minimum weighted fraction of the sum total of weights (of all the input samples) required to be at a leaf node.

SklearnMLPRegressor

The original parameters can be found here

SklearnMLPRegressor standardize:true use_log1p:true shuffle:true learning_rate:adaptive momentum:0.9 optimizer:sgd use_dense:false alpha:0.000001 hidden:50,50 activation:relu epsilon:0.00000001 max_iter:50 learning_rate_init:0.01 batch_size:8 tol:0.0001 validation_split:0.2 seed:1 verbose:false
Parameter Explanation
hidden Toral Comma-separated integers defining the number of hidden layers along with teh hidden units. This is important
epochs Maximum number of iterations. This is important
activation Activation function for the hidden layer coudl be identity, logistic, tanh, relu. This is important
alpha L2 regularization on the weights. This is important
learning_rate_init The (initial) learning rate used. This is important
learning_rate Could be adaptive ,constant or invscaling.
batch_size Number of cases(samples) in a batch.
optimizer could adam, bfgs or sgd.
tol Tolerance to determine the end of the optimization.
momentum Only applicable for optimizer=sgd. Nesterov's is on by default.
epsilon Value for numerical stability in adam.
shuffle true Enable shuffling of training data (on each epoc).
standardize true to standardize dense data (e.g use_dense=true) or to use absmaxscaling on sparse data (use_dense=false).
use_log1p converts the data matrix to log plus 1.
validation_split Split percentage to use for early stopping. Best epoc is determived via cv after 2 consecutive worse loss estimateds.
use_dense True to Use dense data. If your data is in dense format, do select true as all files get loaded as sparse by default in python-based modules.

SklearnSGDRegressor

The original parameters can be found here

SklearnSGDRegressor standardize:true use_log1p:true shuffle:true learning_rate:constant l1_ratio:0.1 penalty:l2 use_dense:false alpha:0.00001 loss:squared_loss epsilon:0.00000001 n_iter:50 eta0:0.01 power_t:0.25 seed:1 threads:3 verbose:false
Parameter Explanation
n_iter Maximum number of iterations. This is important
alpha Regularization on the weights. This is important
eta0 The (initial) learning rate used. This is important
learning_rate Could be optimal, constant or invscaling.
loss could be squared_loss, huber, epsilon_insensitive or squared_epsilon_insensitive.
epsilon For huber, determines the threshold at which it becomes less important to get the prediction right. For epsilon-insensitive, any differences between the current prediction and the correct label are ignored if they are less than this.
l1_ratio The Elastic Net mixing parameter, with 0 <= l1_ratio <= 1. l1_ratio=0 corresponds to L2 penalty, l1_ratio=1 to L1.
penalty The penalty (aka regularization term) to be used. could be l2, l1, or elasticnet .
power_t The exponent for inverse scaling learning rate [default 0.5].
shuffle true Enable shuffling of training data (on each iteration).
standardize true to standardize dense data (e.g use_dense=true) or to use absmaxscaling on sparse data (use_dense=false).
use_log1p converts the data matrix to log plus 1.
use_dense True to Use dense data. If your data is in dense format, do select true as all files get loaded as sparse by default in python-based modules.

SklearnknnRegressor

The original parameters can be found here

SklearnknnRegressor seed:1 usedense:true use_scale:false distance:cityblock metric:uniform n_neighbors:3 thread:1 verbose:false
Parameter Explanation
n_neighbors Number of neighbors to use by default for k_neighbors queries. This is important
distance It must be one of euclidean, cosine, manhattan or cityblock
metric Weight function used in prediction. Possible values: uniform or distance.
use_scale true to use absmaxscaling.
use_dense True to Use dense data. If your data is in dense format, do select true as all files get loaded as sparse by default in python-based modules.

SklearnsvmRegressor

The original parameters can be found here

SklearnsvmRegressor seed:1 usedense:false use_scale:false max_iter:-1 kernel:rbf degree:3 C:1.0 tol:0.0001 coef0:0.0 gamma:0.0 verbose:False	
Parameter Explanation
max_iter Maximum number of iterations. This is important
kernel Kernel type could be linear, poly, rbf or sigmoid. This is important
C The Penalty parameter C of the error term. This is important
tol Tolerance to determine the end of the optimization.
degree Degree of the polynomial kernel function (poly).
gamma Kernel coefficient for rbf, poly and sigmoid.
coef Independent term in kernel function.It is only significant in poly and sigmoid.
use_scale true to use absmaxscaling.
use_dense True to Use dense data. If your data is in dense format, do select true as all files get loaded as sparse by default in python-based modules.

KerasnnRegressor

The original parameters can be found sparsely in keras' documentation

KerasnnRegressor loss:mean_squared_error standardize:true use_log1p:true shuffle:true batch_normalization:true weight_init:lecun_uniform momentum:0.9 optimizer:adam use_dense:true l2:0.000001,0.000001 hidden:50,50 activation:relu,relu droupouts:0.1,0.1 epochs:20 lr:0.01 batch_size:8 stopping_rounds:10 validation_split:0.2 seed:1 verbose:false 
Parameter Explanation
hidden Toral Comma-separated integers defining the number of hidden layers along with teh hidden units. This is important
droupouts Toral Comma-separated floats defining the dropout in each layer (defined by hidden). This is important
l2 Toral Comma-separated floats defining the l2 regularization term on the weights in each layer (defined by hidden). This is important
activation Toral Comma-separated strings defining the activation in each hidden layer. This is important
lr The learning rate used. This is important
epochs Maximum number of iterations. This is important
batch_normalization true to add a batch normlization to the layers. This is important
batch_size Number of cases(samples) in a batch. This is important
weight_init The distribution from which initial weights are to be drawn. Has to be RandomNormal, RandomUniform, TruncatedNormal, VarianceScaling, Orthogonal, Identity, lecun_uniform, glorot_normal,glorot_uniform, he_normal, lecun_normal, he_uniform, he_normal.
optimizer Has to be adam, adagrad, nadam, adadelta or sgd.
loss Has to be mean_squared_error, mean_absolute_error, mean_squared_logarithmic_error, squared_hinge, hinge, poisson.
momentum Only applicable for optimizer=sgd. Nesterov's is on by default.
shuffle true Enable shuffling of training data (on each epoc).
standardize true to standardize dense data (e.g use_dense=true) or to use absmaxscaling on sparse data (use_dense=false).
use_log1p converts the data matrix to log plus 1.
validation_split Split percentage to use for early stopping. Best epoc is determived via cv after 2 consecutive worse loss estimateds.
stopping_rounds Split percentage to use for early stopping. Best epoc is determived via cv after 2 consecutive worse loss estimateds.
use_dense True to Use dense data. If your data is in dense format, do select true as all files get loaded as sparse by default in python-based modules.

PythonGenericRegressor

The user can run his/her own python script as long as it is placed in lib/python/ and named after PythonGenericRegressor[INDEX]. Index will be a hyper parameter. Look for PythonGenericRegressor0.py in lib/python/ for an example.

PythonGenericRegressor index:0 seed:1 verbose:False 
Parameter Explanation
index this is the index specifying which PythonGenericRegressor[index].py script to run. This is important

FRGFRegressor

(Some of) the original parameters of fast_rgf can be found here

FRGFRegressor dtree_loss:LS max_nodes:16 ntrees:100 eta:0.1 threads:1 sparse_max_features:1000 max_level:4 dense_max_buckets:250 min_sample:1 min_occurrences:1 opt:rgf sparse_lamL2:1.0 min_bucket_weights:250 new_tree_gain_ratio:1.0 stepsize:0.1 lamL2:1.0 lamL1:1.0 sparse_max_buckets:250 verbose:False
Parameter Explanation
ntrees Toral number of trees to build. This is important
max_level maximum depth of the tree. This is important
lamL2 L2 regularization on the weights. This is important
new_tree_gain_ratio new tree is created when leaf-nodes gain < this value * estimated gain of creating new three. This is important
lamL1 L1 regularization on the weights.
stepsize Step size of epsilon-greedy boosting (inactive for rgf).
min_occurrences minimum number of occurrences for a feature to be selected.
min_sample minimum samples in node.
max_nodes maximum number of nodes.
loss Type of loss. could be LS, MODLS (modified least squares loss), or LOGISTIC.
opt optimization method for training forest. Could be rgf or epsilon-greedy.
sparse_lamL2 L2 regularization parameter for sparse data.
min_bucket_weights Minimum sum of data weights for each discretized value.
dense_max_buckets Maximum bins for dense data.
sparse_max_features You may try a different value in [1000,10000000] for fetaures allowed.
dense_max_buckets Maximum bins for dense data.

OriginalLibFMRegressor

Wraps the original implementation of ibFM, made from Steffen Rendle. The reason this implementation is made, is because internal results show that it has better performance (as in accuracy) than StackNet's internal implementation and has more training methods than just sgd. This implementation may not include all libFM features plus it actually uses a version of it that had a bug(!) on purpose. You can find more information about why this was chosen in the following python wrapper for libFM. It basically had this bug that was allowing you to get the parameters of the trained models for all training methods. These parameters are now extracted once a model has been trained and the scoring uses only these parameters (e.g. not the libFM executable). Don't forget to acknowledge libFM if you publish results produced with this software. Also take note of its licence GNU. More information can be found on libFM's repo on github.

OriginalLibFMRegressor type:als lfeatures:4 learn_rate:0.01 maxim_Iteration:10 init_values:0.01 c:1 c2:1 threads:1 usescale:false seed:1 verbose:true bags:1 
Parameter Explanation
Type Type of algorithm to use. It has to be sgd, als, mcmc. Default is mcmc.
C Regularization value, the more, the stronger the regularization. This is important
C2 Regularization value for the latent features. This is important
lfeatures Number of latent features to use. This is important
init_values Initialise values of the latent features with values between[0,init_values). This is important
maxim_Iteration aximum number of iterations. This is important
learn_rate learn_rate for SGD; default=0.1. This is important

VowpaLWabbitRegressor

Wrapper for vowpal wabbit. It does not contain all features, but a fraction.

VowpaLWabbitRegressor use_ftrl:True learning_rate:0.8 decay_learning_rate:0.97 nn:40 bit_precision:18 ftrl_alpha:0.1 ftrl_beta:0.1 power_t:0.9 initial_t:0.9 l1:0.01 l2:0.01 passes:10 threads:1 make2way:false make3way:false use_dropout:true use_meanfield:false seed:1 verbose:true bags:1  
Parameter Explanation
passes Number of training Passes. This is important
bit_precision number of bits in the feature table.
decay_learning_rate Decay factor for learning_rate between passes.
nn Number of hidden units to use in a sigmoidal feedforward network with nn hidden units
initial_t Initial t value. Affects learning rate's updates
power_t t power value. Affects learning rate's updates
ftrl_alpha ftrl alpha parameter when using ftrl This is important
ftrl_beta ftrl beta stability patameter when using ftrl This is important
learning_rate learning Rate for gradient-based updates
l1 L1 regularization
l2 L2 regularization This is important
use_ftrl o use the ftrl optimization option (instead of adaptive). It is on by default.
make2way if true it creates all possible 2-way interactions of all features
make3way if true it creates all possible 3-way interactions of all features
use_dropout when nn>0, train or test sigmoidal feedforward network using dropout.
use_meanfield when nn>0, train or test sigmoidal feedforward network using mean field.