Current QBoost implementation is just a diminished AdaBoost

[mcfarljm]

As applied to the given demonstration problems, the QBoost algorithm does essentially nothing. AdaBoost is actually run first to pre-select a set of weak classifiers, which are then provided to the QBoost algorithm. With the current settings and demonstration problems, the results are simply that QBoost includes all of the classifiers that AdaBoost provides to it (confirm by seeing the list of "1" weights in the output for QBoost). In other words, for these problems, QBoost could be replaced by the following anti-algorithm:

1. Run AdaBoost
2. Change all of the weights that AdaBoost came up with to 1

Note that the screen output is misleading in terms of the method comparisons. For example, runs with the "mnist" data set often show that "QBoost" is performing better than Adaboost. This is misleading because what is labeled Adaboost is actually sklearn Adaboost running with weak decision tree classifiers at depth 1, whereas QBoost is pulling from a custom Adaboost implementation that uses weak decision tree classifiers with depth 3 (confusingly, this custom Adaboost implementation is labeled "DecisionTree" in the output; see Issue #13).

The key questions are what weak classifiers should be considered in the QBoost algorithm, and what actually is the definition of the QBoost algorithm? The README refers to the earlier 2008 paper (https://arxiv.org/pdf/0811.0416.pdf). Aside from not actually using the QBoost terminology, this paper presents the algorithm as drawing from all possible "degree 1 and 2 decision stumps" (basically decision rules using either a single variable or a product of two variables). As described, this produces a large number of variables if doing a one-shot global optimization: 930 variables for the 30-feature case and many more for the 784-feature case. Compare these numbers to the 35 variables being currently used because QBoost is being fed weak classifiers pre-selected by AdaBoost.

A different version of the method is described in the more recent 2012 paper (http://proceedings.mlr.press/v25/neven12/neven12.pdf), which introduces the name QBoost. This paper reduces the problem size by using "inner" and "outer" loops that pre-select the weak classifiers as detailed by Algorithms 1 and 2. Note that neither paper discusses using AdaBoost to pre-select the weak classifiers, and if one is going to do that, it is unclear what the motivation for QBoost is (the 2008 paper contrasts QBoost as a "global optimization" vs the "greedy" AdaBoost method, but this is nullified if we simply use AdaBoost first).

In conclusion, my suggestions are the following:

• The code should be reworked to use an implementation of QBoost that does not simply re-use the classifiers selected by AdaBoost. Probably the implementation should draw from the 2012 paper, otherwise it is unclear how to select a set of weak classifiers and achieve a reasonable problem size.
• Any comparisons against other boosting methods such as AdaBoost should use the same pool of weak classifiers (as done in the papers mentioned above). Otherwise, it is not an apples-to-apples comparison.

Both the 2008 and 2012 papers show improved performance relative to AdaBoost when using the same pool of weak classifiers. It would be nice to illustrate that through this demo as well.

[mcfarljm]

Resolved by PR #19.