Multinomial logistic regression

[YuhanLiin]

Implementation of multinomial logistic regression. Also added some tests. Still need to add docs and possibly refactor.

[codecov-commenter]

Codecov Report

Merging #159 (65dc913) into master (992938e) will increase coverage by 0.62%.
The diff coverage is 75.61%.

@@            Coverage Diff             @@
##           master     #159      +/-   ##
==========================================
+ Coverage   59.84%   60.46%   +0.62%     
==========================================
  Files          84       84              
  Lines        7748     7923     +175     
==========================================
+ Hits         4637     4791     +154     
- Misses       3111     3132      +21     

Impacted Files	Coverage Δ
algorithms/linfa-logistic/src/float.rs	0.00% <0.00%> (ø)
algorithms/linfa-logistic/src/argmin_param.rs	47.05% <36.36%> (+25.63%)	⬆️
algorithms/linfa-logistic/src/lib.rs	78.02% <77.77%> (+5.04%)	⬆️
src/correlation.rs	52.45% <0.00%> (-0.88%)	⬇️
algorithms/linfa-hierarchical/src/lib.rs	52.23% <0.00%> (-0.80%)	⬇️
algorithms/linfa-linear/src/glm.rs	46.66% <0.00%> (-0.40%)	⬇️
...linfa-clustering/src/gaussian_mixture/algorithm.rs	56.52% <0.00%> (-0.10%)	⬇️
src/dataset/mod.rs	88.09% <0.00%> (ø)
algorithms/linfa-nn/tests/nn.rs	85.13% <0.00%> (ø)
algorithms/linfa-pls/src/pls_generic.rs	69.76% <0.00%> (+0.27%)	⬆️
... and 5 more

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 992938e...65dc913. Read the comment docs.

[bytesnake]

would a review be helpful, despite being a draft?

[YuhanLiin]

One observation I have of the multinomial implementation is that it tends to diverge when the input is not normalized well. Not sure if this is a natural property of the algorithm itself.
Specifically, the algorithm gets stuck on the wine quality data due to divergence, even though binomial regression works fine on that data.

[YuhanLiin]

The unnormalized wine quality also diverges on scikit-learn. However, the algorithm actually terminates since it hits the max iteration limit. With this implementation the algorithm just gets stuck on the first iteration of the solver forever. This likely has something to do with the linear search library we're using.

[bytesnake]

nice work 👍

[bytesnake]

doesn't this try to achieve what we are doing with FitWith? (i.e. you can also initialize a linear model with these parameters and call fit_with on that)

[YuhanLiin]

I though FitWith was meant to be used for "online" versions of algorithms for faster fitting? This method existed before so I just kept it. It has the same purpose as setting the initial centroids when using KMeans, which is useful even when using FitWith.

[bytesnake]

yes, but I think it's more intuitive if a fitted model can be constructed (i.e. having a FittedLogisticRegression::new function) with a given intercept and hyperplane. You can then update those initial weights with FitWith instead of putting them to the hyperparameters.

[YuhanLiin]

Both FittedLogisticRegression and MultiFittedLogisticRegression require the set of labels to be specified at construction, which isn't info that's available before fitting. If we make FitWith perform a single iteration then the users will have to write their own loops and stopping conditions, which is more work. We could make FitWith perform the complete algorithm to solve this. Also, if we have the Fit version of the algorithm always start with param arrays of 0s, it'd be pretty much useless since the optimal starting params are random values, not 0s. I just don't think it makes sense to delegate the offline and online versions of the algorithm to the same impl. What we can do is have separate param types for the online and offline versions of the algorithm, both implementing FitWith. The offline impl would run the entire algorithm while the online impl runs a single step. This makes sense because the offline algorithm requires a few hyperparams that the online version doesn't need.