IGNITE-5217: Add Gradient Descent and QR-based trainers for Linear Regression #3308
Conversation
datasets, add BarzilaiBorwein gradient descent updater.
| * @param model linear regression model | ||
| * @return formatted string representation | ||
| */ | ||
| private static String formatLinearRegressionModelPrettyPrint(LinearRegressionModel model) { |
There was a problem hiding this comment.
Better to add into LinearRegressionModel. I'd like the pretty print. Also you could add constraint in 30 or 50 vars to print out this model.
There was a problem hiding this comment.
Added into toString of LinearRegressionModel.
| System.out.println(">>> ---------------------------------"); | ||
| System.out.println(">>> | Prediction\t| Ground Truth\t|"); | ||
| System.out.println(">>> ---------------------------------"); | ||
| for (double[] observation : data) { |
There was a problem hiding this comment.
The best way here to use LabeledDatasets (but it can be a separate ticket)
There was a problem hiding this comment.
Good idea, will be done in separate task.
| * @return This gradient descent instance | ||
| */ | ||
| public GradientDescent withMaxIterations(int maxIterations) { | ||
| if (maxIterations < 0) |
There was a problem hiding this comment.
Mostly we use asserts to check input arguments, Let's discuss it with @ybabak @YuriBabak
There was a problem hiding this comment.
Replaced with asserts.
| if (convergenceTol < 0) | ||
| throw new IllegalArgumentException("Convergence tolerance must be non-negative but got " + convergenceTol); | ||
| this.convergenceTol = convergenceTol; | ||
| return this; |
There was a problem hiding this comment.
Mostly we don't use chain and use only setters. Let's discuss it with @ybabak @YuriBabak
I'd like chain
There was a problem hiding this comment.
As discussed will keep this approach with chain methods.
| */ | ||
| public Vector optimize(Matrix data, Vector initialWeights) { | ||
| Vector weights = initialWeights, oldWeights = null, oldGradient = null; | ||
| if (data instanceof SparseDistributedMatrix) { |
There was a problem hiding this comment.
Could we refactor it to one method to avoid copy-paste?
There was a problem hiding this comment.
Good idea, done.
| long ts1 = System.currentTimeMillis(); | ||
| Vector groundTruth = extractGroundTruth(data); | ||
| Matrix inputs = extractInputs(data); | ||
| long ts2 = System.currentTimeMillis(); |
There was a problem hiding this comment.
Don't forget remove these timestamps
| 83.6427296424, 27.4571268153, 73.5881193584, 27.1465364511, 79.4095449062}, -5.14077007134); | ||
|
|
||
| /** */ | ||
| public static class Dataset { |
There was a problem hiding this comment.
Please rename this class to TestDataset or something else. Or use LabeledDataset after update
|
|
||
| // Check expected residuals from R | ||
| mdl.estimateResiduals(); | ||
| LinearRegressionSGDTrainer trainer = new LinearRegressionSGDTrainer(100_000, 1e-12); |
There was a problem hiding this comment.
But this was about OLS Regression and you use SGDTrainer. It's strange
There was a problem hiding this comment.
Generally speaking SGDTrainer also makes OLS Regression just because it uses Least Square as a loss function. Anyway, I replaced it with QR trainer for better transparency.
| 20.6, 4.83567, 0.0, 18.1, 0.0, 0.583, 5.905, 53.2, 3.1523, 24.0, 666.0, 20.2, 388.22, 11.45 | ||
| 15.2, 0.15086, 0.0, 27.74, 0.0, 0.609, 5.454, 92.7, 1.8209, 4.0, 711.0, 20.1, 395.09, 18.06 | ||
| 7.0, 0.18337, 0.0, 27.74, 0.0, 0.609, 5.414, 98.3, 1.7554, 4.0, 711.0, 20.1, 344.05, 23.97 | ||
| 8.1, 0.20746, 0.0, 27.74, 0.0, 0.609, 5.093, 98.0, 1.8226, 4.0, 711.0, 20.1, 318.43, 29.68 |
There was a problem hiding this comment.
Don't forget add readme file with short description about attributes, licenses or link to the dataset repository
There was a problem hiding this comment.
README.txt with dataset descriptions added.
| SparseDistributedMatrix distributedMatrix = new SparseDistributedMatrix(data); | ||
|
|
||
| System.out.println(">>> Create new linear regression trainer object."); | ||
| Trainer<LinearRegressionModel, Matrix> trainer = new LinearRegressionQRTrainer(); |
There was a problem hiding this comment.
OLS could be solved not by QR decomposition only.
There was a problem hiding this comment.
Added separate examples for QRTrainer and SGDTrainer.
No description provided.