diff --git a/README.md b/README.md index fa643f8e..5c93275c 100644 --- a/README.md +++ b/README.md @@ -5,85 +5,39 @@ # Machine learning algorithms with dart -**Table of contents** -- [What for is the library?](#what-is-the-ml_algo-for) -- [The library's structure](#the-librarys-structure) -- [Examples](#examples) - - [Logistic regression](#logistic-regression) - - [Softmax regression](#softmax-regression) - - [KNN regression](#k-nearest-neighbour-regression) - ## What is the ml_algo for? The main purpose of the library - to give developers, interested both in Dart language and data science, native Dart implementation of machine learning algorithms. This library targeted to dart vm, so, to get smoothest experience with the lib, please, do not use it in a browser. -**Following algorithms are implemented:** -- *Linear regression:* - - Gradient descent based linear regression - - Coordinate descent based linear regression - -- *Linear classifier:* - - Logistic regression - - Softmax regression - -- *Non-linear classifier:* - - Decision tree classifier (majority-based prediction) - -- *Non-parametric regression:* - - KNN regression - -## The library's structure +## The library's content - #### Model selection - [CrossValidator](https://github.com/gyrdym/ml_algo/blob/master/lib/src/model_selection/cross_validator/cross_validator.dart). - Factory, that creates instances of a cross validator. Cross validation allows researchers to fit different + Factory, that creates instances of cross validators. Cross validation allows researchers to fit different [hyperparameters](https://en.wikipedia.org/wiki/Hyperparameter_(machine_learning)) of machine learning algorithms, assessing prediction quality on different parts of a dataset. - #### Classification algorithms - - ##### Linear classification - - ###### Logistic regression - An algorithm, that performs linear binary classification. - - [LogisticRegressor.gradient](https://github.com/gyrdym/ml_algo/blob/master/lib/src/classifier/linear/logistic_regressor/logistic_regressor.dart). - Logistic regression with gradient ascent optimization of log-likelihood cost function. To use this kind - of classifier your data have to be [linearly separable](https://en.wikipedia.org/wiki/Linear_separability). - - - [LogisticRegressor.coordinate](https://github.com/gyrdym/ml_algo/blob/master/lib/src/classifier/linear/logistic_regressor/logistic_regressor.dart). - Not implemented yet. Logistic regression with coordinate descent optimization of negated log-likelihood cost - function. Coordinate descent allows to do feature selection (aka `L1 regularization`) To use this kind of - classifier your data have to be [linearly separable](https://en.wikipedia.org/wiki/Linear_separability). + - [LogisticRegressor](https://github.com/gyrdym/ml_algo/blob/master/lib/src/classifier/logistic_regressor.dart). + A class, that performs linear binary classification of data. To use this kind of classifier your data have to be + [linearly separable](https://en.wikipedia.org/wiki/Linear_separability). + + - [SoftmaxRegressor](https://github.com/gyrdym/ml_algo/blob/master/lib/src/classifier/softmax_regressor.dart). + A class, that performs linear multiclass classification of data. To use this kind of classifier your data have to be + [linearly separable](https://en.wikipedia.org/wiki/Linear_separability). - - ##### Softmax regression - An algorithm, that performs linear multiclass classification. - - [SoftmaxRegressor.gradient](https://github.com/gyrdym/ml_algo/blob/master/lib/src/classifier/linear/softmax_regressor/softmax_regressor.dart). - Softmax regression with gradient ascent optimization of log-likelihood cost function. To use this kind - of classifier your data have to be [linearly separable](https://en.wikipedia.org/wiki/Linear_separability). - - - [SoftmaxRegressor.coordinate](https://github.com/gyrdym/ml_algo/blob/master/lib/src/classifier/linear/softmax_regressor/logistic_regressor.dart). - Not implemented yet. Softmax regression with coordinate descent optimization of negated log-likelihood cost - function. As in case of logistic regression, coordinate descent allows to do feature selection (aka - `L1 regularization`) To use this kind of classifier your data have to be - [linearly separable](https://en.wikipedia.org/wiki/Linear_separability). - - ##### Nonlinear classification - - ###### Decision tree classifier - - [DecisionTreeClassifier.majority](https://github.com/gyrdym/ml_algo/blob/master/lib/src/classifier/non_linear/decision_tree/decision_tree_classifier.dart) + - [DecisionTreeClassifier](https://github.com/gyrdym/ml_algo/blob/master/lib/src/classifier/decision_tree_classifier.dart) + A class, that performs classification, using decision trees. May work with data with non-linear patterns. - #### Regression algorithms - - ##### Linear regression - - [LinearRegressor.gradient](https://github.com/gyrdym/ml_algo/blob/master/lib/src/regressor/linear_regressor.dart). A - well-known algorithm, that performs linear regression using [gradient vector](https://en.wikipedia.org/wiki/Gradient) of a cost - function. - - - [LinearRegressor.coordinate](https://github.com/gyrdym/ml_algo/blob/master/lib/src/regressor/linear_regressor.dart) An algorithm, - that uses coordinate descent in order to find optimal value of a cost function. Coordinate descent allows to - perform feature selection along with regression process (This technique often calls `Lasso regression`). - - - ##### Nonlinear regression - - [ParameterlessRegressor.knn](https://github.com/gyrdym/ml_algo/blob/master/lib/src/regressor/non_parametric_regressor.dart) - An algorithm, that makes prediction for each new observation based on first `k` closest observations from training data. - It has quite high computational complexity, but in the same time it may easily catch non-linear pattern of the data. + - [LinearRegressor](https://github.com/gyrdym/ml_algo/blob/master/lib/src/regressor/linear_regressor.dart). A + class, that finds a linear pattern in training data and predicts a real numbers depending on the pattern. + + - [KnnRegressor](https://github.com/gyrdym/ml_algo/blob/master/lib/src/regressor/knn_regressor.dart) + A class, that makes prediction for each new observation basing on first `k` closest observations from + training data. It may catch non-linear pattern of the data. ## Examples @@ -92,21 +46,23 @@ the lib, please, do not use it in a browser. Let's classify records from well-known dataset - [Pima Indians Diabets Database](https://www.kaggle.com/uciml/pima-indians-diabetes-database) via [Logistic regressor](https://github.com/gyrdym/ml_algo/blob/master/lib/src/classifier/linear_classifier.dart) -Import all necessary packages. First, it's needed to ensure, if you have `ml_preprocessing` package in your -dependencies: +Import all necessary packages. First, it's needed to ensure, if you have `ml_preprocessing` and `ml_dataframe` package +in your dependencies: ```` dependencies: - ml_preprocessing: ^3.2.0 + ml_dataframe: ^0.0.11 + ml_preprocessing: ^5.0.1 ```` -We need this repo to parse raw data in order to use it farther. For more details, please, +We need these repos to parse raw data in order to use it farther. For more details, please, visit [ml_preprocessing](https://github.com/gyrdym/ml_preprocessing) repository page. ````dart import 'dart:async'; import 'package:ml_algo/ml_algo.dart'; +import 'package:ml_dataframe/ml_dataframe.dart'; import 'package:ml_preprocessing/ml_preprocessing.dart'; ```` @@ -114,46 +70,45 @@ Download dataset from [Pima Indians Diabets Database](https://www.kaggle.com/uci read it (of course, you should provide a proper path to your downloaded file): ````dart -final data = DataFrame.fromCsv('datasets/pima_indians_diabetes_database.csv', - labelName: 'class variable (0 or 1)'); -final features = (await data.features) - .mapColumns((column) => column.normalize()); // it's needed to normalize the matrix column-wise to reach - // computational stability and provide uniform scale for all - // the values in the column -final labels = await data.labels; +final samples = await fromCsv('datasets/pima_indians_diabetes_database.csv', headerExists: true); ```` Data in this file is represented by 768 records and 8 features. 9th column is a label column, it contains either 0 or 1 -on each row. This column is our target - we should predict a class label for each observation. Therefore, we -should point, where to get label values. Let's use `labelName` parameter for that (labels column name, 'class variable -(0 or 1)' in our case). - -Processed features and labels are contained in data structures of `Matrix` type. To get more information about -`Matrix` type, please, visit [ml_linal repo](https://github.com/gyrdym/ml_linalg) +on each row. This column is our target - we should predict a class label for each observation. The column's name is +`class variable (0 or 1)`. Let's store it: +````dart +final targetColumnName = 'class variable (0 or 1)'; +```` + Then, we should create an instance of `CrossValidator` class for fitting [hyperparameters](https://en.wikipedia.org/wiki/Hyperparameter_(machine_learning)) -of our model +of our model. We should pass training data (our `samples` variable), a list of target column names (in our case it's +just a name stored in `targetColumnName` variable) and a number of folds into CrossValidator constructor. + ````dart -final validator = CrossValidator.KFold(numberOfFolds: 5); +final validator = CrossValidator.KFold(samples, [targetColumnName], numberOfFolds: 5); ```` All are set, so, we can do our classification. Evaluate our model via accuracy metric: + ````dart -final accuracy = validator.evaluate((trainFeatures, trainLabels) => - LogisticRegressor.gradient( - trainFeatures, trainLabels, +final accuracy = validator.evaluate((samples, targetNames) => + LogisticRegressor( + samples, + targetNames[0], // remember, we provided a list of just a single name + optimizerType: LinearOptimizerType.gradient, initialLearningRate: .8, iterationsLimit: 500, - batchSize: 768, + batchSize: samples.rows.length, fitIntercept: true, interceptScale: .1, - learningRateType: LearningRateType.constant), - features, labels, MetricType.accuracy); + learningRateType: LearningRateType.constant + ), MetricType.accuracy); ```` -Let's print score: +Let's print the score: ````dart print('accuracy on classification: ${accuracy.toStringAsFixed(2)}'); ```` @@ -165,133 +120,30 @@ acuracy on classification: 0.77 ```` All the code above all together: -````dart -import 'dart:async'; - -import 'package:ml_algo/ml_algo.dart'; -import 'package:ml_preprocessing/ml_preprocessing.dart'; - -Future main() async { - final data = DataFrame.fromCsv('datasets/pima_indians_diabetes_database.csv', - labelName: 'class variable (0 or 1)'); - final features = (await data.features).mapColumns((column) => column.normalize()); - final labels = await data.labels; - final validator = CrossValidator.kFold(numberOfFolds: 5); - final accuracy = validator.evaluate((trainFeatures, trainLabels) => - LogisticRegressor.gradient( - trainFeatures, trainLabels, - initialLearningRate: .8, - iterationsLimit: 500, - batchSize: 768, - fitIntercept: true, - interceptScale: .1, - learningRateType: LearningRateType.constant), - features, labels, MetricType.accuracy); - - print('accuracy on classification: ${accuracy.toStringFixed(2)}'); -} -```` - -### Softmax regression -Let's classify another famous dataset - [Iris dataset](https://www.kaggle.com/uciml/iris). Data in this csv is separated into 3 classes - therefore we need -to use different approach to data classification - [Softmax regression](http://deeplearning.stanford.edu/tutorial/supervised/SoftmaxRegression/). - -As usual, start with data preparation. Before we start, we should update our pubspec's dependencies with [xrange](https://github.com/gyrdym/xrange)` -library: - -```` -dependencies: - ... - xrange: ^0.0.5 - ... -```` - -Download the file and read it: - -````Dart -final data = DataFrame.fromCsv('datasets/iris.csv', - labelName: 'Species', - columns: [ZRange.closed(1, 5)], - categories: { - 'Species': CategoricalDataEncoderType.oneHot, - }, -); - -final features = await data.features; -final labels = await data.labels; -```` - -The csv database has 6 columns, but we need to get rid of the first column, because it contains just ID of every -observation - it's absolutely useless data. So, as you may notice, we provided a columns range to exclude ID-column: - -````Dart -columns: [ZRange.closed(1, 5)] -```` - -Also, since the label column 'Species' has categorical data, we encoded it to numerical format: - -````Dart -categories: { - 'Species': CategoricalDataEncoderType.oneHot, -}, -```` - -Next step - create a cross validator instance: - -````Dart -final validator = CrossValidator.kFold(numberOfFolds: 5); -```` - -Evaluate quality of prediction: - -````Dart -final accuracy = validator.evaluate((trainFeatures, trainLabels) => - LinearClassifier.softmaxRegressor( - trainFeatures, trainLabels, - initialLearningRate: 0.03, - iterationsLimit: null, - minWeightsUpdate: 1e-6, - randomSeed: 46, - learningRateType: LearningRateType.constant - ), features, labels, MetricType.accuracy); - -print('Iris dataset, softmax regression: accuracy is ' - '${accuracy.toStringAsFixed(2)}'); // It yields 0.93 -```` - -Gather all the code above all together: - -````Dart -import 'dart:async'; +````dart import 'package:ml_algo/ml_algo.dart'; +import 'package:ml_dataframe/ml_dataframe.dart'; import 'package:ml_preprocessing/ml_preprocessing.dart'; -import 'package:xrange/zrange.dart'; Future main() async { - final data = DataFrame.fromCsv('datasets/iris.csv', - labelName: 'Species', - columns: [ZRange.closed(1, 5)], - categories: { - 'Species': CategoricalDataEncoderType.oneHot, - }, - ); - - final features = await data.features; - final labels = await data.labels; - final validator = CrossValidator.kFold(numberOfFolds: 5); - final accuracy = validator.evaluate((trainFeatures, trainLabels) => - LinearClassifier.softmaxRegressor( - trainFeatures, trainLabels, - initialLearningRate: 0.03, - iterationsLimit: null, - minWeightsUpdate: 1e-6, - randomSeed: 46, + final samples = await fromCsv('datasets/pima_indians_diabetes_database.csv', headerExists: true); + final targetColumnName = 'class variable (0 or 1)'; + final validator = CrossValidator.KFold(samples, [targetColumnName], numberOfFolds: 5); + final accuracy = validator.evaluate((samples, targetNames) => + LogisticRegressor( + samples, + targetNames[0], // remember, we provide a list of just a single name + optimizerType: LinearOptimizerType.gradient, + initialLearningRate: .8, + iterationsLimit: 500, + batchSize: 768, + fitIntercept: true, + interceptScale: .1, learningRateType: LearningRateType.constant - ), features, labels, MetricType.accuracy); + ), MetricType.accuracy); - print('Iris dataset, softmax regression: accuracy is ' - '${accuracy.toStringAsFixed(2)}'); + print('accuracy on classification: ${accuracy.toStringFixed(2)}'); } ```` @@ -303,39 +155,42 @@ state of the art dataset - [boston housing](https://www.kaggle.com/c/boston-hous As usual, import all necessary packages ````dart -import 'dart:async'; - import 'package:ml_algo/ml_algo.dart'; +import 'package:ml_dataframe/ml_dataframe.dart'; import 'package:ml_preprocessing/ml_preprocessing.dart'; -import 'package:xrange/zrange.dart'; ```` and download and read the data ````dart -final data = DataFrame.fromCsv('lib/_datasets/housing.csv', +final samples = await fromCsv('lib/_datasets/housing.csv', headerExists: false, fieldDelimiter: ' ', - labelIdx: 13, ); ```` -As you can see, the dataset is headless, that means, that there is no a descriptive line in the beginning of the file, -hence we can just use the index-based approach to point, where the outcomes column resides (13 index in our case) +As you can see, the dataset is headless, that means, that there is no a descriptive line in the beginning of the file. +So, we may use an autogenerated header in order to point, from what column we should take our target labels: + +```dart +print(samples.header); +``` + +It will output the following: +````dart +(col_0, col_1, col_2, col_3, col_4, col_5, col_6, col_7, col_8, col_9, col_10, col_11, col_12, col_13) +```` -Extract features and labels +Our target is `col_13`. Let's store it: ````dart -// As in example above, it's needed to normalize the matrix column-wise to reach computational stability and provide -// uniform scale for all the values in the column -final features = (await data.features).mapColumns((column) => column.normalize()); -final labels = await data.labels; +final targetColumnName = 'col_13'; ```` -Create a cross-validator instance +Let's create a cross-validator instance: ````dart -final validator = CrossValidator.kFold(numberOfFolds: 5); +final validator = CrossValidator.KFold(samples, [targetColumnName], numberOfFolds: 5); ```` Let the `k` parameter be equal to `4`. @@ -343,8 +198,8 @@ Let the `k` parameter be equal to `4`. Assess a knn regressor with the chosen `k` value using MAPE metric ````dart -final error = validator.evaluate((trainFeatures, trainLabels) => - ParameterlessRegressor.knn(trainFeatures, trainLabels, k: 4), features, labels, MetricType.mape); +final error = validator.evaluate((samples, targetNames) => + ParameterlessRegressor.knn(samples, targetNames[0], k: 4), MetricType.mape); ```` Let's print our error diff --git a/benchmark/cross_validator.dart b/benchmark/cross_validator.dart index 42c8460e..0c43c6bb 100644 --- a/benchmark/cross_validator.dart +++ b/benchmark/cross_validator.dart @@ -1,19 +1,16 @@ // 8.5 sec -import 'dart:async'; - import 'package:benchmark_harness/benchmark_harness.dart'; import 'package:ml_algo/ml_algo.dart'; +import 'package:ml_dataframe/ml_dataframe.dart'; import 'package:ml_linalg/matrix.dart'; import 'package:ml_linalg/vector.dart'; const observationsNum = 1000; -const featuresNum = 20; +const columnsNum = 21; class CrossValidatorBenchmark extends BenchmarkBase { CrossValidatorBenchmark() : super('Cross validator benchmark'); - Matrix features; - Matrix labels; CrossValidator crossValidator; static void main() { @@ -22,18 +19,20 @@ class CrossValidatorBenchmark extends BenchmarkBase { @override void run() { - crossValidator.evaluate((trainFeatures, trainLabels) => - ParameterlessRegressor.knn(trainFeatures, trainLabels, k: 7), - features, labels, MetricType.mape); + crossValidator.evaluate((trainSamples, targetFeatureNames) => + KnnRegressor(trainSamples, targetFeatureNames.first, k: 7), + MetricType.mape); } @override void setup() { - features = Matrix.fromRows(List.generate(observationsNum, - (i) => Vector.randomFilled(featuresNum))); - labels = Matrix.fromColumns([Vector.randomFilled(observationsNum)]); + final samples = Matrix.fromRows(List.generate(observationsNum, + (i) => Vector.randomFilled(columnsNum))); + + final dataFrame = DataFrame.fromMatrix(samples); - crossValidator = CrossValidator.kFold(numberOfFolds: 5); + crossValidator = CrossValidator.kFold(dataFrame, ['col_20'], + numberOfFolds: 5); } void tearDown() {} diff --git a/benchmark/gradient_descent_regression.dart b/benchmark/gradient_descent_regression.dart deleted file mode 100644 index 4aa343d4..00000000 --- a/benchmark/gradient_descent_regression.dart +++ /dev/null @@ -1,34 +0,0 @@ -import 'dart:async'; - -import 'package:benchmark_harness/benchmark_harness.dart'; -import 'package:ml_algo/ml_algo.dart'; -import 'package:ml_linalg/dtype.dart'; -import 'package:ml_linalg/matrix.dart'; -import 'package:ml_linalg/vector.dart'; - -const observationsNum = 200; -const featuresNum = 20; - -class GDRegressorBenchmark extends BenchmarkBase { - GDRegressorBenchmark() : super('Gradient descent regressor'); - - final Matrix features = Matrix.fromRows(List.generate(observationsNum, - (i) => Vector.randomFilled(featuresNum))); - - final Matrix labels = Matrix.fromColumns([Vector.randomFilled(observationsNum)]); - - static void main() { - GDRegressorBenchmark().report(); - } - - @override - void run() { - LinearRegressor.gradient(features, labels, dtype: DType.float32); - } - - void tearDown() {} -} - -Future main() async { - GDRegressorBenchmark.main(); -} diff --git a/benchmark/knn_regression.dart b/benchmark/knn_regressor.dart similarity index 61% rename from benchmark/knn_regression.dart rename to benchmark/knn_regressor.dart index 443b0023..7bf8ecc2 100644 --- a/benchmark/knn_regression.dart +++ b/benchmark/knn_regressor.dart @@ -1,8 +1,8 @@ -// 5.7 sec -import 'dart:async'; - +// 10.0 sec (MacBook Air mid 2017) import 'package:benchmark_harness/benchmark_harness.dart'; import 'package:ml_algo/ml_algo.dart'; +import 'package:ml_algo/src/regressor/knn_regressor_impl.dart'; +import 'package:ml_dataframe/ml_dataframe.dart'; import 'package:ml_linalg/matrix.dart'; import 'package:ml_linalg/vector.dart'; @@ -10,13 +10,13 @@ const observationsNum = 500; const featuresNum = 20; class KnnRegressorBenchmark extends BenchmarkBase { - KnnRegressorBenchmark() : super('KNN regression benchmark'); + KnnRegressorBenchmark() : super('Knn regression benchmark'); Matrix features; - Matrix testFeatures; + DataFrame testFeatures; Matrix labels; Matrix testLabels; - ParameterlessRegressor regressor; + KnnRegressor regressor; static void main() { @@ -34,11 +34,17 @@ class KnnRegressorBenchmark extends BenchmarkBase { (i) => Vector.randomFilled(featuresNum))); labels = Matrix.fromColumns([Vector.randomFilled(observationsNum * 2)]); - testFeatures = Matrix.fromRows(List.generate(observationsNum, - (i) => Vector.randomFilled(featuresNum))); + testFeatures = DataFrame.fromMatrix( + Matrix.fromRows( + List.generate( + observationsNum, + (i) => Vector.randomFilled(featuresNum), + ), + ), + ); testLabels = Matrix.fromColumns([Vector.randomFilled(observationsNum)]); - regressor = ParameterlessRegressor.knn(features, labels, k: 7); + regressor = KnnRegressorImpl(features, labels, 'target', k: 7); } void tearDown() {} diff --git a/benchmark/linear_regressor.dart b/benchmark/linear_regressor.dart new file mode 100644 index 00000000..b27d40df --- /dev/null +++ b/benchmark/linear_regressor.dart @@ -0,0 +1,44 @@ +import 'package:benchmark_harness/benchmark_harness.dart'; +import 'package:ml_algo/ml_algo.dart'; +import 'package:ml_dataframe/ml_dataframe.dart'; +import 'package:ml_linalg/matrix.dart'; +import 'package:ml_linalg/vector.dart'; + +const observationsNum = 200; +const featuresNum = 20; + +class LinearRegressorBenchmark extends BenchmarkBase { + LinearRegressorBenchmark() : super('Linear regressor'); + + DataFrame fittingData; + + static void main() { + LinearRegressorBenchmark().report(); + } + + @override + void run() { + LinearRegressor(fittingData, 'col_20'); + } + + @override + void setup() { + final features = Matrix.fromRows(List.generate(observationsNum, + (i) => Vector.randomFilled(featuresNum))); + + final labels = Matrix.fromColumns([Vector.randomFilled(observationsNum)]); + + fittingData = DataFrame.fromMatrix( + Matrix.fromColumns([ + ...features.columns, + ...labels.columns, + ]), + ); + } + + void tearDown() {} +} + +Future main() async { + LinearRegressorBenchmark.main(); +} diff --git a/benchmark/logistic_regression.dart b/benchmark/logistic_regression.dart deleted file mode 100644 index 9918656f..00000000 --- a/benchmark/logistic_regression.dart +++ /dev/null @@ -1,37 +0,0 @@ -import 'dart:async'; - -import 'package:benchmark_harness/benchmark_harness.dart'; -import 'package:ml_algo/src/classifier/linear/logistic_regressor/logistic_regressor.dart'; -import 'package:ml_linalg/dtype.dart'; -import 'package:ml_linalg/matrix.dart'; -import 'package:ml_linalg/vector.dart'; - -const observationsNum = 200; -const featuresNum = 20; - -class LogisticRegressorBenchmark extends BenchmarkBase { - LogisticRegressorBenchmark() : super('Logistic regressor'); - - final Matrix features = Matrix.fromRows(List.generate(observationsNum, - (i) => Vector.randomFilled(featuresNum))); - - final Matrix outcomes = Matrix.fromColumns([ - Vector.randomFilled(observationsNum), - ]); - - static void main() { - LogisticRegressorBenchmark().report(); - } - - @override - void run() { - LogisticRegressor.gradient(features, outcomes, - dtype: DType.float32, minWeightsUpdate: null, iterationsLimit: 200); - } - - void tearDown() {} -} - -Future main() async { - LogisticRegressorBenchmark.main(); -} diff --git a/benchmark/logistic_regressor.dart b/benchmark/logistic_regressor.dart new file mode 100644 index 00000000..a2b02519 --- /dev/null +++ b/benchmark/logistic_regressor.dart @@ -0,0 +1,42 @@ +import 'package:benchmark_harness/benchmark_harness.dart'; +import 'package:ml_algo/ml_algo.dart'; +import 'package:ml_dataframe/ml_dataframe.dart'; +import 'package:ml_linalg/matrix.dart'; +import 'package:ml_linalg/vector.dart'; + +const observationsNum = 200; +const columnsNum = 21; + +class LogisticRegressorBenchmark extends BenchmarkBase { + LogisticRegressorBenchmark() : super('Logistic regressor'); + + DataFrame _data; + + static void main() { + LogisticRegressorBenchmark().report(); + } + + @override + void run() { + LogisticRegressor( + _data, + 'col_20', + minCoefficientsUpdate: null, + iterationsLimit: 200, + ); + } + + @override + void setup() { + final Matrix observations = Matrix.fromRows(List.generate(observationsNum, + (i) => Vector.randomFilled(columnsNum))); + + _data = DataFrame.fromMatrix(observations); + } + + void tearDown() {} +} + +Future main() async { + LogisticRegressorBenchmark.main(); +} diff --git a/benchmark/main.dart b/benchmark/main.dart index 2aae7ef9..ddda6506 100644 --- a/benchmark/main.dart +++ b/benchmark/main.dart @@ -1,12 +1,10 @@ -import 'dart:async'; - -import 'gradient_descent_regression.dart' as gradientDescentRegressionBenchmark; -import 'logistic_regression.dart' as logisticRegressionBenchmark; -import 'algorithms/knn.dart' as knnBenchmark; +import 'linear_regressor.dart' as gradient_descent_regression_benchmark; +import 'logistic_regressor.dart' as logistic_regression_benchmark; +import 'algorithms/knn.dart' as knn_regressor_benchmark; Future main() async { // (MacBook Air mid 2017) - await gradientDescentRegressionBenchmark.main(); // 0.07 sec - await logisticRegressionBenchmark.main(); // 0.12 sec - await knnBenchmark.main(); // 5 sec + await gradient_descent_regression_benchmark.main(); // 0.07 sec + await logistic_regression_benchmark.main(); // 0.12 sec + await knn_regressor_benchmark.main(); // 5 sec } diff --git a/example/main.dart b/example/main.dart index b2454e2c..e29a72b7 100644 --- a/example/main.dart +++ b/example/main.dart @@ -1,29 +1,20 @@ -import 'dart:async'; - import 'package:ml_algo/ml_algo.dart'; -import 'package:ml_linalg/matrix.dart'; +import 'package:ml_dataframe/ml_dataframe.dart'; /// A simple usage example using synthetic data. To see more complex examples, /// please, visit other directories in this folder Future main() async { - // Let's create a feature matrix (a set of independent variables) - final features = Matrix.fromList([ - [2.0, 3.0, 4.0, 5.0], - [12.0, 32.0, 1.0, 3.0], - [27.0, 3.0, 0.0, 59.0], - ]); - - // Let's create dependent variables vector. It will be used as `true` values - // to adjust regression coefficients - final labels = Matrix.fromList([ - [4.3], - [3.5], - [2.1], - ]); + // Let's create a dataframe with fitting data, let's assume, that the target + // column is the fifth column (column with index 4) + final dataFrame = DataFrame(>[ + [ 2, 3, 4, 5, 4.3], + [12, 32, 1, 3, 3.5], + [27, 3, 0, 59, 2.1], + ], headerExists: false); // Let's create a regressor itself and train it - final regressor = LinearRegressor.gradient( - features, labels, + final regressor = LinearRegressor( + dataFrame, 'col_4', iterationsLimit: 100, initialLearningRate: 0.0005, learningRateType: LearningRateType.constant); diff --git a/lib/ml_algo.dart b/lib/ml_algo.dart index caea8351..bef50456 100644 --- a/lib/ml_algo.dart +++ b/lib/ml_algo.dart @@ -1,10 +1,13 @@ export 'package:ml_algo/src/algorithms/knn/kernel_type.dart'; -export 'package:ml_algo/src/classifier/linear/logistic_regressor/logistic_regressor.dart'; -export 'package:ml_algo/src/classifier/linear/softmax_regressor/softmax_regressor.dart'; +export 'package:ml_algo/src/classifier/decision_tree_classifier.dart'; +export 'package:ml_algo/src/classifier/logistic_regressor.dart'; +export 'package:ml_algo/src/classifier/softmax_regressor.dart'; +export 'package:ml_algo/src/linear_optimizer/gradient_optimizer/learning_rate_generator/learning_rate_type.dart'; +export 'package:ml_algo/src/linear_optimizer/linear_optimizer_type.dart'; +export 'package:ml_algo/src/linear_optimizer/regularization_type.dart'; export 'package:ml_algo/src/metric/classification/type.dart'; export 'package:ml_algo/src/metric/metric_type.dart'; export 'package:ml_algo/src/metric/regression/type.dart'; export 'package:ml_algo/src/model_selection/cross_validator/cross_validator.dart'; +export 'package:ml_algo/src/regressor/knn_regressor.dart'; export 'package:ml_algo/src/regressor/linear_regressor.dart'; -export 'package:ml_algo/src/regressor/parameterless_regressor.dart'; -export 'package:ml_algo/src/solver/linear/gradient/learning_rate_generator/learning_rate_type.dart'; diff --git a/lib/src/classifier/_helpers/log_likelihood_optimizer_factory.dart b/lib/src/classifier/_helpers/log_likelihood_optimizer_factory.dart new file mode 100644 index 00000000..1be314d3 --- /dev/null +++ b/lib/src/classifier/_helpers/log_likelihood_optimizer_factory.dart @@ -0,0 +1,82 @@ +import 'package:ml_algo/src/cost_function/cost_function_factory.dart'; +import 'package:ml_algo/src/cost_function/cost_function_type.dart'; +import 'package:ml_algo/src/di/injector.dart'; +import 'package:ml_algo/src/helpers/add_intercept_if.dart'; +import 'package:ml_algo/src/helpers/features_target_split.dart'; +import 'package:ml_algo/src/linear_optimizer/gradient_optimizer/learning_rate_generator/learning_rate_type.dart'; +import 'package:ml_algo/src/linear_optimizer/initial_coefficients_generator/initial_coefficients_type.dart'; +import 'package:ml_algo/src/linear_optimizer/linear_optimizer.dart'; +import 'package:ml_algo/src/linear_optimizer/linear_optimizer_factory.dart'; +import 'package:ml_algo/src/linear_optimizer/linear_optimizer_type.dart'; +import 'package:ml_algo/src/linear_optimizer/regularization_type.dart'; +import 'package:ml_algo/src/link_function/link_function_factory.dart'; +import 'package:ml_algo/src/link_function/link_function_type.dart'; +import 'package:ml_dataframe/ml_dataframe.dart'; +import 'package:ml_linalg/dtype.dart'; +import 'package:ml_linalg/matrix.dart'; + +LinearOptimizer createLogLikelihoodOptimizer( + DataFrame fittingData, + Iterable targetNames, + LinkFunctionType linkFunctionType, { + LinearOptimizerType optimizerType, + int iterationsLimit, + double initialLearningRate, + double minCoefficientsUpdate, + double probabilityThreshold, + double lambda, + RegularizationType regularizationType, + int randomSeed, + int batchSize, + bool fitIntercept, + double interceptScale, + bool isFittingDataNormalized, + LearningRateType learningRateType, + InitialCoefficientsType initialWeightsType, + Matrix initialWeights, + DType dtype, +}) { + final splits = featuresTargetSplit(fittingData, + targetNames: targetNames, + ).toList(); + + final points = splits[0].toMatrix(); + final labels = splits[1].toMatrix(); + + final dependencies = getDependencies(); + + final optimizerFactory = dependencies + .getDependency(); + + final linkFunctionFactory = dependencies + .getDependency(); + + final linkFunction = linkFunctionFactory + .createByType(linkFunctionType, dtype: dtype); + + final costFunctionFactory = dependencies + .getDependency(); + + final costFunction = costFunctionFactory.createByType( + CostFunctionType.logLikelihood, + linkFunction: linkFunction, + ); + + return optimizerFactory.createByType( + optimizerType, + addInterceptIf(fitIntercept, points, interceptScale), + labels, + costFunction: costFunction, + iterationLimit: iterationsLimit, + initialLearningRate: initialLearningRate, + minCoefficientsUpdate: minCoefficientsUpdate, + lambda: lambda, + regularizationType: regularizationType, + randomSeed: randomSeed, + batchSize: batchSize, + learningRateType: learningRateType, + initialCoefficientsType: initialWeightsType, + dtype: dtype, + isFittingDataNormalized: isFittingDataNormalized, + ); +} diff --git a/lib/src/classifier/_mixin/asessable_classifier_mixin.dart b/lib/src/classifier/_mixin/asessable_classifier_mixin.dart deleted file mode 100644 index e2e87de0..00000000 --- a/lib/src/classifier/_mixin/asessable_classifier_mixin.dart +++ /dev/null @@ -1,13 +0,0 @@ -import 'package:ml_algo/src/classifier/classifier.dart'; -import 'package:ml_algo/src/metric/factory.dart'; -import 'package:ml_algo/src/metric/metric_type.dart'; -import 'package:ml_algo/src/model_selection/assessable.dart'; -import 'package:ml_linalg/matrix.dart'; - -mixin AssessableClassifierMixin implements Assessable, Classifier { - @override - double assess(Matrix features, Matrix origLabels, MetricType metricType) { - final metric = MetricFactory.createByType(metricType); - return metric.getScore(predictClasses(features), origLabels); - } -} diff --git a/lib/src/classifier/_mixin/linear_classifier_mixin.dart b/lib/src/classifier/_mixin/linear_classifier_mixin.dart deleted file mode 100644 index 72324cb3..00000000 --- a/lib/src/classifier/_mixin/linear_classifier_mixin.dart +++ /dev/null @@ -1,19 +0,0 @@ -import 'package:ml_algo/src/classifier/linear/linear_classifier.dart'; -import 'package:ml_algo/src/helpers/add_intercept_if.dart'; -import 'package:ml_algo/src/helpers/get_probabilities.dart'; -import 'package:ml_algo/src/link_function/link_function.dart'; -import 'package:ml_linalg/matrix.dart'; - -mixin LinearClassifierMixin implements LinearClassifier { - LinkFunction get linkFunction; - bool get fitIntercept; - double get interceptScale; - - @override - Matrix predictProbabilities(Matrix features) { - final processedFeatures = addInterceptIf(fitIntercept, features, - interceptScale); - return getProbabilities(processedFeatures, - coefficientsByClasses, linkFunction); - } -} diff --git a/lib/src/classifier/_mixins/linear_classifier_mixin.dart b/lib/src/classifier/_mixins/linear_classifier_mixin.dart new file mode 100644 index 00000000..4da421bf --- /dev/null +++ b/lib/src/classifier/_mixins/linear_classifier_mixin.dart @@ -0,0 +1,26 @@ +import 'package:ml_algo/src/classifier/linear_classifier.dart'; +import 'package:ml_algo/src/helpers/add_intercept_if.dart'; +import 'package:ml_algo/src/helpers/get_probabilities.dart'; +import 'package:ml_dataframe/ml_dataframe.dart'; + +mixin LinearClassifierMixin implements LinearClassifier { + @override + DataFrame predictProbabilities(DataFrame features) { + final processedFeatures = addInterceptIf( + fitIntercept, + features.toMatrix(), + interceptScale, + ); + + final probabilities = getProbabilities( + processedFeatures, + coefficientsByClasses, + linkFunction, + ); + + return DataFrame.fromMatrix( + probabilities, + header: classNames, + ); + } +} diff --git a/lib/src/classifier/classifier.dart b/lib/src/classifier/classifier.dart index e9592d0c..df827014 100644 --- a/lib/src/classifier/classifier.dart +++ b/lib/src/classifier/classifier.dart @@ -1,16 +1,12 @@ -import 'package:ml_linalg/matrix.dart'; +import 'package:ml_algo/src/predictor/predictor.dart'; +import 'package:ml_dataframe/ml_dataframe.dart'; /// An interface for any classifier (linear, non-linear, parametric, /// non-parametric, etc.) -abstract class Classifier { - /// A collection of encoded class labels - Matrix get classLabels; +abstract class Classifier extends Predictor { + List get classNames; /// Returns predicted distribution of probabilities for each observation in /// the passed [features] - Matrix predictProbabilities(Matrix features); - - /// Return a collection of predicted class labels for each observation in the - /// passed [features] - Matrix predictClasses(Matrix features); + DataFrame predictProbabilities(DataFrame features); } diff --git a/lib/src/classifier/decision_tree_classifier.dart b/lib/src/classifier/decision_tree_classifier.dart new file mode 100644 index 00000000..eefca305 --- /dev/null +++ b/lib/src/classifier/decision_tree_classifier.dart @@ -0,0 +1,52 @@ +import 'package:ml_algo/src/classifier/classifier.dart'; +import 'package:ml_algo/src/classifier/decision_tree_classifier_impl.dart'; +import 'package:ml_algo/src/decision_tree_solver/solver_factory/greedy_solver.dart'; +import 'package:ml_algo/src/model_selection/assessable.dart'; +import 'package:ml_dataframe/ml_dataframe.dart'; + +/// A class that performs decision tree-based classification +/// +/// Decision tree is an algorithm that recursively splits the input data into +/// subsets until the subsets conforming certain stop criteria are found. +/// +/// Process of obtaining such a recursive subsets structure is called +/// decision tree learning. Once a decision tree learned, it may use to classify +/// new samples with the same features, as were used to learn the tree. +abstract class DecisionTreeClassifier implements Classifier, Assessable { + /// Parameters: + /// + /// [fittingData] A [DataFrame] with observations, that will be used by the + /// classifier to learn a decision tree. Must contain [targetName] column. + /// + /// [targetName] A name of a column in [fittingData] that contains class + /// labels + /// + /// [minError] A value from range 0..1 (both inclusive). The value is used as + /// a stop criteria to avoid farther decision's tree node splitting: if the + /// node is good enough, there is no need to split it and thus it will become + /// a leaf. + /// + /// [minSamplesCount] A minimal number of samples (observations) on the + /// decision's tree node. The value is used as a stop criteria to avoid + /// farther decision's tree node splitting: if the node contains less than or + /// equal to [minSamplesCount] observations, the node turns into the leaf. + /// + /// [maxDepth] A maximum number of decision tree levels. + factory DecisionTreeClassifier( + DataFrame fittingData, + String targetName, { + double minError, + int minSamplesCount, + int maxDepth, + }) { + final solver = createGreedySolver( + fittingData, + targetName, + minError, + minSamplesCount, + maxDepth, + ); + + return DecisionTreeClassifierImpl(solver, targetName); + } +} diff --git a/lib/src/classifier/decision_tree_classifier_impl.dart b/lib/src/classifier/decision_tree_classifier_impl.dart new file mode 100644 index 00000000..de5fa8e0 --- /dev/null +++ b/lib/src/classifier/decision_tree_classifier_impl.dart @@ -0,0 +1,59 @@ +import 'package:ml_algo/src/classifier/decision_tree_classifier.dart'; +import 'package:ml_algo/src/predictor/assessable_predictor_mixin.dart'; +import 'package:ml_algo/src/decision_tree_solver/decision_tree_solver.dart'; +import 'package:ml_dataframe/ml_dataframe.dart'; +import 'package:ml_linalg/matrix.dart'; +import 'package:ml_linalg/vector.dart'; + +class DecisionTreeClassifierImpl with AssessablePredictorMixin + implements DecisionTreeClassifier { + + DecisionTreeClassifierImpl(this._solver, String className) + : classNames = [className]; + + final DecisionTreeSolver _solver; + + @override + final List classNames; + + @override + DataFrame predict(DataFrame features) { + final predictedLabels = features + .toMatrix() + .rows + .map(_solver.getLabelForSample); + + if (predictedLabels.isEmpty) { + return DataFrame([[]]); + } + + final outcomeList = predictedLabels + .map((label) => label.value) + .toList(growable: false); + final outcomeVector = Vector.fromList(outcomeList); + + return DataFrame.fromMatrix( + Matrix.fromColumns([outcomeVector]), + header: classNames, + ); + } + + @override + DataFrame predictProbabilities(DataFrame features) { + final probabilities = Matrix.fromColumns([ + Vector.fromList( + features + .toMatrix() + .rows + .map(_solver.getLabelForSample) + .map((label) => label.probability) + .toList(growable: false), + ), + ]); + + return DataFrame.fromMatrix( + probabilities, + header: classNames, + ); + } +} diff --git a/lib/src/classifier/linear/linear_classifier.dart b/lib/src/classifier/linear/linear_classifier.dart deleted file mode 100644 index be6ea091..00000000 --- a/lib/src/classifier/linear/linear_classifier.dart +++ /dev/null @@ -1,8 +0,0 @@ -import 'package:ml_algo/src/classifier/classifier.dart'; -import 'package:ml_linalg/matrix.dart'; - -abstract class LinearClassifier implements Classifier { - /// A matrix, where each column is a vector of coefficients, associated with - /// the specific class - Matrix get coefficientsByClasses; -} \ No newline at end of file diff --git a/lib/src/classifier/linear/logistic_regressor/gradient_logistic_regressor.dart b/lib/src/classifier/linear/logistic_regressor/gradient_logistic_regressor.dart deleted file mode 100644 index 829ef1d5..00000000 --- a/lib/src/classifier/linear/logistic_regressor/gradient_logistic_regressor.dart +++ /dev/null @@ -1,95 +0,0 @@ -import 'package:ml_algo/src/classifier/linear/logistic_regressor/logistic_regressor.dart'; -import 'package:ml_algo/src/classifier/_mixin/asessable_classifier_mixin.dart'; -import 'package:ml_algo/src/classifier/_mixin/linear_classifier_mixin.dart'; -import 'package:ml_algo/src/cost_function/log_likelihood.dart'; -import 'package:ml_algo/src/helpers/add_intercept_if.dart'; -import 'package:ml_algo/src/helpers/get_probabilities.dart'; -import 'package:ml_algo/src/link_function/link_function.dart'; -import 'package:ml_algo/src/link_function/logit/inverse_logit_link_function.dart'; -import 'package:ml_algo/src/solver/linear/gradient/learning_rate_generator/learning_rate_type.dart'; -import 'package:ml_algo/src/solver/linear/initial_weights_generator/initial_weights_type.dart'; -import 'package:ml_algo/src/solver/linear/linear_optimizer_factory.dart'; -import 'package:ml_algo/src/solver/linear/linear_optimizer_factory_impl.dart'; -import 'package:ml_algo/src/utils/default_parameter_values.dart'; -import 'package:ml_linalg/dtype.dart'; -import 'package:ml_linalg/matrix.dart'; -import 'package:ml_linalg/vector.dart'; - -class GradientLogisticRegressor with LinearClassifierMixin, - AssessableClassifierMixin implements LogisticRegressor { - - GradientLogisticRegressor( - Matrix trainingFeatures, - Matrix trainingOutcomes, { - // public arguments - int iterationsLimit = DefaultParameterValues.iterationsLimit, - double initialLearningRate = DefaultParameterValues.initialLearningRate, - double minWeightsUpdate = DefaultParameterValues.minCoefficientsUpdate, - double lambda, - int randomSeed, - int batchSize = 1, - bool fitIntercept = false, - double interceptScale = 1.0, - LearningRateType learningRateType = LearningRateType.constant, - InitialWeightsType initialWeightsType = InitialWeightsType.zeroes, - Matrix initialWeights, - this.dtype = DefaultParameterValues.dtype, - this.probabilityThreshold = 0.5, - - LinearOptimizerFactory optimizerFactory = - const LinearOptimizerFactoryImpl(), - }) : - fitIntercept = fitIntercept, - interceptScale = interceptScale, - linkFunction = InverseLogitLinkFunction(dtype), - classLabels = trainingOutcomes.uniqueRows(), - coefficientsByClasses = optimizerFactory.gradient( - addInterceptIf(fitIntercept, trainingFeatures, interceptScale), - trainingOutcomes, - dtype: dtype, - costFunction: LogLikelihoodCost(InverseLogitLinkFunction(dtype)), - learningRateType: learningRateType, - initialWeightsType: initialWeightsType, - initialLearningRate: initialLearningRate, - minCoefficientsUpdate: minWeightsUpdate, - iterationLimit: iterationsLimit, - lambda: lambda, - batchSize: batchSize, - randomSeed: randomSeed, - ).findExtrema( - initialWeights: initialWeights, - isMinimizingObjective: false, - ); - - @override - final Matrix coefficientsByClasses; - - @override - final Matrix classLabels; - - @override - final bool fitIntercept; - - @override - final double interceptScale; - - final DType dtype; - - final double probabilityThreshold; - - @override - final LinkFunction linkFunction; - - @override - Matrix predictClasses(Matrix features) { - final processedFeatures = addInterceptIf(fitIntercept, features, - interceptScale); - final classesSource = getProbabilities(processedFeatures, - coefficientsByClasses, linkFunction) - .getColumn(0) - // TODO: use SIMD - .map((value) => value >= probabilityThreshold ? 1.0 : 0.0) - .toList(growable: false); - return Matrix.fromColumns([Vector.fromList(classesSource)]); - } -} diff --git a/lib/src/classifier/linear/logistic_regressor/logistic_regressor.dart b/lib/src/classifier/linear/logistic_regressor/logistic_regressor.dart deleted file mode 100644 index 216f3318..00000000 --- a/lib/src/classifier/linear/logistic_regressor/logistic_regressor.dart +++ /dev/null @@ -1,123 +0,0 @@ -import 'package:ml_algo/src/classifier/linear/logistic_regressor/gradient_logistic_regressor.dart'; -import 'package:ml_algo/src/classifier/linear/linear_classifier.dart'; -import 'package:ml_algo/src/model_selection/assessable.dart'; -import 'package:ml_algo/src/solver/linear/gradient/learning_rate_generator/learning_rate_type.dart'; -import 'package:ml_linalg/dtype.dart'; -import 'package:ml_linalg/matrix.dart'; - -/// A factory that creates different presets of logistic regressors -/// -/// Logistic regression is an algorithm that solves a binary classification -/// problem. The algorithm uses maximization of the passed data likelihood. -/// In other words, the regressor iteratively tries to select coefficients, -/// that makes combination of passed features and these coefficients most -/// likely. -abstract class LogisticRegressor implements LinearClassifier, Assessable { - /// Parameters: - /// - /// [trainingFeatures] A matrix with observations, that will be used by the - /// classifier to learn coefficients of the hyperplane, which divides the - /// features space, forming classes of the features - /// - /// [trainingOutcomes] A matrix with outcomes (class labels, or dependant - /// variables) for each observation from [trainingFeatures] - /// - /// [iterationsLimit] A number of fitting iterations. Uses as a condition of - /// convergence in the solver. Default value is 100 - /// - /// [initialLearningRate] A value, defining velocity of the convergence of the - /// gradient descent solver. Default value is 1e-3 - /// - /// [minWeightsUpdate] A minimum distance between weights vectors in two - /// subsequent iterations. Uses as a condition of convergence in the - /// solver. In other words, if difference is small, there is no reason to - /// continue fitting. Default value is 1e-12 - /// - /// [probabilityThreshold] A probability, on the basis of which it is decided, - /// whether an observation relates to positive class label (label = 1) or - /// negative class label (label = 0). The greater the probability, the more - /// strict the classifier is. Default value is `0.5` - /// - /// [lambda] A coefficient of regularization. In gradient version of - /// logistic regression L2 regularization is used - /// - /// [randomSeed] A seed, that will be passed to a random value generator, - /// used by stochastic optimizers. Will be ignored, if the solver is not - /// a stochastic. Remember, each time you run the regressor based on, for - /// instance, stochastic gradient descent, with the same parameters, you will - /// receive a different result. To avoid it, define [randomSeed] - /// - /// [batchSize] A size of data (in rows), that will be used for fitting per - /// one iteration. If [batchSize] == `1` when stochastic gradient descent is - /// used; if `1` < [batchSize] < `total number of rows`, when mini-batch - /// gradient descent is used; if [batchSize] == `total number of rows`, - /// when full-batch gradient descent is used - /// - /// [fitIntercept] Whether or not to fit intercept term. Default value is - /// `false`. - /// - /// [interceptScale] A value, defining a size of the intercept term - /// - /// [learningRateType] A value, defining a strategy for the learning rate - /// behaviour throughout the whole fitting process. - /// - /// [dtype] A data type for all the numeric values, used by the algorithm. Can - /// affect performance or accuracy of the computations. Default value is - /// [DType.float32] - factory LogisticRegressor.gradient( - Matrix trainingFeatures, - Matrix trainingOutcomes, { - int iterationsLimit, - double initialLearningRate, - double minWeightsUpdate, - double probabilityThreshold, - double lambda, - int randomSeed, - int batchSize, - bool fitIntercept, - double interceptScale, - LearningRateType learningRateType, - Matrix initialWeights, - DType dtype, - }) = GradientLogisticRegressor; - - /// Creates a logistic regressor classifier based on coordinate descent - /// - /// Parameters: - /// - /// [trainingFeatures] A matrix with observations, that will be used by the - /// classifier to learn coefficients of the hyperplane, which divides the - /// features space, forming classes of the features - /// - /// [trainingOutcomes] A matrix with outcomes (class labels, or dependant - /// variables) for each observation from [trainingFeatures] - /// - /// [iterationsLimit] A number of fitting iterations. Uses as a condition of - /// convergence in the [solver]. Default value is 100 - /// - /// [minWeightsUpdate] A minimum distance between weights vectors in two - /// subsequent iterations. Uses as a condition of convergence in the - /// - /// [lambda] A coefficient of regularization. In coordinate descent version - /// of the classifier L1 regularization is used - /// - /// [fitIntercept] Whether or not to fit intercept term. Default value is - /// `false`. - /// - /// [interceptScale] A value, defining a size of the intercept term - /// - /// [dtype] A data type for all the numeric values, used by the algorithm. Can - /// affect performance or accuracy of the computations. Default value is - /// [DType.float32] - factory LogisticRegressor.coordinate( - Matrix trainingFeatures, - Matrix trainingOutcomes, { - int iterationsLimit, - double minWeightsUpdate, - double lambda, - bool fitIntercept, - double interceptScale, - Matrix initialWeights, - DType dtype, - }) => throw UnimplementedError(); -} diff --git a/lib/src/classifier/linear/softmax_regressor/gradient_softmax_regressor.dart b/lib/src/classifier/linear/softmax_regressor/gradient_softmax_regressor.dart deleted file mode 100644 index e077dc30..00000000 --- a/lib/src/classifier/linear/softmax_regressor/gradient_softmax_regressor.dart +++ /dev/null @@ -1,91 +0,0 @@ -import 'package:ml_algo/src/classifier/_mixin/asessable_classifier_mixin.dart'; -import 'package:ml_algo/src/classifier/_mixin/linear_classifier_mixin.dart'; -import 'package:ml_algo/src/classifier/linear/softmax_regressor/softmax_regressor.dart'; -import 'package:ml_algo/src/cost_function/log_likelihood.dart'; -import 'package:ml_algo/src/helpers/add_intercept_if.dart'; -import 'package:ml_algo/src/helpers/get_probabilities.dart'; -import 'package:ml_algo/src/link_function/link_function.dart'; -import 'package:ml_algo/src/link_function/softmax/softmax_link_function.dart'; -import 'package:ml_algo/src/solver/linear/gradient/learning_rate_generator/learning_rate_type.dart'; -import 'package:ml_algo/src/solver/linear/initial_weights_generator/initial_weights_type.dart'; -import 'package:ml_algo/src/solver/linear/linear_optimizer_factory.dart'; -import 'package:ml_algo/src/solver/linear/linear_optimizer_factory_impl.dart'; -import 'package:ml_algo/src/utils/default_parameter_values.dart'; -import 'package:ml_linalg/dtype.dart'; -import 'package:ml_linalg/matrix.dart'; - -class GradientSoftmaxRegressor with LinearClassifierMixin, - AssessableClassifierMixin implements SoftmaxRegressor { - - GradientSoftmaxRegressor( - Matrix trainingFeatures, - Matrix trainingOutcomes, { - // public arguments - int iterationsLimit = DefaultParameterValues.iterationsLimit, - double initialLearningRate = DefaultParameterValues.initialLearningRate, - double minWeightsUpdate = DefaultParameterValues.minCoefficientsUpdate, - double lambda, - int randomSeed, - int batchSize = 1, - bool fitIntercept = false, - double interceptScale = 1.0, - Matrix initialWeights, - LearningRateType learningRateType = LearningRateType.constant, - InitialWeightsType initialWeightsType = InitialWeightsType.zeroes, - - this.dtype = DefaultParameterValues.dtype, - - LinearOptimizerFactory optimizerFactory = - const LinearOptimizerFactoryImpl(), - }) : - fitIntercept = fitIntercept, - interceptScale = interceptScale, - linkFunction = SoftmaxLinkFunction(dtype), - classLabels = trainingOutcomes.uniqueRows(), - coefficientsByClasses = optimizerFactory.gradient( - addInterceptIf(fitIntercept, trainingFeatures, interceptScale), - trainingOutcomes, - dtype: dtype, - costFunction: LogLikelihoodCost(SoftmaxLinkFunction(dtype)), - learningRateType: learningRateType, - initialWeightsType: initialWeightsType, - initialLearningRate: initialLearningRate, - minCoefficientsUpdate: minWeightsUpdate, - iterationLimit: iterationsLimit, - lambda: lambda, - batchSize: batchSize, - randomSeed: randomSeed, - ).findExtrema( - initialWeights: initialWeights, - isMinimizingObjective: false, - ); - - @override - final bool fitIntercept; - - @override - final double interceptScale; - - @override - final Matrix classLabels; - - @override - final Matrix coefficientsByClasses; - - final DType dtype; - - @override - final LinkFunction linkFunction; - - @override - Matrix predictClasses(Matrix features) { - final processedFeatures = addInterceptIf(fitIntercept, features, - interceptScale); - return getProbabilities(processedFeatures, coefficientsByClasses, - linkFunction) - .mapRows((probabilities) { - final labelIdx = probabilities.toList().indexOf(probabilities.max()); - return classLabels.getRow(labelIdx); - }); - } -} diff --git a/lib/src/classifier/linear/softmax_regressor/softmax_regressor.dart b/lib/src/classifier/linear/softmax_regressor/softmax_regressor.dart deleted file mode 100644 index a9d49659..00000000 --- a/lib/src/classifier/linear/softmax_regressor/softmax_regressor.dart +++ /dev/null @@ -1,86 +0,0 @@ -import 'package:ml_algo/src/classifier/linear/softmax_regressor/gradient_softmax_regressor.dart'; -import 'package:ml_algo/src/classifier/linear/linear_classifier.dart'; -import 'package:ml_algo/src/model_selection/assessable.dart'; -import 'package:ml_algo/src/solver/linear/gradient/learning_rate_generator/learning_rate_type.dart'; -import 'package:ml_linalg/dtype.dart'; -import 'package:ml_linalg/matrix.dart'; - -/// A factory that creates different presets of softmax regressors -/// -/// Softmax regression is an algorithm that solves a multiclass classification -/// problem. The algorithm uses maximization of the passed -/// data likelihood (as well as -/// [Logistic regression](https://en.wikipedia.org/wiki/Logistic_regression). -/// In other words, the regressor iteratively tries to select coefficients, -/// that makes combination of passed features and these coefficients most -/// likely. But, instead of [Logit link function](https://en.wikipedia.org/wiki/Logit) -/// it uses [Softmax link function](https://en.wikipedia.org/wiki/Softmax_function), -/// that's why the algorithm has such a name. -/// -/// Also, it is worth to mention, that the algorithm is a generalization of -/// [Logistic regression](https://en.wikipedia.org/wiki/Logistic_regression)) -abstract class SoftmaxRegressor implements LinearClassifier, Assessable { - /// Creates a gradient descent based softmax regressor classifier. - /// - /// Parameters: - /// - /// [trainingFeatures] A matrix with observations, that will be used by the - /// classifier to learn coefficients of the hyperplane, which divides the - /// features space, forming classes of the features - /// - /// [trainingOutcomes] A matrix with outcomes (class labels, or dependant - /// variables) for each observation from [trainingFeatures] - /// - /// [iterationsLimit] A number of fitting iterations. Uses as a condition of - /// convergence in the [solver]. Default value is 100 - /// - /// [initialLearningRate] A value, defining velocity of the convergence of the - /// gradient descent solver. Default value is 1e-3 - /// - /// [minWeightsUpdate] A minimum distance between weights vectors in two - /// subsequent iterations. Uses as a condition of convergence in the - /// [solver]. In other words, if difference is small, there is no reason to - /// continue fitting. Default value is 1e-12 - /// - /// [lambda] A coefficient of regularization. In gradient version of softmax - /// regression L2 regularisation is used. - /// - /// [randomSeed] A seed, that will be passed to a random value generator, - /// used by stochastic optimizers. Will be ignored, if the [solver] is not - /// a stochastic. Remember, each time you run the regressor based on, for - /// instance, stochastic gradient descent, with the same parameters, you will - /// receive a different result. To avoid it, define [randomSeed] - /// - /// [batchSize] A size of data (in rows), that will be used for fitting per - /// one iteration. If [batchSize] == `1` when stochastic gradient descent is - /// used; if `1` < [batchSize] < `total number of rows`, when mini-batch - /// gradient descent is used; if [batchSize] == `total number of rows`, - /// when full-batch gradient descent is used - /// - /// [fitIntercept] Whether or not to fit intercept term. Default value is - /// `false`. - /// - /// [interceptScale] A value, defining a size of the intercept term - /// - /// [learningRateType] A value, defining a strategy for the learning rate - /// behaviour throughout the whole fitting process. - /// - /// [dtype] A data type for all the numeric values, used by the algorithm. Can - /// affect performance or accuracy of the computations. Default value is - /// [DType.float32] - factory SoftmaxRegressor.gradient( - Matrix trainingFeatures, - Matrix trainingOutcomes, { - int iterationsLimit, - double initialLearningRate, - double minWeightsUpdate, - double lambda, - int randomSeed, - int batchSize, - bool fitIntercept, - double interceptScale, - LearningRateType learningRateType, - Matrix initialWeights, - DType dtype, - }) = GradientSoftmaxRegressor; -} diff --git a/lib/src/classifier/linear_classifier.dart b/lib/src/classifier/linear_classifier.dart new file mode 100644 index 00000000..89bef0c0 --- /dev/null +++ b/lib/src/classifier/linear_classifier.dart @@ -0,0 +1,22 @@ +import 'package:ml_algo/src/classifier/classifier.dart'; +import 'package:ml_algo/src/link_function/link_function.dart'; +import 'package:ml_linalg/matrix.dart'; + +/// An interface for all types of linear classifiers +abstract class LinearClassifier implements Classifier { + /// A function that is used for converting learned coefficients into + /// probabilities + LinkFunction get linkFunction; + + /// A flag, that denotes, whether the intercept term considered during + /// learning of the classifier or not + bool get fitIntercept; + + /// A value, that defines a size of the intercept, if [fitIntercept] is + /// `true` + double get interceptScale; + + /// A matrix, where each column is a vector of coefficients, associated with + /// the specific class + Matrix get coefficientsByClasses; +} \ No newline at end of file diff --git a/lib/src/classifier/logistic_regressor.dart b/lib/src/classifier/logistic_regressor.dart new file mode 100644 index 00000000..6d29ca3d --- /dev/null +++ b/lib/src/classifier/logistic_regressor.dart @@ -0,0 +1,183 @@ +import 'package:ml_algo/src/classifier/_helpers/log_likelihood_optimizer_factory.dart'; +import 'package:ml_algo/src/classifier/linear_classifier.dart'; +import 'package:ml_algo/src/classifier/logistic_regressor_impl.dart'; +import 'package:ml_algo/src/di/injector.dart'; +import 'package:ml_algo/src/linear_optimizer/gradient_optimizer/learning_rate_generator/learning_rate_type.dart'; +import 'package:ml_algo/src/linear_optimizer/initial_coefficients_generator/initial_coefficients_type.dart'; +import 'package:ml_algo/src/linear_optimizer/linear_optimizer_type.dart'; +import 'package:ml_algo/src/linear_optimizer/regularization_type.dart'; +import 'package:ml_algo/src/link_function/link_function_factory.dart'; +import 'package:ml_algo/src/link_function/link_function_type.dart'; +import 'package:ml_algo/src/model_selection/assessable.dart'; +import 'package:ml_dataframe/ml_dataframe.dart'; +import 'package:ml_linalg/dtype.dart'; +import 'package:ml_linalg/vector.dart'; + +/// A class, performing logistic regression-based classification. +/// +/// Logistic regression is an algorithm that solves a binary classification +/// problem. The algorithm uses maximization of the passed data likelihood. +/// In other words, the regressor iteratively tries to select coefficients, +/// that makes combination of passed features and their coefficients most +/// likely. +abstract class LogisticRegressor implements LinearClassifier, Assessable { + /// Parameters: + /// + /// [fittingData] A [DataFrame] with observations, that will be used by the + /// classifier to learn coefficients of the hyperplane, which divides the + /// features space, forming clusters of positive and negative classes of the + /// observations. Must contain [targetName] column. + /// + /// [targetName] A string, that serves as a name of the target column (a + /// column, that contains class labels or outcomes for the associated + /// features). + /// + /// [optimizerType] Defines an algorithm of optimization, that will be used + /// to find the best coefficients of log-likelihood cost function. Also + /// defines, which regularization type (L1 or L2) one may use to learn a + /// logistic regressor. + /// + /// [iterationsLimit] A number of fitting iterations. Uses as a condition of + /// convergence in the optimization algorithm. Default value is `100`. + /// + /// [initialLearningRate] A value, defining velocity of the convergence of the + /// gradient descent optimizer. Default value is `1e-3`. + /// + /// [minCoefficientsUpdate] A minimum distance between coefficient vectors in + /// two contiguous iterations. Uses as a condition of convergence in the + /// optimization algorithm. If difference between the two vectors is small + /// enough, there is no reason to continue fitting. Default value is `1e-12` + /// + /// [probabilityThreshold] A probability, on the basis of which it is decided, + /// whether an observation relates to positive class label (see + /// [positiveLabel] parameter) or to negative class label (see [negativeLabel] + /// parameter). The greater the probability, the more strict the classifier + /// is. Default value is `0.5`. + /// + /// [lambda] A coefficient of regularization. Uses to prevent the classifier's + /// overfitting. The more the value of [lambda], the more regular the + /// coefficients of log-likelihood cost function are. Extremely large [lambda] + /// may decrease the coefficients to nothing, otherwise too small [lambda] may + /// be a cause of too large absolute values of the coefficients. + /// + /// [regularizationType] A way the coefficients of the classifier will be + /// regularized to prevent a model overfitting. + /// + /// [randomSeed] A seed, that will be passed to a random value generator, + /// used by stochastic optimizers. Will be ignored, if the solver is not + /// a stochastic. Remember, each time you run the regressor based on, for + /// instance, stochastic gradient descent, with the same parameters, you will + /// receive a different result. To avoid it, define [randomSeed] + /// + /// [batchSize] A size of data (in rows), that will be used for fitting per + /// one iteration. Applicable not for all optimizers. If gradient-based + /// optimizer uses and If [batchSize] == `1`, stochastic mode will be + /// activated; if `1` < [batchSize] < `total number of rows`, mini-batch mode + /// will be activated; if [batchSize] == `total number of rows`, full-batch + /// mode will be activated. + /// + /// [fitIntercept] Whether or not to fit intercept term. Default value is + /// `false`. Intercept in 2-dimensional space is a bias of the line (relative + /// to X-axis) to be learned by the classifier + /// + /// [interceptScale] A value, defining a size of the intercept. + /// + /// [isFittingDataNormalized] Defines, whether the [fittingData] normalized + /// or not. Normalization should be performed column-wise. Normalized data + /// may be needed for some optimizers (e.g., for + /// [LinearOptimizerType.coordinate]) + /// + /// [learningRateType] A value, defining a strategy for the learning rate + /// behaviour throughout the whole fitting process. + /// + /// [initialCoefficientsType] Defines the coefficients, that will be + /// autogenerated before the first iteration of optimization. By default, + /// all the autogenerated coefficients are equal to zeroes at the start. + /// If [initialCoefficients] are provided, the parameter will be ignored + /// + /// [initialCoefficients] Coefficients to be used in the first iteration of + /// optimization algorithm. [initialCoefficients] is a vector, length of which + /// must be equal to the number of features in [fittingData] : in case of + /// logistic regression only one column from [fittingData] is used as a + /// prediction target column, thus the number of features is equal to + /// the number of columns in [fittingData] minus 1 (target column). + /// + /// [positiveLabel] Defines the value, that will be used for `positive` class. + /// By default, `1`. + /// + /// [negativeLabel] Defines the value, that will be used for `negative` class. + /// By default, `0`. + /// + /// [dtype] A data type for all the numeric values, used by the algorithm. Can + /// affect performance or accuracy of the computations. Default value is + /// [DType.float32] + factory LogisticRegressor( + DataFrame fittingData, + String targetName, { + LinearOptimizerType optimizerType = LinearOptimizerType.gradient, + int iterationsLimit = 100, + double initialLearningRate = 1e-3, + double minCoefficientsUpdate = 1e-12, + double probabilityThreshold = 0.5, + double lambda = 0.0, + RegularizationType regularizationType, + int randomSeed, + int batchSize = 1, + bool fitIntercept = false, + double interceptScale = 1.0, + bool isFittingDataNormalized = false, + LearningRateType learningRateType = LearningRateType.constant, + InitialCoefficientsType initialCoefficientsType = + InitialCoefficientsType.zeroes, + Vector initialCoefficients, + num positiveLabel = 1, + num negativeLabel = 0, + DType dtype = DType.float32, + }) { + if (fittingData[targetName] == null) { + throw Exception('Target column with name $targetName does not exist in ' + 'the fitting data. All existing columns: ${fittingData.header}'); + } + + final dependencies = getDependencies(); + + final linkFunctionFactory = dependencies + .getDependency(); + + final linkFunction = linkFunctionFactory + .createByType(LinkFunctionType.inverseLogit, dtype: dtype); + + final optimizer = createLogLikelihoodOptimizer( + fittingData, + [targetName], + LinkFunctionType.inverseLogit, + optimizerType: optimizerType, + iterationsLimit: iterationsLimit, + initialLearningRate: initialLearningRate, + minCoefficientsUpdate: minCoefficientsUpdate, + lambda: lambda, + regularizationType: regularizationType, + randomSeed: randomSeed, + batchSize: batchSize, + learningRateType: learningRateType, + initialWeightsType: initialCoefficientsType, + fitIntercept: fitIntercept, + interceptScale: interceptScale, + isFittingDataNormalized: isFittingDataNormalized, + dtype: dtype, + ); + + return LogisticRegressorImpl( + optimizer, + targetName, + linkFunction, + probabilityThreshold: probabilityThreshold, + fitIntercept: fitIntercept, + interceptScale: interceptScale, + initialCoefficients: initialCoefficients, + positiveLabel: positiveLabel, + negativeLabel: negativeLabel, + dtype: dtype, + ); + } +} diff --git a/lib/src/classifier/logistic_regressor_impl.dart b/lib/src/classifier/logistic_regressor_impl.dart new file mode 100644 index 00000000..61f44ced --- /dev/null +++ b/lib/src/classifier/logistic_regressor_impl.dart @@ -0,0 +1,93 @@ +import 'package:ml_algo/src/classifier/_mixins/linear_classifier_mixin.dart'; +import 'package:ml_algo/src/classifier/logistic_regressor.dart'; +import 'package:ml_algo/src/helpers/add_intercept_if.dart'; +import 'package:ml_algo/src/helpers/get_probabilities.dart'; +import 'package:ml_algo/src/link_function/link_function.dart'; +import 'package:ml_algo/src/predictor/assessable_predictor_mixin.dart'; +import 'package:ml_algo/src/linear_optimizer/linear_optimizer.dart'; +import 'package:ml_dataframe/ml_dataframe.dart'; +import 'package:ml_linalg/dtype.dart'; +import 'package:ml_linalg/matrix.dart'; +import 'package:ml_linalg/vector.dart'; + +class LogisticRegressorImpl with LinearClassifierMixin, + AssessablePredictorMixin implements LogisticRegressor { + + LogisticRegressorImpl( + LinearOptimizer _optimizer, + String className, + this.linkFunction, { + bool fitIntercept = false, + double interceptScale = 1.0, + Vector initialCoefficients, + this.dtype = DType.float32, + this.probabilityThreshold = 0.5, + num negativeLabel = 0, + num positiveLabel = 1, + }) : + classNames = [className], + fitIntercept = fitIntercept, + interceptScale = interceptScale, + _negativeLabel = negativeLabel, + _positiveLabel = positiveLabel, + coefficientsByClasses = _optimizer.findExtrema( + initialCoefficients: initialCoefficients != null + ? Matrix.fromColumns([initialCoefficients], dtype: dtype) + : null, + isMinimizingObjective: false, + ); + + @override + final Matrix coefficientsByClasses; + + @override + final List classNames; + + @override + final bool fitIntercept; + + @override + final double interceptScale; + + final DType dtype; + + final double probabilityThreshold; + + final num _positiveLabel; + + final num _negativeLabel; + + @override + final LinkFunction linkFunction; + + @override + DataFrame predict(DataFrame features) { + final processedFeatures = addInterceptIf( + fitIntercept, + features.toMatrix(), + interceptScale, + ); + + final classesList = getProbabilities( + processedFeatures, + coefficientsByClasses, + linkFunction, + ) + .getColumn(0) + // TODO: use SIMD + .map((value) => value >= probabilityThreshold + ? _positiveLabel + : _negativeLabel, + ) + .toList(growable: false); + + final classesMatrix = Matrix.fromColumns([ + Vector.fromList(classesList), + ]); + + return DataFrame.fromMatrix( + classesMatrix, + header: classNames, + ); + } +} diff --git a/lib/src/classifier/non_linear/decision_tree/decision_tree_classifier.dart b/lib/src/classifier/non_linear/decision_tree/decision_tree_classifier.dart deleted file mode 100644 index 735dfbc7..00000000 --- a/lib/src/classifier/non_linear/decision_tree/decision_tree_classifier.dart +++ /dev/null @@ -1,36 +0,0 @@ -import 'package:ml_algo/src/classifier/classifier.dart'; -import 'package:ml_algo/src/classifier/non_linear/decision_tree/decision_tree_classifier_impl.dart'; -import 'package:ml_algo/src/model_selection/assessable.dart'; -import 'package:ml_algo/src/solver/non_linear/decision_tree/solver_factory/greedy_solver.dart'; -import 'package:ml_dataframe/ml_dataframe.dart'; - -/// A factory that creates different presets of decision tree classifier -/// -/// Decision tree is an algorithm that recursively splits the input data into -/// subsets until the bests possible data subsets will be found. -abstract class DecisionTreeClassifier implements Classifier, Assessable { - /// Creates majority based decision tree classifier. - /// - /// Majority based decision tree - a simplest tree algorithm, that uses - /// majority label on a tree node as a prediction. - factory DecisionTreeClassifier.majority( - DataFrame samples, - { - int targetId, - String targetName, - double minError, - int minSamplesCount, - int maxDepth, - } - ) { - final solver = createGreedySolver( - samples, - targetId, - targetName, - minError, - minSamplesCount, - maxDepth, - ); - return DecisionTreeClassifierImpl(solver); - } -} diff --git a/lib/src/classifier/non_linear/decision_tree/decision_tree_classifier_impl.dart b/lib/src/classifier/non_linear/decision_tree/decision_tree_classifier_impl.dart deleted file mode 100644 index 3614adb0..00000000 --- a/lib/src/classifier/non_linear/decision_tree/decision_tree_classifier_impl.dart +++ /dev/null @@ -1,42 +0,0 @@ -import 'package:ml_algo/src/classifier/_mixin/asessable_classifier_mixin.dart'; -import 'package:ml_algo/src/classifier/non_linear/decision_tree/decision_tree_classifier.dart'; -import 'package:ml_algo/src/solver/non_linear/decision_tree/decision_tree_solver.dart'; -import 'package:ml_linalg/matrix.dart'; -import 'package:ml_linalg/vector.dart'; - -class DecisionTreeClassifierImpl with AssessableClassifierMixin - implements DecisionTreeClassifier { - - DecisionTreeClassifierImpl(this._solver); - - final DecisionTreeSolver _solver; - - @override - Matrix get classLabels => null; - - @override - Matrix predictClasses(Matrix features) { - final predictedLabels = features.rows.map(_solver.getLabelForSample); - - if (predictedLabels.isEmpty) { - return Matrix.fromColumns([]); - } - - final outcomeList = predictedLabels - .map((label) => label.value) - .toList(growable: false); - final outcomeVector = Vector.fromList(outcomeList); - - return Matrix.fromColumns([outcomeVector]); - } - - @override - Matrix predictProbabilities(Matrix features) => Matrix.fromColumns([ - Vector.fromList( - features.rows - .map(_solver.getLabelForSample) - .map((label) => label.probability) - .toList(growable: false), - ), - ]); -} diff --git a/lib/src/classifier/softmax_regressor.dart b/lib/src/classifier/softmax_regressor.dart new file mode 100644 index 00000000..dbd8b3bd --- /dev/null +++ b/lib/src/classifier/softmax_regressor.dart @@ -0,0 +1,191 @@ +import 'package:ml_algo/src/classifier/_helpers/log_likelihood_optimizer_factory.dart'; +import 'package:ml_algo/src/classifier/linear_classifier.dart'; +import 'package:ml_algo/src/classifier/softmax_regressor_impl.dart'; +import 'package:ml_algo/src/di/injector.dart'; +import 'package:ml_algo/src/linear_optimizer/gradient_optimizer/learning_rate_generator/learning_rate_type.dart'; +import 'package:ml_algo/src/linear_optimizer/initial_coefficients_generator/initial_coefficients_type.dart'; +import 'package:ml_algo/src/linear_optimizer/linear_optimizer_type.dart'; +import 'package:ml_algo/src/linear_optimizer/regularization_type.dart'; +import 'package:ml_algo/src/link_function/link_function_factory.dart'; +import 'package:ml_algo/src/link_function/link_function_type.dart'; +import 'package:ml_algo/src/model_selection/assessable.dart'; +import 'package:ml_dataframe/ml_dataframe.dart'; +import 'package:ml_linalg/dtype.dart'; +import 'package:ml_linalg/matrix.dart'; + +/// A class that performs softmax regression-based classification +/// +/// Softmax regression is an algorithm that solves a multiclass classification +/// problem. The algorithm uses maximization of the passed +/// data likelihood (as well as +/// [Logistic regression](https://en.wikipedia.org/wiki/Logistic_regression). +/// In other words, the regressor iteratively tries to select coefficients, +/// that makes combination of passed features and their coefficients most +/// likely. But, instead of [Logit link function](https://en.wikipedia.org/wiki/Logit) +/// it uses [Softmax link function](https://en.wikipedia.org/wiki/Softmax_function), +/// that's why the algorithm has such a name. +/// +/// Also, it is worth to mention, that the algorithm is a generalization of +/// [Logistic regression](https://en.wikipedia.org/wiki/Logistic_regression)) +abstract class SoftmaxRegressor implements LinearClassifier, Assessable { + /// Parameters: + /// + /// [fittingData] A [DataFrame] with observations, that will be used by the + /// classifier to learn coefficients. Must contain [targetNames] columns. + /// + /// [targetNames] A collection of strings, that serves as names for the + /// encoded target columns (a column, that contains class labels or outcomes + /// for the associated features). + /// + /// [optimizerType] Defines an algorithm of optimization, that will be used + /// to find the best coefficients of log-likelihood cost function. Also + /// defines, which regularization type (L1 or L2) one may use to learn a + /// logistic regressor. + /// + /// [iterationsLimit] A number of fitting iterations. Uses as a condition of + /// convergence in the optimization algorithm. Default value is `100`. + /// + /// [initialLearningRate] A value, defining velocity of the convergence of the + /// gradient descent optimizer. Default value is `1e-3`. + /// + /// [minCoefficientsUpdate] A minimum distance between coefficient vectors in + /// two contiguous iterations. Uses as a condition of convergence in the + /// optimization algorithm. If difference between the two vectors is small + /// enough, there is no reason to continue fitting. Default value is `1e-12`. + /// + /// [lambda] A coefficient of regularization. Uses to prevent the classifier's + /// overfitting. The more the value of [lambda], the more regular the + /// coefficients of log-likelihood cost function are. Extremely large [lambda] + /// may decrease the coefficients to nothing, otherwise too small [lambda] may + /// be a cause of too large absolute values of the coefficients. + /// + /// [regularizationType] A way the coefficients of the classifier will be + /// regularized to prevent a model overfitting. + /// + /// [randomSeed] A seed, that will be passed to a random value generator, + /// used by stochastic optimizers. Will be ignored, if the [solver] is not + /// a stochastic. Remember, each time you run the regressor based on, for + /// instance, stochastic gradient descent, with the same parameters, you will + /// receive a different result. To avoid it, define [randomSeed]. + /// + /// [batchSize] A size of data (in rows), that will be used for fitting per + /// one iteration. Applicable not for all optimizers. If gradient-based + /// optimizer uses and If [batchSize] == `1`, stochastic mode will be + /// activated; if `1` < [batchSize] < `total number of rows`, mini-batch mode + /// will be activated; if [batchSize] == `total number of rows`, full-batch + /// mode will be activated. + /// + /// [fitIntercept] Whether or not to fit intercept term. Default value is + /// `false`. Intercept in 2-dimensional space is a bias of the line (relative + /// to X-axis) to be learned by the classifier. + /// + /// [interceptScale] A value, defining a size of the intercept. + /// + /// [learningRateType] A value, defining a strategy for the learning rate + /// behaviour throughout the whole fitting process. + /// + /// [isFittingDataNormalized] Defines, whether the [fittingData] normalized + /// or not. Normalization should be performed column-wise. Normalized data + /// may be needed for some optimizers (e.g., for + /// [LinearOptimizerType.coordinate]) + /// + /// [initialCoefficientsType] Defines the coefficients, that will be + /// autogenerated before the first iteration of optimization. By default, + /// all the autogenerated coefficients are equal to zeroes at the start. + /// If [initialCoefficients] are provided, the parameter will be ignored + /// + /// [initialCoefficients] Coefficients to be used in the first iteration of + /// an optimization algorithm. [initialCoefficients] is a [Matrix], where the + /// number of columns must be equal to the number of classes (or + /// length of [targetNames]) and the number of rows must be equal to the + /// number of features in [fittingData]. In other words, every column of + /// [initialCoefficients] matrix is a vector of coefficients of a certain + /// class. + /// + /// [positiveLabel] Defines the value, that will be used for `positive` class. + /// By default, `1`. + /// + /// [negativeLabel] Defines the value, that will be used for `negative` class. + /// By default, `0`. + /// + /// [dtype] A data type for all the numeric values, used by the algorithm. Can + /// affect performance or accuracy of the computations. Default value is + /// [DType.float32] + factory SoftmaxRegressor( + DataFrame fittingData, + List targetNames, { + LinearOptimizerType optimizerType = LinearOptimizerType.gradient, + int iterationsLimit = 100, + double initialLearningRate = 1e-3, + double minCoefficientsUpdate = 1e-12, + double lambda, + RegularizationType regularizationType, + int randomSeed, + int batchSize = 1, + bool fitIntercept = false, + double interceptScale = 1.0, + LearningRateType learningRateType, + bool isFittingDataNormalized, + InitialCoefficientsType initialCoefficientsType = + InitialCoefficientsType.zeroes, + Matrix initialCoefficients, + num positiveLabel = 1, + num negativeLabel = 0, + DType dtype = DType.float32, + }) { + if (targetNames.isNotEmpty && targetNames.length < 2) { + throw Exception('The target column should be encoded properly ' + '(e.g., via one-hot encoder)'); + } + + if (fittingData.series + .where((series) => targetNames.contains(series.name)) + .isEmpty + ) { + throw Exception('Target columns with names $targetNames do not ' + 'exist in the fitting data. All the existing columns: ' + '${fittingData.header}'); + } + + final dependencies = getDependencies(); + + final linkFunctionFactory = dependencies + .getDependency(); + + final linkFunction = linkFunctionFactory + .createByType(LinkFunctionType.softmax, dtype: dtype); + + final optimizer = createLogLikelihoodOptimizer( + fittingData, + targetNames, + LinkFunctionType.softmax, + optimizerType: optimizerType, + iterationsLimit: iterationsLimit, + initialLearningRate: initialLearningRate, + minCoefficientsUpdate: minCoefficientsUpdate, + lambda: lambda, + regularizationType: regularizationType, + randomSeed: randomSeed, + batchSize: batchSize, + learningRateType: learningRateType, + initialWeightsType: initialCoefficientsType, + fitIntercept: fitIntercept, + interceptScale: interceptScale, + isFittingDataNormalized: isFittingDataNormalized, + dtype: dtype, + ); + + return SoftmaxRegressorImpl( + optimizer, + targetNames, + linkFunction, + batchSize: batchSize, + fitIntercept: fitIntercept, + interceptScale: interceptScale, + initialCoefficients: initialCoefficients, + positiveLabel: positiveLabel, + negativeLabel: negativeLabel, + dtype: dtype, + ); + } +} diff --git a/lib/src/classifier/softmax_regressor_impl.dart b/lib/src/classifier/softmax_regressor_impl.dart new file mode 100644 index 00000000..e5040c39 --- /dev/null +++ b/lib/src/classifier/softmax_regressor_impl.dart @@ -0,0 +1,90 @@ +import 'package:ml_algo/src/classifier/_mixins/linear_classifier_mixin.dart'; +import 'package:ml_algo/src/classifier/softmax_regressor.dart'; +import 'package:ml_algo/src/helpers/add_intercept_if.dart'; +import 'package:ml_algo/src/helpers/get_probabilities.dart'; +import 'package:ml_algo/src/linear_optimizer/linear_optimizer.dart'; +import 'package:ml_algo/src/link_function/link_function.dart'; +import 'package:ml_algo/src/predictor/assessable_predictor_mixin.dart'; +import 'package:ml_dataframe/ml_dataframe.dart'; +import 'package:ml_linalg/dtype.dart'; +import 'package:ml_linalg/linalg.dart'; +import 'package:ml_linalg/matrix.dart'; + +class SoftmaxRegressorImpl with LinearClassifierMixin, + AssessablePredictorMixin implements SoftmaxRegressor { + + SoftmaxRegressorImpl( + LinearOptimizer optimizer, + this.classNames, + this.linkFunction, { + int batchSize = 1, + bool fitIntercept = false, + double interceptScale = 1.0, + Matrix initialCoefficients, + num positiveLabel = 1, + num negativeLabel = 0, + this.dtype = DType.float32, + }) : + fitIntercept = fitIntercept, + interceptScale = interceptScale, + _positiveLabel = positiveLabel, + _negativeLabel = negativeLabel, + coefficientsByClasses = optimizer.findExtrema( + initialCoefficients: initialCoefficients, + isMinimizingObjective: false, + ); + + @override + final List classNames; + + @override + final bool fitIntercept; + + @override + final double interceptScale; + + @override + final Matrix coefficientsByClasses; + + final DType dtype; + + @override + final LinkFunction linkFunction; + + final num _positiveLabel; + + final num _negativeLabel; + + @override + DataFrame predict(DataFrame features) { + final processedFeatures = addInterceptIf( + fitIntercept, + features.toMatrix(), + interceptScale, + ); + + final classes = getProbabilities( + processedFeatures, + coefficientsByClasses, + linkFunction + ).mapRows((probabilities) { + final labelIdx = probabilities + .toList() + .indexOf(probabilities.max()); + + final allZeroes = List.filled( + coefficientsByClasses.columnsNum, + _negativeLabel, + ); + + allZeroes[labelIdx] = _positiveLabel; + + return Vector.fromList(allZeroes, dtype: dtype); + }); + + return DataFrame.fromMatrix( + classes, + header: classNames, + ); + } +} diff --git a/lib/src/common/serializable/serializable.dart b/lib/src/common/serializable/serializable.dart index a316a39a..485e60e2 100644 --- a/lib/src/common/serializable/serializable.dart +++ b/lib/src/common/serializable/serializable.dart @@ -1,6 +1,9 @@ import 'dart:io'; abstract class Serializable { + /// Returns a serializable object Map serialize(); + + /// Saves a json file in [fileName] file Future saveAsJSON(String fileName); } diff --git a/lib/src/cost_function/cost_function.dart b/lib/src/cost_function/cost_function.dart index 0bd9d4da..6d319a7f 100644 --- a/lib/src/cost_function/cost_function.dart +++ b/lib/src/cost_function/cost_function.dart @@ -4,5 +4,5 @@ import 'package:ml_linalg/vector.dart'; abstract class CostFunction { double getCost(double predictedLabel, double originalLabel); Matrix getGradient(Matrix x, Matrix w, Matrix y); - Vector getSubDerivative(int j, Matrix x, Matrix w, Matrix y); + Vector getSubGradient(int j, Matrix x, Matrix w, Matrix y); } diff --git a/lib/src/cost_function/cost_function_factory.dart b/lib/src/cost_function/cost_function_factory.dart new file mode 100644 index 00000000..1943da11 --- /dev/null +++ b/lib/src/cost_function/cost_function_factory.dart @@ -0,0 +1,9 @@ +import 'package:ml_algo/src/cost_function/cost_function.dart'; +import 'package:ml_algo/src/cost_function/cost_function_type.dart'; +import 'package:ml_algo/src/link_function/link_function.dart'; + +abstract class CostFunctionFactory { + CostFunction createByType(CostFunctionType type, { + LinkFunction linkFunction, + }); +} diff --git a/lib/src/cost_function/cost_function_factory_impl.dart b/lib/src/cost_function/cost_function_factory_impl.dart new file mode 100644 index 00000000..e8938c8a --- /dev/null +++ b/lib/src/cost_function/cost_function_factory_impl.dart @@ -0,0 +1,30 @@ +import 'package:ml_algo/src/cost_function/cost_function.dart'; +import 'package:ml_algo/src/cost_function/cost_function_factory.dart'; +import 'package:ml_algo/src/cost_function/cost_function_type.dart'; +import 'package:ml_algo/src/cost_function/log_likelihood.dart'; +import 'package:ml_algo/src/cost_function/squared.dart'; +import 'package:ml_algo/src/link_function/link_function.dart'; + +class CostFunctionFactoryImpl implements CostFunctionFactory { + const CostFunctionFactoryImpl(); + + @override + CostFunction createByType(CostFunctionType type, { + LinkFunction linkFunction, + }) { + switch (type) { + case CostFunctionType.logLikelihood: + if (linkFunction == null) { + throw Exception('Link function must be specified if log likelihood ' + 'cost function is going to be used'); + } + return LogLikelihoodCost(linkFunction); + + case CostFunctionType.squared: + return const SquaredCost(); + + default: + throw UnsupportedError('Unsupported cost function type - $type'); + } + } +} diff --git a/lib/src/cost_function/cost_function_type.dart b/lib/src/cost_function/cost_function_type.dart new file mode 100644 index 00000000..54d15426 --- /dev/null +++ b/lib/src/cost_function/cost_function_type.dart @@ -0,0 +1,8 @@ +/// A types of cost functions, which are used by different linear optimizers. +enum CostFunctionType { + /// A logarithmic form of likelihood function. + logLikelihood, + + /// A squared difference between actual and predicted values. + squared, +} diff --git a/lib/src/cost_function/log_likelihood.dart b/lib/src/cost_function/log_likelihood.dart index ec00dab9..ff11e0a8 100644 --- a/lib/src/cost_function/log_likelihood.dart +++ b/lib/src/cost_function/log_likelihood.dart @@ -17,6 +17,6 @@ class LogLikelihoodCost implements CostFunction { x.transpose() * (y - _linkFunction.link(x * w)); @override - Vector getSubDerivative(int wIdx, Matrix x, Matrix w, Matrix y) => + Vector getSubGradient(int wIdx, Matrix x, Matrix w, Matrix y) => throw UnimplementedError(); } diff --git a/lib/src/cost_function/squared.dart b/lib/src/cost_function/squared.dart index e01a5602..6aae8bb3 100644 --- a/lib/src/cost_function/squared.dart +++ b/lib/src/cost_function/squared.dart @@ -2,7 +2,7 @@ import 'dart:math' as math; import 'package:ml_algo/src/cost_function/cost_function.dart'; import 'package:ml_linalg/linalg.dart'; -import 'package:xrange/zrange.dart'; +import 'package:xrange/integers.dart'; class SquaredCost implements CostFunction { const SquaredCost(); @@ -16,9 +16,8 @@ class SquaredCost implements CostFunction { x.transpose() * -2 * (y - x * w); @override - Vector getSubDerivative(int j, Matrix x, Matrix w, Matrix y) { - final xj = x.pick(columnRanges: List.generate(y.columnsNum, - (i) => ZRange.singleton(j))); + Vector getSubGradient(int j, Matrix x, Matrix w, Matrix y) { + final xj = x.sample(columnIndices: List.generate(y.columnsNum, (_) => j)); final xWithoutJ = _excludeColumn(x, j); final wWithoutJ = _excludeColumn(w, j); final predictionWithoutJ = xWithoutJ * wWithoutJ.transpose(); @@ -28,13 +27,14 @@ class SquaredCost implements CostFunction { Matrix _excludeColumn(Matrix x, int column) { if (column == 0) { - return x.submatrix(columns: ZRange.closedOpen(1, x.columnsNum)); + return x.sample(columnIndices: integers(1, x.columnsNum, + upperClosed: false)); } else if (column == x.columnsNum - 1) { - return x.submatrix(columns: ZRange.closedOpen(0, column)); + return x.sample(columnIndices: integers(0, column, upperClosed: false)); } else { - return x.pick(columnRanges: [ - ZRange.closedOpen(0, column), - ZRange.closedOpen(column + 1, x.columnsNum) + return x.sample(columnIndices: [ + ...integers(0, column, upperClosed: false), + ...integers(column + 1, x.columnsNum, upperClosed: false), ]); } } diff --git a/lib/src/solver/non_linear/decision_tree/decision_tree_leaf_label.dart b/lib/src/decision_tree_solver/decision_tree_leaf_label.dart similarity index 100% rename from lib/src/solver/non_linear/decision_tree/decision_tree_leaf_label.dart rename to lib/src/decision_tree_solver/decision_tree_leaf_label.dart diff --git a/lib/src/solver/non_linear/decision_tree/decision_tree_node.dart b/lib/src/decision_tree_solver/decision_tree_node.dart similarity index 90% rename from lib/src/solver/non_linear/decision_tree/decision_tree_node.dart rename to lib/src/decision_tree_solver/decision_tree_node.dart index a5144f11..7aba3d23 100644 --- a/lib/src/solver/non_linear/decision_tree/decision_tree_node.dart +++ b/lib/src/decision_tree_solver/decision_tree_node.dart @@ -1,5 +1,5 @@ import 'package:ml_algo/src/common/serializable/serializable_mixin.dart'; -import 'package:ml_algo/src/solver/non_linear/decision_tree/decision_tree_leaf_label.dart'; +import 'package:ml_algo/src/decision_tree_solver/decision_tree_leaf_label.dart'; import 'package:ml_linalg/vector.dart'; typedef TestSamplePredicate = bool Function(Vector sample); diff --git a/lib/src/solver/non_linear/decision_tree/decision_tree_solver.dart b/lib/src/decision_tree_solver/decision_tree_solver.dart similarity index 75% rename from lib/src/solver/non_linear/decision_tree/decision_tree_solver.dart rename to lib/src/decision_tree_solver/decision_tree_solver.dart index e9babe20..154389b8 100644 --- a/lib/src/solver/non_linear/decision_tree/decision_tree_solver.dart +++ b/lib/src/decision_tree_solver/decision_tree_solver.dart @@ -1,27 +1,27 @@ -import 'package:ml_algo/src/solver/non_linear/decision_tree/decision_tree_leaf_label.dart'; -import 'package:ml_algo/src/solver/non_linear/decision_tree/decision_tree_node.dart'; -import 'package:ml_algo/src/solver/non_linear/decision_tree/leaf_detector/leaf_detector.dart'; -import 'package:ml_algo/src/solver/non_linear/decision_tree/leaf_label_factory/leaf_label_factory.dart'; -import 'package:ml_algo/src/solver/non_linear/decision_tree/split_selector/split_selector.dart'; +import 'package:ml_algo/src/decision_tree_solver/decision_tree_leaf_label.dart'; +import 'package:ml_algo/src/decision_tree_solver/decision_tree_node.dart'; +import 'package:ml_algo/src/decision_tree_solver/leaf_detector/leaf_detector.dart'; +import 'package:ml_algo/src/decision_tree_solver/leaf_label_factory/leaf_label_factory.dart'; +import 'package:ml_algo/src/decision_tree_solver/split_selector/split_selector.dart'; import 'package:ml_linalg/matrix.dart'; import 'package:ml_linalg/vector.dart'; class DecisionTreeSolver { DecisionTreeSolver( Matrix samples, - this._featuresColumnIdxs, + this._featureIndices, this._targetIdx, - this._colIdxToUniqueValues, + this._featureToUniqueValues, this._leafDetector, this._leafLabelFactory, this._splitSelector, ) { - _root = _createNode(samples, null, null, null, _featuresColumnIdxs, 0); + _root = _createNode(samples, null, null, null, _featureIndices, 0); } - final Iterable _featuresColumnIdxs; + final Iterable _featureIndices; final int _targetIdx; - final Map> _colIdxToUniqueValues; + final Map> _featureToUniqueValues; final LeafDetector _leafDetector; final DecisionTreeLeafLabelFactory _leafLabelFactory; final SplitSelector _splitSelector; @@ -56,7 +56,7 @@ class DecisionTreeSolver { samples, _targetIdx, featuresColumnIdxs, - _colIdxToUniqueValues, + _featureToUniqueValues, ); final newLevel = level + 1; @@ -65,7 +65,7 @@ class DecisionTreeSolver { final splitNode = entry.key; final splitSamples = entry.value; - final isSplitByNominalValue = _colIdxToUniqueValues + final isSplitByNominalValue = _featureToUniqueValues .containsKey(splitNode.splittingIdx); final updatedColumnRanges = isSplitByNominalValue diff --git a/lib/src/decision_tree_solver/decision_tree_solver_type.dart b/lib/src/decision_tree_solver/decision_tree_solver_type.dart new file mode 100644 index 00000000..e69de29b diff --git a/lib/src/solver/non_linear/decision_tree/leaf_detector/leaf_detector.dart b/lib/src/decision_tree_solver/leaf_detector/leaf_detector.dart similarity index 82% rename from lib/src/solver/non_linear/decision_tree/leaf_detector/leaf_detector.dart rename to lib/src/decision_tree_solver/leaf_detector/leaf_detector.dart index 95c91a26..5fb87b41 100644 --- a/lib/src/solver/non_linear/decision_tree/leaf_detector/leaf_detector.dart +++ b/lib/src/decision_tree_solver/leaf_detector/leaf_detector.dart @@ -1,5 +1,4 @@ import 'package:ml_linalg/matrix.dart'; -import 'package:xrange/zrange.dart'; abstract class LeafDetector { bool isLeaf(Matrix sample, int targetIdx, diff --git a/lib/src/solver/non_linear/decision_tree/leaf_detector/leaf_detector_impl.dart b/lib/src/decision_tree_solver/leaf_detector/leaf_detector_impl.dart similarity index 81% rename from lib/src/solver/non_linear/decision_tree/leaf_detector/leaf_detector_impl.dart rename to lib/src/decision_tree_solver/leaf_detector/leaf_detector_impl.dart index c67a589f..da975f77 100644 --- a/lib/src/solver/non_linear/decision_tree/leaf_detector/leaf_detector_impl.dart +++ b/lib/src/decision_tree_solver/leaf_detector/leaf_detector_impl.dart @@ -1,5 +1,5 @@ -import 'package:ml_algo/src/solver/non_linear/decision_tree/leaf_detector/leaf_detector.dart'; -import 'package:ml_algo/src/solver/non_linear/decision_tree/split_assessor/split_assessor.dart'; +import 'package:ml_algo/src/decision_tree_solver/leaf_detector/leaf_detector.dart'; +import 'package:ml_algo/src/decision_tree_solver/split_assessor/split_assessor.dart'; import 'package:ml_linalg/matrix.dart'; class LeafDetectorImpl implements LeafDetector { diff --git a/lib/src/solver/non_linear/decision_tree/leaf_label_factory/leaf_label_factory.dart b/lib/src/decision_tree_solver/leaf_label_factory/leaf_label_factory.dart similarity index 62% rename from lib/src/solver/non_linear/decision_tree/leaf_label_factory/leaf_label_factory.dart rename to lib/src/decision_tree_solver/leaf_label_factory/leaf_label_factory.dart index 6b372a45..3275913c 100644 --- a/lib/src/solver/non_linear/decision_tree/leaf_label_factory/leaf_label_factory.dart +++ b/lib/src/decision_tree_solver/leaf_label_factory/leaf_label_factory.dart @@ -1,4 +1,4 @@ -import 'package:ml_algo/src/solver/non_linear/decision_tree/decision_tree_leaf_label.dart'; +import 'package:ml_algo/src/decision_tree_solver/decision_tree_leaf_label.dart'; import 'package:ml_linalg/matrix.dart'; abstract class DecisionTreeLeafLabelFactory { diff --git a/lib/src/solver/non_linear/decision_tree/leaf_label_factory/majority_leaf_label_factory.dart b/lib/src/decision_tree_solver/leaf_label_factory/majority_leaf_label_factory.dart similarity index 88% rename from lib/src/solver/non_linear/decision_tree/leaf_label_factory/majority_leaf_label_factory.dart rename to lib/src/decision_tree_solver/leaf_label_factory/majority_leaf_label_factory.dart index 80f8cf92..211ae87f 100644 --- a/lib/src/solver/non_linear/decision_tree/leaf_label_factory/majority_leaf_label_factory.dart +++ b/lib/src/decision_tree_solver/leaf_label_factory/majority_leaf_label_factory.dart @@ -1,8 +1,8 @@ import 'dart:collection'; import 'package:ml_algo/src/common/sequence_elements_distribution_calculator/distribution_calculator.dart'; -import 'package:ml_algo/src/solver/non_linear/decision_tree/decision_tree_leaf_label.dart'; -import 'package:ml_algo/src/solver/non_linear/decision_tree/leaf_label_factory/leaf_label_factory.dart'; +import 'package:ml_algo/src/decision_tree_solver/decision_tree_leaf_label.dart'; +import 'package:ml_algo/src/decision_tree_solver/leaf_label_factory/leaf_label_factory.dart'; import 'package:ml_linalg/matrix.dart'; import 'package:quiver/iterables.dart' as quiver_iterables; diff --git a/lib/src/solver/non_linear/decision_tree/solver_factory/greedy_solver.dart b/lib/src/decision_tree_solver/solver_factory/greedy_solver.dart similarity index 64% rename from lib/src/solver/non_linear/decision_tree/solver_factory/greedy_solver.dart rename to lib/src/decision_tree_solver/solver_factory/greedy_solver.dart index 7ca21e4b..336ce74a 100644 --- a/lib/src/solver/non_linear/decision_tree/solver_factory/greedy_solver.dart +++ b/lib/src/decision_tree_solver/solver_factory/greedy_solver.dart @@ -1,18 +1,17 @@ import 'package:ml_algo/src/common/sequence_elements_distribution_calculator/distribution_calculator_impl.dart'; -import 'package:ml_algo/src/solver/non_linear/decision_tree/decision_tree_solver.dart'; -import 'package:ml_algo/src/solver/non_linear/decision_tree/leaf_detector/leaf_detector_impl.dart'; -import 'package:ml_algo/src/solver/non_linear/decision_tree/leaf_label_factory/majority_leaf_label_factory.dart'; -import 'package:ml_algo/src/solver/non_linear/decision_tree/split_assessor/majority_split_assessor.dart'; -import 'package:ml_algo/src/solver/non_linear/decision_tree/split_selector/greedy_split_selector.dart'; -import 'package:ml_algo/src/solver/non_linear/decision_tree/splitter/greedy_splitter.dart'; -import 'package:ml_algo/src/solver/non_linear/decision_tree/splitter/nominal_splitter/nominal_splitter_impl.dart'; -import 'package:ml_algo/src/solver/non_linear/decision_tree/splitter/numerical_splitter/numerical_splitter_impl.dart'; +import 'package:ml_algo/src/decision_tree_solver/decision_tree_solver.dart'; +import 'package:ml_algo/src/decision_tree_solver/leaf_detector/leaf_detector_impl.dart'; +import 'package:ml_algo/src/decision_tree_solver/leaf_label_factory/majority_leaf_label_factory.dart'; +import 'package:ml_algo/src/decision_tree_solver/split_assessor/majority_split_assessor.dart'; +import 'package:ml_algo/src/decision_tree_solver/split_selector/greedy_split_selector.dart'; +import 'package:ml_algo/src/decision_tree_solver/splitter/greedy_splitter.dart'; +import 'package:ml_algo/src/decision_tree_solver/splitter/nominal_splitter/nominal_splitter_impl.dart'; +import 'package:ml_algo/src/decision_tree_solver/splitter/numerical_splitter/numerical_splitter_impl.dart'; import 'package:ml_dataframe/ml_dataframe.dart'; import 'package:quiver/iterables.dart'; DecisionTreeSolver createGreedySolver( DataFrame samples, - int targetIdx, String targetName, double minErrorOnNode, int minSamplesCountOnNode, @@ -27,6 +26,10 @@ DecisionTreeSolver createGreedySolver( final distributionCalculator = const SequenceElementsDistributionCalculatorImpl(); + final targetIdx = enumerate(samples.header) + .firstWhere((indexedName) => indexedName.value == targetName) + .index; + final featuresIndexedSeries = enumerate(samples.series) .where((indexed) => indexed.index != targetIdx); diff --git a/lib/src/solver/non_linear/decision_tree/split_assessor/majority_split_assessor.dart b/lib/src/decision_tree_solver/split_assessor/majority_split_assessor.dart similarity index 94% rename from lib/src/solver/non_linear/decision_tree/split_assessor/majority_split_assessor.dart rename to lib/src/decision_tree_solver/split_assessor/majority_split_assessor.dart index fae7e027..a08d0675 100644 --- a/lib/src/solver/non_linear/decision_tree/split_assessor/majority_split_assessor.dart +++ b/lib/src/decision_tree_solver/split_assessor/majority_split_assessor.dart @@ -1,7 +1,7 @@ import 'dart:collection'; import 'dart:math' as math; -import 'package:ml_algo/src/solver/non_linear/decision_tree/split_assessor/split_assessor.dart'; +import 'package:ml_algo/src/decision_tree_solver/split_assessor/split_assessor.dart'; import 'package:ml_linalg/matrix.dart'; import 'package:ml_linalg/vector.dart'; diff --git a/lib/src/solver/non_linear/decision_tree/split_assessor/split_assessor.dart b/lib/src/decision_tree_solver/split_assessor/split_assessor.dart similarity index 100% rename from lib/src/solver/non_linear/decision_tree/split_assessor/split_assessor.dart rename to lib/src/decision_tree_solver/split_assessor/split_assessor.dart diff --git a/lib/src/solver/non_linear/decision_tree/split_selector/greedy_split_selector.dart b/lib/src/decision_tree_solver/split_selector/greedy_split_selector.dart similarity index 73% rename from lib/src/solver/non_linear/decision_tree/split_selector/greedy_split_selector.dart rename to lib/src/decision_tree_solver/split_selector/greedy_split_selector.dart index 5f25c047..25be9885 100644 --- a/lib/src/solver/non_linear/decision_tree/split_selector/greedy_split_selector.dart +++ b/lib/src/decision_tree_solver/split_selector/greedy_split_selector.dart @@ -1,7 +1,7 @@ -import 'package:ml_algo/src/solver/non_linear/decision_tree/decision_tree_node.dart'; -import 'package:ml_algo/src/solver/non_linear/decision_tree/split_assessor/split_assessor.dart'; -import 'package:ml_algo/src/solver/non_linear/decision_tree/split_selector/split_selector.dart'; -import 'package:ml_algo/src/solver/non_linear/decision_tree/splitter/splitter.dart'; +import 'package:ml_algo/src/decision_tree_solver/decision_tree_node.dart'; +import 'package:ml_algo/src/decision_tree_solver/split_assessor/split_assessor.dart'; +import 'package:ml_algo/src/decision_tree_solver/split_selector/split_selector.dart'; +import 'package:ml_algo/src/decision_tree_solver/splitter/splitter.dart'; import 'package:ml_linalg/matrix.dart'; class GreedySplitSelector implements SplitSelector { diff --git a/lib/src/solver/non_linear/decision_tree/split_selector/split_selector.dart b/lib/src/decision_tree_solver/split_selector/split_selector.dart similarity index 73% rename from lib/src/solver/non_linear/decision_tree/split_selector/split_selector.dart rename to lib/src/decision_tree_solver/split_selector/split_selector.dart index 26d6d3dd..de59b409 100644 --- a/lib/src/solver/non_linear/decision_tree/split_selector/split_selector.dart +++ b/lib/src/decision_tree_solver/split_selector/split_selector.dart @@ -1,4 +1,4 @@ -import 'package:ml_algo/src/solver/non_linear/decision_tree/decision_tree_node.dart'; +import 'package:ml_algo/src/decision_tree_solver/decision_tree_node.dart'; import 'package:ml_linalg/matrix.dart'; abstract class SplitSelector { diff --git a/lib/src/solver/non_linear/decision_tree/splitter/greedy_splitter.dart b/lib/src/decision_tree_solver/splitter/greedy_splitter.dart similarity index 72% rename from lib/src/solver/non_linear/decision_tree/splitter/greedy_splitter.dart rename to lib/src/decision_tree_solver/splitter/greedy_splitter.dart index 90066612..c7beab74 100644 --- a/lib/src/solver/non_linear/decision_tree/splitter/greedy_splitter.dart +++ b/lib/src/decision_tree_solver/splitter/greedy_splitter.dart @@ -1,11 +1,11 @@ -import 'package:ml_algo/src/solver/non_linear/decision_tree/decision_tree_node.dart'; -import 'package:ml_algo/src/solver/non_linear/decision_tree/split_assessor/split_assessor.dart'; -import 'package:ml_algo/src/solver/non_linear/decision_tree/splitter/nominal_splitter/nominal_splitter.dart'; -import 'package:ml_algo/src/solver/non_linear/decision_tree/splitter/numerical_splitter/numerical_splitter.dart'; -import 'package:ml_algo/src/solver/non_linear/decision_tree/splitter/splitter.dart'; +import 'package:ml_algo/src/decision_tree_solver/decision_tree_node.dart'; +import 'package:ml_algo/src/decision_tree_solver/split_assessor/split_assessor.dart'; +import 'package:ml_algo/src/decision_tree_solver/splitter/nominal_splitter/nominal_splitter.dart'; +import 'package:ml_algo/src/decision_tree_solver/splitter/numerical_splitter/numerical_splitter.dart'; +import 'package:ml_algo/src/decision_tree_solver/splitter/splitter.dart'; import 'package:ml_linalg/axis.dart'; import 'package:ml_linalg/matrix.dart'; -import 'package:xrange/zrange.dart'; +import 'package:xrange/integers.dart'; class GreedySplitter implements Splitter { GreedySplitter(this._assessor, this._numericalSplitter, @@ -27,7 +27,7 @@ class GreedySplitter implements Splitter { if (splittingIdx < 0 || splittingIdx > samples.columnsNum) { throw Exception('Unappropriate range given: $splittingIdx, ' 'expected a range within or equal ' - '${ZRange.closed(0, samples.columnsNum)}'); + '${integers(0, samples.columnsNum)}'); } return _nominalSplitter.split(samples, splittingIdx, values); } @@ -39,7 +39,13 @@ class GreedySplitter implements Splitter { var prevValue = sortedRows.first[splittingIdx]; for (final row in sortedRows.skip(1)) { - final splittingValue = (prevValue + row[splittingIdx]) / 2; + final nextValue = row[splittingIdx]; + + if (prevValue == nextValue) { + continue; + } + + final splittingValue = (prevValue + nextValue) / 2; final split = _numericalSplitter .split(samples, splittingIdx, splittingValue); final error = _assessor.getAggregatedError(split.values, targetId); diff --git a/lib/src/solver/non_linear/decision_tree/splitter/nominal_splitter/nominal_splitter.dart b/lib/src/decision_tree_solver/splitter/nominal_splitter/nominal_splitter.dart similarity index 71% rename from lib/src/solver/non_linear/decision_tree/splitter/nominal_splitter/nominal_splitter.dart rename to lib/src/decision_tree_solver/splitter/nominal_splitter/nominal_splitter.dart index a4a19e6a..801bee80 100644 --- a/lib/src/solver/non_linear/decision_tree/splitter/nominal_splitter/nominal_splitter.dart +++ b/lib/src/decision_tree_solver/splitter/nominal_splitter/nominal_splitter.dart @@ -1,4 +1,4 @@ -import 'package:ml_algo/src/solver/non_linear/decision_tree/decision_tree_node.dart'; +import 'package:ml_algo/src/decision_tree_solver/decision_tree_node.dart'; import 'package:ml_linalg/linalg.dart'; import 'package:ml_linalg/matrix.dart'; diff --git a/lib/src/solver/non_linear/decision_tree/splitter/nominal_splitter/nominal_splitter_impl.dart b/lib/src/decision_tree_solver/splitter/nominal_splitter/nominal_splitter_impl.dart similarity index 79% rename from lib/src/solver/non_linear/decision_tree/splitter/nominal_splitter/nominal_splitter_impl.dart rename to lib/src/decision_tree_solver/splitter/nominal_splitter/nominal_splitter_impl.dart index 5ed31e54..99f79a60 100644 --- a/lib/src/solver/non_linear/decision_tree/splitter/nominal_splitter/nominal_splitter_impl.dart +++ b/lib/src/decision_tree_solver/splitter/nominal_splitter/nominal_splitter_impl.dart @@ -1,5 +1,5 @@ -import 'package:ml_algo/src/solver/non_linear/decision_tree/decision_tree_node.dart'; -import 'package:ml_algo/src/solver/non_linear/decision_tree/splitter/nominal_splitter/nominal_splitter.dart'; +import 'package:ml_algo/src/decision_tree_solver/decision_tree_node.dart'; +import 'package:ml_algo/src/decision_tree_solver/splitter/nominal_splitter/nominal_splitter.dart'; import 'package:ml_linalg/matrix.dart'; import 'package:ml_linalg/vector.dart'; diff --git a/lib/src/solver/non_linear/decision_tree/splitter/numerical_splitter/numerical_splitter.dart b/lib/src/decision_tree_solver/splitter/numerical_splitter/numerical_splitter.dart similarity index 59% rename from lib/src/solver/non_linear/decision_tree/splitter/numerical_splitter/numerical_splitter.dart rename to lib/src/decision_tree_solver/splitter/numerical_splitter/numerical_splitter.dart index 25937dae..0ee11f1f 100644 --- a/lib/src/solver/non_linear/decision_tree/splitter/numerical_splitter/numerical_splitter.dart +++ b/lib/src/decision_tree_solver/splitter/numerical_splitter/numerical_splitter.dart @@ -1,6 +1,5 @@ -import 'package:ml_algo/src/solver/non_linear/decision_tree/decision_tree_node.dart'; +import 'package:ml_algo/src/decision_tree_solver/decision_tree_node.dart'; import 'package:ml_linalg/matrix.dart'; -import 'package:xrange/zrange.dart'; abstract class NumericalSplitter { Map split(Matrix samples, int splittingIdx, diff --git a/lib/src/solver/non_linear/decision_tree/splitter/numerical_splitter/numerical_splitter_impl.dart b/lib/src/decision_tree_solver/splitter/numerical_splitter/numerical_splitter_impl.dart similarity index 83% rename from lib/src/solver/non_linear/decision_tree/splitter/numerical_splitter/numerical_splitter_impl.dart rename to lib/src/decision_tree_solver/splitter/numerical_splitter/numerical_splitter_impl.dart index b3fd7abf..2a0c5f67 100644 --- a/lib/src/solver/non_linear/decision_tree/splitter/numerical_splitter/numerical_splitter_impl.dart +++ b/lib/src/decision_tree_solver/splitter/numerical_splitter/numerical_splitter_impl.dart @@ -1,5 +1,5 @@ -import 'package:ml_algo/src/solver/non_linear/decision_tree/decision_tree_node.dart'; -import 'package:ml_algo/src/solver/non_linear/decision_tree/splitter/numerical_splitter/numerical_splitter.dart'; +import 'package:ml_algo/src/decision_tree_solver/decision_tree_node.dart'; +import 'package:ml_algo/src/decision_tree_solver/splitter/numerical_splitter/numerical_splitter.dart'; import 'package:ml_linalg/matrix.dart'; import 'package:ml_linalg/vector.dart'; diff --git a/lib/src/solver/non_linear/decision_tree/splitter/splitter.dart b/lib/src/decision_tree_solver/splitter/splitter.dart similarity index 68% rename from lib/src/solver/non_linear/decision_tree/splitter/splitter.dart rename to lib/src/decision_tree_solver/splitter/splitter.dart index 8d22d61b..7d06138f 100644 --- a/lib/src/solver/non_linear/decision_tree/splitter/splitter.dart +++ b/lib/src/decision_tree_solver/splitter/splitter.dart @@ -1,4 +1,4 @@ -import 'package:ml_algo/src/solver/non_linear/decision_tree/decision_tree_node.dart'; +import 'package:ml_algo/src/decision_tree_solver/decision_tree_node.dart'; import 'package:ml_linalg/matrix.dart'; abstract class Splitter { diff --git a/lib/src/di/injector.dart b/lib/src/di/injector.dart new file mode 100644 index 00000000..25442c6b --- /dev/null +++ b/lib/src/di/injector.dart @@ -0,0 +1,40 @@ +import 'package:injector/injector.dart'; +import 'package:ml_algo/src/cost_function/cost_function_factory.dart'; +import 'package:ml_algo/src/cost_function/cost_function_factory_impl.dart'; +import 'package:ml_algo/src/linear_optimizer/convergence_detector/convergence_detector_factory.dart'; +import 'package:ml_algo/src/linear_optimizer/convergence_detector/convergence_detector_factory_impl.dart'; +import 'package:ml_algo/src/linear_optimizer/gradient_optimizer/learning_rate_generator/learning_rate_generator_factory.dart'; +import 'package:ml_algo/src/linear_optimizer/gradient_optimizer/learning_rate_generator/learning_rate_generator_factory_impl.dart'; +import 'package:ml_algo/src/linear_optimizer/initial_coefficients_generator/initial_coefficients_generator_factory.dart'; +import 'package:ml_algo/src/linear_optimizer/initial_coefficients_generator/initial_coefficients_generator_factory_impl.dart'; +import 'package:ml_algo/src/linear_optimizer/linear_optimizer_factory.dart'; +import 'package:ml_algo/src/linear_optimizer/linear_optimizer_factory_impl.dart'; +import 'package:ml_algo/src/link_function/link_function_factory.dart'; +import 'package:ml_algo/src/link_function/link_function_factory_impl.dart'; +import 'package:ml_algo/src/math/randomizer/randomizer_factory.dart'; +import 'package:ml_algo/src/math/randomizer/randomizer_factory_impl.dart'; + +Injector injector; + +Injector getDependencies() => + injector ??= Injector() + ..registerSingleton( + (_) => const LinearOptimizerFactoryImpl()) + + ..registerSingleton( + (_) => const RandomizerFactoryImpl()) + + ..registerSingleton( + (_) => const LearningRateGeneratorFactoryImpl()) + + ..registerSingleton( + (_) => const InitialCoefficientsGeneratorFactoryImpl()) + + ..registerDependency( + (_) => const ConvergenceDetectorFactoryImpl()) + + ..registerSingleton( + (_) => const CostFunctionFactoryImpl()) + + ..registerSingleton( + (_) => const LinkFunctionFactoryImpl()); diff --git a/lib/src/helpers/features_target_split.dart b/lib/src/helpers/features_target_split.dart new file mode 100644 index 00000000..0455e42e --- /dev/null +++ b/lib/src/helpers/features_target_split.dart @@ -0,0 +1,35 @@ +import 'package:ml_dataframe/ml_dataframe.dart'; +import 'package:quiver/iterables.dart'; + +Iterable featuresTargetSplit(DataFrame dataset, { + Iterable targetIndices = const [], + Iterable targetNames = const [], +}) { + if (targetIndices.isNotEmpty) { + final uniqueTargetIndices = Set.from(targetIndices); + + final featuresIndices = enumerate(dataset.header) + .where((indexedName) => !uniqueTargetIndices.contains(indexedName.index)) + .map((indexedName) => indexedName.index); + + return [ + dataset.sampleFromSeries(indices: featuresIndices), + dataset.sampleFromSeries(indices: uniqueTargetIndices), + ]; + } + + if (targetNames.isNotEmpty) { + final uniqueTargetNames = Set.from(targetNames); + + final featuresNames = dataset + .header + .where((name) => !uniqueTargetNames.contains(name)); + + return [ + dataset.sampleFromSeries(names: featuresNames), + dataset.sampleFromSeries(names: uniqueTargetNames), + ]; + } + + throw Exception('Neither target index, nor target name are provided'); +} diff --git a/lib/src/solver/linear/convergence_detector/convergence_detector.dart b/lib/src/linear_optimizer/convergence_detector/convergence_detector.dart similarity index 100% rename from lib/src/solver/linear/convergence_detector/convergence_detector.dart rename to lib/src/linear_optimizer/convergence_detector/convergence_detector.dart diff --git a/lib/src/solver/linear/convergence_detector/convergence_detector_factory.dart b/lib/src/linear_optimizer/convergence_detector/convergence_detector_factory.dart similarity index 55% rename from lib/src/solver/linear/convergence_detector/convergence_detector_factory.dart rename to lib/src/linear_optimizer/convergence_detector/convergence_detector_factory.dart index d065d799..3e6aa9f0 100644 --- a/lib/src/solver/linear/convergence_detector/convergence_detector_factory.dart +++ b/lib/src/linear_optimizer/convergence_detector/convergence_detector_factory.dart @@ -1,4 +1,4 @@ -import 'package:ml_algo/src/solver/linear/convergence_detector/convergence_detector.dart'; +import 'package:ml_algo/src/linear_optimizer/convergence_detector/convergence_detector.dart'; abstract class ConvergenceDetectorFactory { ConvergenceDetector create(double minUpdate, int iterationsLimit); diff --git a/lib/src/linear_optimizer/convergence_detector/convergence_detector_factory_impl.dart b/lib/src/linear_optimizer/convergence_detector/convergence_detector_factory_impl.dart new file mode 100644 index 00000000..68ce37b9 --- /dev/null +++ b/lib/src/linear_optimizer/convergence_detector/convergence_detector_factory_impl.dart @@ -0,0 +1,11 @@ +import 'package:ml_algo/src/linear_optimizer/convergence_detector/convergence_detector.dart'; +import 'package:ml_algo/src/linear_optimizer/convergence_detector/convergence_detector_factory.dart'; +import 'package:ml_algo/src/linear_optimizer/convergence_detector/convergence_detector_impl.dart'; + +class ConvergenceDetectorFactoryImpl implements ConvergenceDetectorFactory { + const ConvergenceDetectorFactoryImpl(); + + @override + ConvergenceDetector create(double minUpdate, int iterationsLimit) => + ConvergenceDetectorImpl(minUpdate, iterationsLimit); +} diff --git a/lib/src/linear_optimizer/convergence_detector/convergence_detector_impl.dart b/lib/src/linear_optimizer/convergence_detector/convergence_detector_impl.dart new file mode 100644 index 00000000..2d3f9eb1 --- /dev/null +++ b/lib/src/linear_optimizer/convergence_detector/convergence_detector_impl.dart @@ -0,0 +1,24 @@ +import 'package:ml_algo/src/linear_optimizer/convergence_detector/convergence_detector.dart'; + +class ConvergenceDetectorImpl implements ConvergenceDetector { + ConvergenceDetectorImpl(this.minDiff, this.iterationsLimit); + + @override + final double minDiff; + + @override + final int iterationsLimit; + + @override + bool isConverged(double coefficientsDiff, int iteration) { + if (iterationsLimit != null && iteration >= iterationsLimit) { + return true; + } + + if (minDiff != null && coefficientsDiff <= minDiff) { + return true; + } + + return false; + } +} diff --git a/lib/src/solver/linear/coordinate/coordinate.dart b/lib/src/linear_optimizer/coordinate_optimizer/coordinate_descent_optimizer.dart similarity index 53% rename from lib/src/solver/linear/coordinate/coordinate.dart rename to lib/src/linear_optimizer/coordinate_optimizer/coordinate_descent_optimizer.dart index 921daff9..e58de9b7 100644 --- a/lib/src/solver/linear/coordinate/coordinate.dart +++ b/lib/src/linear_optimizer/coordinate_optimizer/coordinate_descent_optimizer.dart @@ -1,46 +1,45 @@ import 'package:ml_algo/src/cost_function/cost_function.dart'; -import 'package:ml_algo/src/solver/linear/convergence_detector/convergence_detector.dart'; -import 'package:ml_algo/src/solver/linear/convergence_detector/convergence_detector_factory.dart'; -import 'package:ml_algo/src/solver/linear/convergence_detector/convergence_detector_factory_impl.dart'; -import 'package:ml_algo/src/solver/linear/initial_weights_generator/initial_weights_generator.dart'; -import 'package:ml_algo/src/solver/linear/initial_weights_generator/initial_weights_generator_factory.dart'; -import 'package:ml_algo/src/solver/linear/initial_weights_generator/initial_weights_generator_factory_impl.dart'; -import 'package:ml_algo/src/solver/linear/initial_weights_generator/initial_weights_type.dart'; -import 'package:ml_algo/src/solver/linear/linear_optimizer.dart'; -import 'package:ml_algo/src/utils/default_parameter_values.dart'; +import 'package:ml_algo/src/di/injector.dart'; +import 'package:ml_algo/src/linear_optimizer/convergence_detector/convergence_detector.dart'; +import 'package:ml_algo/src/linear_optimizer/convergence_detector/convergence_detector_factory.dart'; +import 'package:ml_algo/src/linear_optimizer/initial_coefficients_generator/initial_coefficients_generator.dart'; +import 'package:ml_algo/src/linear_optimizer/initial_coefficients_generator/initial_coefficients_generator_factory.dart'; +import 'package:ml_algo/src/linear_optimizer/initial_coefficients_generator/initial_coefficients_type.dart'; +import 'package:ml_algo/src/linear_optimizer/linear_optimizer.dart'; import 'package:ml_linalg/dtype.dart'; import 'package:ml_linalg/linalg.dart'; -class CoordinateOptimizer implements LinearOptimizer { - CoordinateOptimizer(Matrix points, Matrix labels, { - DType dtype = DefaultParameterValues.dtype, +class CoordinateDescentOptimizer implements LinearOptimizer { + CoordinateDescentOptimizer(Matrix fittingPoints, Matrix fittingLabels, { + DType dtype = DType.float32, CostFunction costFunction, - InitialWeightsGeneratorFactory initialWeightsGeneratorFactory = - const InitialWeightsGeneratorFactoryImpl(), - ConvergenceDetectorFactory convergenceDetectorFactory = - const ConvergenceDetectorFactoryImpl(), - double minCoefficientsDiff = DefaultParameterValues.minCoefficientsUpdate, - int iterationsLimit = DefaultParameterValues.iterationsLimit, + double minCoefficientsUpdate = 1e-12, + int iterationsLimit = 100, double lambda, - InitialWeightsType initialWeightsType = InitialWeightsType.zeroes, - bool isTrainDataNormalized = false, + InitialCoefficientsType initialWeightsType = InitialCoefficientsType.zeroes, + bool isFittingDataNormalized = false, }) : _dtype = dtype, - _points = points, - _labels = labels, + _points = fittingPoints, + _labels = fittingLabels, _lambda = lambda ?? 0.0, - _initialCoefficientsGenerator = - initialWeightsGeneratorFactory.fromType(initialWeightsType, dtype), - _convergenceDetector = convergenceDetectorFactory.create( - minCoefficientsDiff, iterationsLimit), + + _initialCoefficientsGenerator = getDependencies() + .getDependency() + .fromType(initialWeightsType, dtype), + + _convergenceDetector = getDependencies() + .getDependency() + .create(minCoefficientsUpdate, iterationsLimit), + _costFn = costFunction, - _normalizer = isTrainDataNormalized - ? Vector.filled(points.columnsNum, 1.0, dtype: dtype) - : points.reduceRows( + _normalizer = isFittingDataNormalized + ? Vector.filled(fittingPoints.columnsNum, 1.0, dtype: dtype) + : fittingPoints.reduceRows( (combine, vector) => (combine + vector * vector)); final Matrix _points; final Matrix _labels; - final InitialWeightsGenerator _initialCoefficientsGenerator; + final InitialCoefficientsGenerator _initialCoefficientsGenerator; final ConvergenceDetector _convergenceDetector; final CostFunction _costFn; final DType _dtype; @@ -49,10 +48,10 @@ class CoordinateOptimizer implements LinearOptimizer { @override Matrix findExtrema({ - Matrix initialWeights, + Matrix initialCoefficients, bool isMinimizingObjective = true, }) { - Matrix coefficients = initialWeights ?? + Matrix coefficients = initialCoefficients ?? Matrix.fromRows(List.generate(_labels.columnsNum, (int i) => _initialCoefficientsGenerator .generate(_points.columnsNum)), dtype: _dtype); @@ -76,13 +75,13 @@ class CoordinateOptimizer implements LinearOptimizer { iteration++; } - return coefficients; + return coefficients.transpose(); } Vector _optimizeCoordinate(int j, Matrix x, Matrix y, Matrix w) { // coefficients variable here contains coefficients on column j per each // label - final coefficients = _costFn.getSubDerivative(j, x, w, y); + final coefficients = _costFn.getSubGradient(j, x, w, y); return Vector // TODO Convert the logic into SIMD-way (SIMD way mapping) .fromList(coefficients.map((coef) => _regularize(coef, _lambda, j)) diff --git a/lib/src/solver/linear/gradient/gradient.dart b/lib/src/linear_optimizer/gradient_optimizer/gradient_optimizer.dart similarity index 52% rename from lib/src/solver/linear/gradient/gradient.dart rename to lib/src/linear_optimizer/gradient_optimizer/gradient_optimizer.dart index 56c859ef..3399c715 100644 --- a/lib/src/solver/linear/gradient/gradient.dart +++ b/lib/src/linear_optimizer/gradient_optimizer/gradient_optimizer.dart @@ -1,47 +1,29 @@ import 'package:ml_algo/src/cost_function/cost_function.dart'; +import 'package:ml_algo/src/di/injector.dart'; +import 'package:ml_algo/src/linear_optimizer/convergence_detector/convergence_detector.dart'; +import 'package:ml_algo/src/linear_optimizer/convergence_detector/convergence_detector_factory.dart'; +import 'package:ml_algo/src/linear_optimizer/gradient_optimizer/learning_rate_generator/learning_rate_generator.dart'; +import 'package:ml_algo/src/linear_optimizer/gradient_optimizer/learning_rate_generator/learning_rate_generator_factory.dart'; +import 'package:ml_algo/src/linear_optimizer/gradient_optimizer/learning_rate_generator/learning_rate_type.dart'; +import 'package:ml_algo/src/linear_optimizer/initial_coefficients_generator/initial_coefficients_generator.dart'; +import 'package:ml_algo/src/linear_optimizer/initial_coefficients_generator/initial_coefficients_generator_factory.dart'; +import 'package:ml_algo/src/linear_optimizer/initial_coefficients_generator/initial_coefficients_type.dart'; +import 'package:ml_algo/src/linear_optimizer/linear_optimizer.dart'; import 'package:ml_algo/src/math/randomizer/randomizer.dart'; import 'package:ml_algo/src/math/randomizer/randomizer_factory.dart'; -import 'package:ml_algo/src/math/randomizer/randomizer_factory_impl.dart'; -import 'package:ml_algo/src/solver/linear/convergence_detector/convergence_detector.dart'; -import 'package:ml_algo/src/solver/linear/convergence_detector/convergence_detector_factory.dart'; -import 'package:ml_algo/src/solver/linear/convergence_detector/convergence_detector_factory_impl.dart'; -import 'package:ml_algo/src/solver/linear/gradient/learning_rate_generator/learning_rate_generator.dart'; -import 'package:ml_algo/src/solver/linear/gradient/learning_rate_generator/learning_rate_generator_factory.dart'; -import 'package:ml_algo/src/solver/linear/gradient/learning_rate_generator/learning_rate_generator_factory_impl.dart'; -import 'package:ml_algo/src/solver/linear/gradient/learning_rate_generator/learning_rate_type.dart'; -import 'package:ml_algo/src/solver/linear/initial_weights_generator/initial_weights_generator.dart'; -import 'package:ml_algo/src/solver/linear/initial_weights_generator/initial_weights_generator_factory.dart'; -import 'package:ml_algo/src/solver/linear/initial_weights_generator/initial_weights_generator_factory_impl.dart'; -import 'package:ml_algo/src/solver/linear/initial_weights_generator/initial_weights_type.dart'; -import 'package:ml_algo/src/solver/linear/linear_optimizer.dart'; -import 'package:ml_algo/src/utils/default_parameter_values.dart'; import 'package:ml_linalg/dtype.dart'; import 'package:ml_linalg/matrix.dart'; -import 'package:xrange/zrange.dart'; +import 'package:xrange/integers.dart'; class GradientOptimizer implements LinearOptimizer { GradientOptimizer(Matrix points, Matrix labels, { - DType dtype = DefaultParameterValues.dtype, - + DType dtype = DType.float32, CostFunction costFunction, - - RandomizerFactory randomizerFactory = - const RandomizerFactoryImpl(), - - LearningRateGeneratorFactory learningRateGeneratorFactory = - const LearningRateGeneratorFactoryImpl(), - - InitialWeightsGeneratorFactory initialWeightsGeneratorFactory = - const InitialWeightsGeneratorFactoryImpl(), - - ConvergenceDetectorFactory convergenceDetectorFactory = - const ConvergenceDetectorFactoryImpl(), - - LearningRateType learningRateType, - InitialWeightsType initialWeightsType, - double initialLearningRate = DefaultParameterValues.initialLearningRate, - double minCoefficientsUpdate = DefaultParameterValues.minCoefficientsUpdate, - int iterationLimit = DefaultParameterValues.iterationsLimit, + LearningRateType learningRateType = LearningRateType.decreasingAdaptive, + InitialCoefficientsType initialCoefficientsType = InitialCoefficientsType.zeroes, + double initialLearningRate = 1e-3, + double minCoefficientsUpdate = 1e-12, + int iterationLimit = 100, double lambda, int batchSize, int randomSeed, @@ -50,14 +32,23 @@ class GradientOptimizer implements LinearOptimizer { _labels = labels, _lambda = lambda ?? 0.0, _batchSize = batchSize, - _initialWeightsGenerator = - initialWeightsGeneratorFactory.fromType(initialWeightsType, dtype), - _learningRateGenerator = - learningRateGeneratorFactory.fromType(learningRateType), _costFunction = costFunction, - _convergenceDetector = convergenceDetectorFactory.create( - minCoefficientsUpdate, iterationLimit), - _randomizer = randomizerFactory.create(randomSeed) { + + _initialWeightsGenerator = getDependencies() + .getDependency() + .fromType(initialCoefficientsType, dtype), + + _learningRateGenerator = getDependencies() + .getDependency() + .fromType(learningRateType), + + _convergenceDetector = getDependencies() + .getDependency() + .create(minCoefficientsUpdate, iterationLimit), + + _randomizer = getDependencies() + .getDependency() + .create(randomSeed) { if (batchSize < 1 || batchSize > points.rowsNum) { throw RangeError.range(batchSize, 1, points.rowsNum, 'Invalid batch size ' 'value'); @@ -70,19 +61,19 @@ class GradientOptimizer implements LinearOptimizer { final Randomizer _randomizer; final CostFunction _costFunction; final LearningRateGenerator _learningRateGenerator; - final InitialWeightsGenerator _initialWeightsGenerator; + final InitialCoefficientsGenerator _initialWeightsGenerator; final ConvergenceDetector _convergenceDetector; final double _lambda; final int _batchSize; @override - Matrix findExtrema({Matrix initialWeights, + Matrix findExtrema({Matrix initialCoefficients, bool isMinimizingObjective = true}) { final batchSize = _batchSize >= _points.rowsNum ? _points.rowsNum : _batchSize; - Matrix coefficients = initialWeights ?? + Matrix coefficients = initialCoefficients ?? Matrix.fromColumns(List.generate(_labels.columnsNum, (i) => _initialWeightsGenerator.generate(_points.columnsNum))); @@ -110,8 +101,10 @@ class GradientOptimizer implements LinearOptimizer { final range = _getBatchRange(batchSize); final start = range.first; final end = range.last; - final pointsBatch = _points.submatrix(rows: ZRange.closedOpen(start, end)); - final labelsBatch = labels.submatrix(rows: ZRange.closedOpen(start, end)); + final pointsBatch = _points + .sample(rowIndices: integers(start, end, upperClosed: false)); + final labelsBatch = labels + .sample(rowIndices: integers(start, end, upperClosed: false)); return _makeGradientStep(coefficients, pointsBatch, labelsBatch, eta, isMinimization: isMinimization); diff --git a/lib/src/solver/linear/gradient/learning_rate_generator/constant.dart b/lib/src/linear_optimizer/gradient_optimizer/learning_rate_generator/constant.dart similarity index 69% rename from lib/src/solver/linear/gradient/learning_rate_generator/constant.dart rename to lib/src/linear_optimizer/gradient_optimizer/learning_rate_generator/constant.dart index aa602848..927f41fe 100644 --- a/lib/src/solver/linear/gradient/learning_rate_generator/constant.dart +++ b/lib/src/linear_optimizer/gradient_optimizer/learning_rate_generator/constant.dart @@ -1,4 +1,4 @@ -import 'package:ml_algo/src/solver/linear/gradient/learning_rate_generator/learning_rate_generator.dart'; +import 'package:ml_algo/src/linear_optimizer/gradient_optimizer/learning_rate_generator/learning_rate_generator.dart'; class ConstantLearningRateGenerator implements LearningRateGenerator { double _initialValue; diff --git a/lib/src/solver/linear/gradient/learning_rate_generator/decreasing.dart b/lib/src/linear_optimizer/gradient_optimizer/learning_rate_generator/decreasing.dart similarity index 74% rename from lib/src/solver/linear/gradient/learning_rate_generator/decreasing.dart rename to lib/src/linear_optimizer/gradient_optimizer/learning_rate_generator/decreasing.dart index c0f654c9..6033cccf 100644 --- a/lib/src/solver/linear/gradient/learning_rate_generator/decreasing.dart +++ b/lib/src/linear_optimizer/gradient_optimizer/learning_rate_generator/decreasing.dart @@ -1,4 +1,4 @@ -import 'package:ml_algo/src/solver/linear/gradient/learning_rate_generator/learning_rate_generator.dart'; +import 'package:ml_algo/src/linear_optimizer/gradient_optimizer/learning_rate_generator/learning_rate_generator.dart'; class DecreasingLearningRateGenerator implements LearningRateGenerator { double _initialValue; diff --git a/lib/src/solver/linear/gradient/learning_rate_generator/learning_rate_generator.dart b/lib/src/linear_optimizer/gradient_optimizer/learning_rate_generator/learning_rate_generator.dart similarity index 100% rename from lib/src/solver/linear/gradient/learning_rate_generator/learning_rate_generator.dart rename to lib/src/linear_optimizer/gradient_optimizer/learning_rate_generator/learning_rate_generator.dart diff --git a/lib/src/linear_optimizer/gradient_optimizer/learning_rate_generator/learning_rate_generator_factory.dart b/lib/src/linear_optimizer/gradient_optimizer/learning_rate_generator/learning_rate_generator_factory.dart new file mode 100644 index 00000000..8d70dff7 --- /dev/null +++ b/lib/src/linear_optimizer/gradient_optimizer/learning_rate_generator/learning_rate_generator_factory.dart @@ -0,0 +1,6 @@ +import 'package:ml_algo/src/linear_optimizer/gradient_optimizer/learning_rate_generator/learning_rate_generator.dart'; +import 'package:ml_algo/src/linear_optimizer/gradient_optimizer/learning_rate_generator/learning_rate_type.dart'; + +abstract class LearningRateGeneratorFactory { + LearningRateGenerator fromType(LearningRateType type); +} diff --git a/lib/src/linear_optimizer/gradient_optimizer/learning_rate_generator/learning_rate_generator_factory_impl.dart b/lib/src/linear_optimizer/gradient_optimizer/learning_rate_generator/learning_rate_generator_factory_impl.dart new file mode 100644 index 00000000..98711e36 --- /dev/null +++ b/lib/src/linear_optimizer/gradient_optimizer/learning_rate_generator/learning_rate_generator_factory_impl.dart @@ -0,0 +1,23 @@ +import 'package:ml_algo/src/linear_optimizer/gradient_optimizer/learning_rate_generator/constant.dart'; +import 'package:ml_algo/src/linear_optimizer/gradient_optimizer/learning_rate_generator/decreasing.dart'; +import 'package:ml_algo/src/linear_optimizer/gradient_optimizer/learning_rate_generator/learning_rate_generator.dart'; +import 'package:ml_algo/src/linear_optimizer/gradient_optimizer/learning_rate_generator/learning_rate_generator_factory.dart'; +import 'package:ml_algo/src/linear_optimizer/gradient_optimizer/learning_rate_generator/learning_rate_type.dart'; + +class LearningRateGeneratorFactoryImpl implements LearningRateGeneratorFactory { + const LearningRateGeneratorFactoryImpl(); + + @override + LearningRateGenerator fromType(LearningRateType type) { + switch (type) { + case LearningRateType.constant: + return ConstantLearningRateGenerator(); + + case LearningRateType.decreasingAdaptive: + return DecreasingLearningRateGenerator(); + + default: + throw UnsupportedError('Unsupported learning rate type - $type'); + } + } +} diff --git a/lib/src/linear_optimizer/gradient_optimizer/learning_rate_generator/learning_rate_type.dart b/lib/src/linear_optimizer/gradient_optimizer/learning_rate_generator/learning_rate_type.dart new file mode 100644 index 00000000..0e19c712 --- /dev/null +++ b/lib/src/linear_optimizer/gradient_optimizer/learning_rate_generator/learning_rate_type.dart @@ -0,0 +1,12 @@ +/// A type of learning rate behaviour +enum LearningRateType { + /// Learning rate will decrease every iteration according to the rule: + /// + /// ````dart + /// learningRateValue / iterationNumber + /// ```` + decreasingAdaptive, + + /// Learning rate will be constant throughout the whole fitting process + constant, +} diff --git a/lib/src/solver/linear/initial_weights_generator/initial_weights_generator.dart b/lib/src/linear_optimizer/initial_coefficients_generator/initial_coefficients_generator.dart similarity index 61% rename from lib/src/solver/linear/initial_weights_generator/initial_weights_generator.dart rename to lib/src/linear_optimizer/initial_coefficients_generator/initial_coefficients_generator.dart index 375adf13..808aff00 100644 --- a/lib/src/solver/linear/initial_weights_generator/initial_weights_generator.dart +++ b/lib/src/linear_optimizer/initial_coefficients_generator/initial_coefficients_generator.dart @@ -1,5 +1,5 @@ import 'package:ml_linalg/linalg.dart'; -abstract class InitialWeightsGenerator { +abstract class InitialCoefficientsGenerator { Vector generate(int length); } diff --git a/lib/src/linear_optimizer/initial_coefficients_generator/initial_coefficients_generator_factory.dart b/lib/src/linear_optimizer/initial_coefficients_generator/initial_coefficients_generator_factory.dart new file mode 100644 index 00000000..1bf7c681 --- /dev/null +++ b/lib/src/linear_optimizer/initial_coefficients_generator/initial_coefficients_generator_factory.dart @@ -0,0 +1,8 @@ +import 'package:ml_algo/src/linear_optimizer/initial_coefficients_generator/initial_coefficients_generator.dart'; +import 'package:ml_algo/src/linear_optimizer/initial_coefficients_generator/initial_coefficients_type.dart'; +import 'package:ml_linalg/dtype.dart'; + +abstract class InitialCoefficientsGeneratorFactory { + InitialCoefficientsGenerator zeroes(DType dtype); + InitialCoefficientsGenerator fromType(InitialCoefficientsType type, DType dtype); +} diff --git a/lib/src/linear_optimizer/initial_coefficients_generator/initial_coefficients_generator_factory_impl.dart b/lib/src/linear_optimizer/initial_coefficients_generator/initial_coefficients_generator_factory_impl.dart new file mode 100644 index 00000000..f1a23367 --- /dev/null +++ b/lib/src/linear_optimizer/initial_coefficients_generator/initial_coefficients_generator_factory_impl.dart @@ -0,0 +1,23 @@ +import 'package:ml_algo/src/linear_optimizer/initial_coefficients_generator/initial_coefficients_generator.dart'; +import 'package:ml_algo/src/linear_optimizer/initial_coefficients_generator/initial_coefficients_generator_factory.dart'; +import 'package:ml_algo/src/linear_optimizer/initial_coefficients_generator/initial_coefficients_type.dart'; +import 'package:ml_algo/src/linear_optimizer/initial_coefficients_generator/zero_coefficients_generator.dart'; +import 'package:ml_linalg/dtype.dart'; + +class InitialCoefficientsGeneratorFactoryImpl + implements InitialCoefficientsGeneratorFactory { + const InitialCoefficientsGeneratorFactoryImpl(); + + @override + InitialCoefficientsGenerator zeroes(DType dtype) => ZeroCoefficientsGenerator(dtype); + + @override + InitialCoefficientsGenerator fromType(InitialCoefficientsType type, DType dtype) { + switch (type) { + case InitialCoefficientsType.zeroes: + return zeroes(dtype); + default: + throw UnsupportedError('Unsupported initial weights type - $type'); + } + } +} diff --git a/lib/src/linear_optimizer/initial_coefficients_generator/initial_coefficients_type.dart b/lib/src/linear_optimizer/initial_coefficients_generator/initial_coefficients_type.dart new file mode 100644 index 00000000..c581b607 --- /dev/null +++ b/lib/src/linear_optimizer/initial_coefficients_generator/initial_coefficients_type.dart @@ -0,0 +1,3 @@ +enum InitialCoefficientsType { + zeroes, +} diff --git a/lib/src/linear_optimizer/initial_coefficients_generator/zero_coefficients_generator.dart b/lib/src/linear_optimizer/initial_coefficients_generator/zero_coefficients_generator.dart new file mode 100644 index 00000000..72e37182 --- /dev/null +++ b/lib/src/linear_optimizer/initial_coefficients_generator/zero_coefficients_generator.dart @@ -0,0 +1,12 @@ +import 'package:ml_algo/src/linear_optimizer/initial_coefficients_generator/initial_coefficients_generator.dart'; +import 'package:ml_linalg/dtype.dart'; +import 'package:ml_linalg/linalg.dart'; + +class ZeroCoefficientsGenerator implements InitialCoefficientsGenerator { + ZeroCoefficientsGenerator(this.dtype); + + final DType dtype; + + @override + Vector generate(int length) => Vector.zero(length, dtype: dtype); +} diff --git a/lib/src/linear_optimizer/linear_optimizer.dart b/lib/src/linear_optimizer/linear_optimizer.dart new file mode 100644 index 00000000..0008a73d --- /dev/null +++ b/lib/src/linear_optimizer/linear_optimizer.dart @@ -0,0 +1,58 @@ +import 'package:ml_linalg/matrix.dart'; + +abstract class LinearOptimizer { + /// Returns a coefficients of the equation of a hyperplane. + /// + /// Dimensions of the returning matrix: + /// + /// ```` + /// number_of_features x number_of_target_columns + /// ```` + /// + /// In other words, one column in the coefficients matrix describes its own + /// dedicated target. + /// + /// Let's say, one has a dataset, consisting of features: + /// + /// ````dart + /// final x = [ + /// [10, 20, 30], + /// [11, 22, 33], + /// [22, 33, 43], + /// [89, 76, 32], + /// ]; + /// ```` + /// + /// And outcomes: + /// + /// ````dart + /// final y = [ + /// [100], + /// [200], + /// [300], + /// [400], + /// ]; + /// ```` + /// + /// After solving the equation of a hyperplane via the [LinearOptimizer], the + /// coefficients will be like that: + /// + /// ````dart + /// [ + /// [1.5], + /// [0.3], + /// [2.4], + /// ] + /// ```` + /// + /// Parameters: + /// + /// [initialCoefficients] initial coefficients that will be used in the first + /// optimization iteration + /// + /// [isMinimizingObjective] should the solver find a maxima or minima + Matrix findExtrema({ + Matrix initialCoefficients, + bool isMinimizingObjective, + }); +} diff --git a/lib/src/linear_optimizer/linear_optimizer_factory.dart b/lib/src/linear_optimizer/linear_optimizer_factory.dart new file mode 100644 index 00000000..32f0ea1f --- /dev/null +++ b/lib/src/linear_optimizer/linear_optimizer_factory.dart @@ -0,0 +1,28 @@ +import 'package:ml_algo/src/cost_function/cost_function.dart'; +import 'package:ml_algo/src/linear_optimizer/gradient_optimizer/learning_rate_generator/learning_rate_type.dart'; +import 'package:ml_algo/src/linear_optimizer/initial_coefficients_generator/initial_coefficients_type.dart'; +import 'package:ml_algo/src/linear_optimizer/linear_optimizer.dart'; +import 'package:ml_algo/src/linear_optimizer/linear_optimizer_type.dart'; +import 'package:ml_algo/src/linear_optimizer/regularization_type.dart'; +import 'package:ml_linalg/dtype.dart'; +import 'package:ml_linalg/matrix.dart'; + +abstract class LinearOptimizerFactory { + LinearOptimizer createByType( + LinearOptimizerType optimizerType, + Matrix points, + Matrix labels, { + DType dtype, + CostFunction costFunction, + LearningRateType learningRateType, + InitialCoefficientsType initialCoefficientsType, + double initialLearningRate, + double minCoefficientsUpdate, + int iterationLimit, + double lambda, + RegularizationType regularizationType, + int batchSize, + int randomSeed, + bool isFittingDataNormalized, + }); +} diff --git a/lib/src/linear_optimizer/linear_optimizer_factory_impl.dart b/lib/src/linear_optimizer/linear_optimizer_factory_impl.dart new file mode 100644 index 00000000..8d7bd22b --- /dev/null +++ b/lib/src/linear_optimizer/linear_optimizer_factory_impl.dart @@ -0,0 +1,76 @@ +import 'package:ml_algo/src/cost_function/cost_function.dart'; +import 'package:ml_algo/src/linear_optimizer/coordinate_optimizer/coordinate_descent_optimizer.dart'; +import 'package:ml_algo/src/linear_optimizer/gradient_optimizer/gradient_optimizer.dart'; +import 'package:ml_algo/src/linear_optimizer/gradient_optimizer/learning_rate_generator/learning_rate_type.dart'; +import 'package:ml_algo/src/linear_optimizer/initial_coefficients_generator/initial_coefficients_type.dart'; +import 'package:ml_algo/src/linear_optimizer/linear_optimizer.dart'; +import 'package:ml_algo/src/linear_optimizer/linear_optimizer_factory.dart'; +import 'package:ml_algo/src/linear_optimizer/linear_optimizer_type.dart'; +import 'package:ml_algo/src/linear_optimizer/optimizer_to_regularization_mapping.dart'; +import 'package:ml_algo/src/linear_optimizer/regularization_type.dart'; +import 'package:ml_linalg/dtype.dart'; +import 'package:ml_linalg/matrix.dart'; + +class LinearOptimizerFactoryImpl implements LinearOptimizerFactory { + const LinearOptimizerFactoryImpl(); + + @override + LinearOptimizer createByType( + LinearOptimizerType optimizerType, + Matrix fittingPoints, + Matrix fittingLabels, { + DType dtype = DType.float32, + CostFunction costFunction, + LearningRateType learningRateType, + InitialCoefficientsType initialCoefficientsType, + double initialLearningRate, + double minCoefficientsUpdate, + int iterationLimit, + double lambda, + RegularizationType regularizationType, + int batchSize, + int randomSeed, + bool isFittingDataNormalized, + }) { + if (regularizationType != null && + !optimizerToRegularization[optimizerType] + .contains(regularizationType)) { + throw UnsupportedError('Regularization type $regularizationType is ' + 'unsupported by optimizer $optimizerType'); + } + + switch (optimizerType) { + + case LinearOptimizerType.gradient: + return GradientOptimizer( + fittingPoints, fittingLabels, + costFunction: costFunction, + learningRateType: learningRateType, + initialCoefficientsType: initialCoefficientsType, + initialLearningRate: initialLearningRate, + minCoefficientsUpdate: minCoefficientsUpdate, + iterationLimit: iterationLimit, + lambda: lambda, + batchSize: batchSize, + randomSeed: randomSeed, + dtype: dtype, + ); + + case LinearOptimizerType.coordinate: + return CoordinateDescentOptimizer( + fittingPoints, fittingLabels, + dtype: dtype, + costFunction: costFunction, + minCoefficientsUpdate: minCoefficientsUpdate, + iterationsLimit: iterationLimit, + lambda: lambda, + initialWeightsType: initialCoefficientsType, + isFittingDataNormalized: isFittingDataNormalized, + ); + + default: + throw UnsupportedError( + 'Unsupported linear optimizer type - $optimizerType'); + } + } +} diff --git a/lib/src/linear_optimizer/linear_optimizer_type.dart b/lib/src/linear_optimizer/linear_optimizer_type.dart new file mode 100644 index 00000000..ab92c419 --- /dev/null +++ b/lib/src/linear_optimizer/linear_optimizer_type.dart @@ -0,0 +1,14 @@ +/// Linear optimization types +/// +/// The whole process of learning linear models confines to just finding +/// coefficients of features of predicting line/hyperplane. There are several +/// algorithms to find the coefficients that minimize a cost function best. +enum LinearOptimizerType { + /// Original gradient descent/ascent optimization, only L2 regularization is + /// applicable while optimizing a function using this method + gradient, + + /// Original coordinate descent optimization, only L1 regularization is + /// applicable while optimizing a function using this method + coordinate, +} diff --git a/lib/src/linear_optimizer/optimizer_to_regularization_mapping.dart b/lib/src/linear_optimizer/optimizer_to_regularization_mapping.dart new file mode 100644 index 00000000..6a78d18e --- /dev/null +++ b/lib/src/linear_optimizer/optimizer_to_regularization_mapping.dart @@ -0,0 +1,7 @@ +import 'package:ml_algo/src/linear_optimizer/linear_optimizer_type.dart'; +import 'package:ml_algo/src/linear_optimizer/regularization_type.dart'; + +const optimizerToRegularization = { + LinearOptimizerType.coordinate: [RegularizationType.L1], + LinearOptimizerType.gradient: [RegularizationType.L2], +}; \ No newline at end of file diff --git a/lib/src/linear_optimizer/regularization_type.dart b/lib/src/linear_optimizer/regularization_type.dart new file mode 100644 index 00000000..a538836d --- /dev/null +++ b/lib/src/linear_optimizer/regularization_type.dart @@ -0,0 +1,21 @@ +import 'package:ml_algo/src/linear_optimizer/linear_optimizer_type.dart'; + +/// Regularization types +/// +/// Machine learning linear models are prone to overfitting, or, in other words, +/// they may lose generalization ability in their learning process: overfitted +/// models show quite high quality on training data, but in the same time they +/// perform very badly on previously unseen data. +/// +/// The main reason of that is uncontrolled growth of linear model coefficients. +/// To avoid this, it is needed to measure a magnitude of coefficients vector +/// and consider it during the model's learning. +enum RegularizationType { + ///uses Manhattan norm of a vector to calculate magnitude of learned + ///coefficients. Applicable for [LinearOptimizerType.coordinate] + L1, + + ///uses Euclidean norm of a vector to calculate magnitude of learned + ///coefficients. Applicable for [LinearOptimizerType.gradient] + L2, +} diff --git a/lib/src/link_function/link_function_factory.dart b/lib/src/link_function/link_function_factory.dart new file mode 100644 index 00000000..0d64522f --- /dev/null +++ b/lib/src/link_function/link_function_factory.dart @@ -0,0 +1,9 @@ +import 'package:ml_algo/src/link_function/link_function.dart'; +import 'package:ml_algo/src/link_function/link_function_type.dart'; +import 'package:ml_linalg/dtype.dart'; + +abstract class LinkFunctionFactory { + LinkFunction createByType(LinkFunctionType type, { + DType dtype = DType.float32, + }); +} diff --git a/lib/src/link_function/link_function_factory_impl.dart b/lib/src/link_function/link_function_factory_impl.dart new file mode 100644 index 00000000..691f6415 --- /dev/null +++ b/lib/src/link_function/link_function_factory_impl.dart @@ -0,0 +1,26 @@ +import 'package:ml_algo/src/link_function/link_function.dart'; +import 'package:ml_algo/src/link_function/link_function_factory.dart'; +import 'package:ml_algo/src/link_function/link_function_type.dart'; +import 'package:ml_algo/src/link_function/logit/inverse_logit_link_function.dart'; +import 'package:ml_algo/src/link_function/softmax/softmax_link_function.dart'; +import 'package:ml_linalg/dtype.dart'; + +class LinkFunctionFactoryImpl implements LinkFunctionFactory { + const LinkFunctionFactoryImpl(); + + @override + LinkFunction createByType(LinkFunctionType type, { + DType dtype = DType.float32, + }) { + switch (type) { + case LinkFunctionType.inverseLogit: + return InverseLogitLinkFunction(dtype); + + case LinkFunctionType.softmax: + return SoftmaxLinkFunction(dtype); + + default: + throw UnsupportedError('Unsupported link function type - $type'); + } + } +} diff --git a/lib/src/link_function/link_function_type.dart b/lib/src/link_function/link_function_type.dart new file mode 100644 index 00000000..26c6f429 --- /dev/null +++ b/lib/src/link_function/link_function_type.dart @@ -0,0 +1,4 @@ +enum LinkFunctionType { + inverseLogit, + softmax, +} diff --git a/lib/src/model_selection/assessable.dart b/lib/src/model_selection/assessable.dart index e2d6f22e..72fee2e7 100644 --- a/lib/src/model_selection/assessable.dart +++ b/lib/src/model_selection/assessable.dart @@ -1,7 +1,9 @@ import 'package:ml_algo/src/metric/metric_type.dart'; -import 'package:ml_linalg/matrix.dart'; +import 'package:ml_dataframe/ml_dataframe.dart'; +import 'package:meta/meta.dart'; abstract class Assessable { - /// Assesses model according to provided [metric] - double assess(Matrix observations, Matrix outcomes, MetricType metric); + /// Assesses model according to provided [metricType] + double assess(DataFrame observations, Iterable targetNames, + MetricType metricType); } diff --git a/lib/src/model_selection/cross_validator/cross_validator.dart b/lib/src/model_selection/cross_validator/cross_validator.dart index 8a99dfdd..6095e3df 100644 --- a/lib/src/model_selection/cross_validator/cross_validator.dart +++ b/lib/src/model_selection/cross_validator/cross_validator.dart @@ -1,10 +1,13 @@ import 'package:ml_algo/src/metric/metric_type.dart'; +import 'package:ml_algo/src/model_selection/assessable.dart'; import 'package:ml_algo/src/model_selection/cross_validator/cross_validator_impl.dart'; import 'package:ml_algo/src/model_selection/data_splitter/k_fold.dart'; import 'package:ml_algo/src/model_selection/data_splitter/leave_p_out.dart'; -import 'package:ml_algo/src/model_selection/assessable.dart'; +import 'package:ml_dataframe/ml_dataframe.dart'; import 'package:ml_linalg/dtype.dart'; -import 'package:ml_linalg/matrix.dart'; + +typedef PredictorFactory = Assessable Function(DataFrame observations, + Iterable targetNames); /// A factory and an interface for all the cross validator types abstract class CrossValidator { @@ -12,17 +15,37 @@ abstract class CrossValidator { /// /// It splits a dataset into [numberOfFolds] test sets and subsequently /// evaluates the predictor on each produced test set - factory CrossValidator.kFold({DType dtype, int numberOfFolds}) => - CrossValidatorImpl(dtype, KFoldSplitter(numberOfFolds)); + factory CrossValidator.kFold( + DataFrame samples, + Iterable targetColumnNames, { + int numberOfFolds = 5, + DType dtype = DType.float32, + }) => + CrossValidatorImpl( + samples, + targetColumnNames, + KFoldSplitter(numberOfFolds), + dtype, + ); /// Creates LPO validator to evaluate quality of a predictor. /// /// It splits a dataset into all possible test sets of size [p] and /// subsequently evaluates quality of the predictor on each produced test set - factory CrossValidator.lpo({DType dtype, int p}) => - CrossValidatorImpl(dtype, LeavePOutSplitter(p)); + factory CrossValidator.lpo( + DataFrame samples, + Iterable targetColumnNames, + int p, { + DType dtype = DType.float32, + }) => + CrossValidatorImpl( + samples, + targetColumnNames, + LeavePOutSplitter(p), + dtype, + ); - /// Returns a score of quality of passed predictor depending on given [metric] - double evaluate(Assessable predictorFactory(Matrix features, Matrix outcomes), - Matrix observations, Matrix outcomes, MetricType metric); + /// Returns a score of quality of passed predictor depending on given + /// [metricType] + double evaluate(PredictorFactory predictorFactory, MetricType metricType); } diff --git a/lib/src/model_selection/cross_validator/cross_validator_impl.dart b/lib/src/model_selection/cross_validator/cross_validator_impl.dart index bbe83a52..cf6dde18 100644 --- a/lib/src/model_selection/cross_validator/cross_validator_impl.dart +++ b/lib/src/model_selection/cross_validator/cross_validator_impl.dart @@ -1,66 +1,64 @@ import 'package:ml_algo/src/metric/metric_type.dart'; import 'package:ml_algo/src/model_selection/cross_validator/cross_validator.dart'; import 'package:ml_algo/src/model_selection/data_splitter/splitter.dart'; -import 'package:ml_algo/src/model_selection/assessable.dart'; -import 'package:ml_algo/src/utils/default_parameter_values.dart'; +import 'package:ml_dataframe/ml_dataframe.dart'; import 'package:ml_linalg/dtype.dart'; import 'package:ml_linalg/matrix.dart'; import 'package:ml_linalg/vector.dart'; +import 'package:quiver/iterables.dart'; class CrossValidatorImpl implements CrossValidator { - CrossValidatorImpl(DType dtype, this._splitter) - : dtype = dtype ?? DefaultParameterValues.dtype; + CrossValidatorImpl(this.samples, this.targetNames, this._splitter, this.dtype); + final DataFrame samples; final DType dtype; + final Iterable targetNames; final Splitter _splitter; @override - double evaluate(Assessable predictorFactory(Matrix features, Matrix outcomes), - Matrix observations, Matrix labels, MetricType metric) { - if (observations.rowsNum != labels.rowsNum) { - throw Exception( - 'Number of feature objects must be equal to the number of labels!'); - } + double evaluate(PredictorFactory predictorFactory, MetricType metricType) { + final samplesAsMatrix = samples.toMatrix(); + final discreteColumns = enumerate(samples.series) + .where((indexedSeries) => indexedSeries.value.isDiscrete) + .map((indexedSeries) => indexedSeries.index); - final allIndicesGroups = _splitter.split(observations.rowsNum); + final allIndicesGroups = _splitter.split(samplesAsMatrix.rowsNum); var score = 0.0; var folds = 0; - for (final testIndices in allIndicesGroups) { - final testIndicesAsSet = Set.from(testIndices); - final trainFeatures = - List(observations.rowsNum - testIndicesAsSet.length); - final trainLabels = - List(observations.rowsNum - testIndicesAsSet.length); - - final testFeatures = List(testIndicesAsSet.length); - final testLabels = List(testIndicesAsSet.length); + for (final testRowsIndices in allIndicesGroups) { + final testRowsIndicesAsSet = Set.from(testRowsIndices); + final trainSamples = + List(samplesAsMatrix.rowsNum - testRowsIndicesAsSet.length); + final testSamples = List(testRowsIndicesAsSet.length); int trainPointsCounter = 0; int testPointsCounter = 0; - for (int index = 0; index < observations.rowsNum; index++) { - if (testIndicesAsSet.contains(index)) { - testFeatures[testPointsCounter] = observations.getRow(index); - testLabels[testPointsCounter] = labels.getRow(index); - testPointsCounter++; + samplesAsMatrix.rowIndices.forEach((i) { + if (testRowsIndicesAsSet.contains(i)) { + testSamples[testPointsCounter++] = samplesAsMatrix[i]; } else { - trainFeatures[trainPointsCounter] = observations.getRow(index); - trainLabels[trainPointsCounter] = labels.getRow(index); - trainPointsCounter++; + trainSamples[trainPointsCounter++] = samplesAsMatrix[i]; } - } + }); - final predictor = predictorFactory( - Matrix.fromRows(trainFeatures, dtype: dtype), - Matrix.fromRows(trainLabels, dtype: dtype), + final trainingDataFrame = DataFrame.fromMatrix( + Matrix.fromRows(trainSamples), + header: samples.header, + discreteColumns: discreteColumns, ); - score += predictor.assess( - Matrix.fromRows(testFeatures, dtype: dtype), - Matrix.fromRows(testLabels, dtype: dtype), - metric + final testingDataFrame = DataFrame.fromMatrix( + Matrix.fromRows(testSamples), + header: samples.header, + discreteColumns: discreteColumns, ); + + final predictor = predictorFactory(trainingDataFrame, targetNames); + + score += predictor.assess(testingDataFrame, targetNames, metricType); + folds++; } diff --git a/lib/src/model_selection/data_splitter/k_fold.dart b/lib/src/model_selection/data_splitter/k_fold.dart index 2e03208f..85e9001e 100644 --- a/lib/src/model_selection/data_splitter/k_fold.dart +++ b/lib/src/model_selection/data_splitter/k_fold.dart @@ -1,5 +1,5 @@ import 'package:ml_algo/src/model_selection/data_splitter/splitter.dart'; -import 'package:xrange/zrange.dart'; +import 'package:xrange/integers.dart'; class KFoldSplitter implements Splitter { KFoldSplitter(this._numberOfFolds) { @@ -21,10 +21,10 @@ class KFoldSplitter implements Splitter { final remainder = numOfObservations % _numberOfFolds; final foldSize = numOfObservations ~/ _numberOfFolds; for (int i = 0, startIdx = 0, endIdx = 0; i < _numberOfFolds; i++) { - // if we reached last fold of size [foldSize], all the next folds up + // if we reached the last fold of size [foldSize], all the next folds up // to the last fold will have size that is equal to [foldSize] + 1 endIdx = startIdx + foldSize + (i >= _numberOfFolds - remainder ? 1 : 0); - yield ZRange.closedOpen(startIdx, endIdx).values(); + yield integers(startIdx, endIdx, upperClosed: false); startIdx = endIdx; } } diff --git a/lib/src/predictor/assessable_predictor_mixin.dart b/lib/src/predictor/assessable_predictor_mixin.dart new file mode 100644 index 00000000..8eeaf63d --- /dev/null +++ b/lib/src/predictor/assessable_predictor_mixin.dart @@ -0,0 +1,22 @@ +import 'package:ml_algo/src/helpers/features_target_split.dart'; +import 'package:ml_algo/src/metric/factory.dart'; +import 'package:ml_algo/src/metric/metric_type.dart'; +import 'package:ml_algo/src/model_selection/assessable.dart'; +import 'package:ml_algo/src/predictor/predictor.dart'; +import 'package:ml_dataframe/ml_dataframe.dart'; + +mixin AssessablePredictorMixin implements Assessable, Predictor { + @override + double assess(DataFrame samples, Iterable targetNames, + MetricType metricType) { + final splits = featuresTargetSplit(samples, + targetNames: targetNames, + ).toList(); + + final metric = MetricFactory.createByType(metricType); + final prediction = predict(splits[0]); + final origLabels = splits[1].toMatrix(); + + return metric.getScore(prediction.toMatrix(), origLabels); + } +} diff --git a/lib/src/predictor/predictor.dart b/lib/src/predictor/predictor.dart new file mode 100644 index 00000000..1268bb0c --- /dev/null +++ b/lib/src/predictor/predictor.dart @@ -0,0 +1,7 @@ +import 'package:ml_dataframe/ml_dataframe.dart'; + +/// A common interface for all types of classifiers and regressors +abstract class Predictor { + /// Returns prediction, based on the model learned parameters + DataFrame predict(DataFrame features); +} diff --git a/lib/src/regressor/_helpers/squared_cost_optimizer_factory.dart b/lib/src/regressor/_helpers/squared_cost_optimizer_factory.dart new file mode 100644 index 00000000..6ab040e1 --- /dev/null +++ b/lib/src/regressor/_helpers/squared_cost_optimizer_factory.dart @@ -0,0 +1,71 @@ +import 'package:ml_algo/src/cost_function/cost_function_factory.dart'; +import 'package:ml_algo/src/cost_function/cost_function_type.dart'; +import 'package:ml_algo/src/di/injector.dart'; +import 'package:ml_algo/src/helpers/add_intercept_if.dart'; +import 'package:ml_algo/src/helpers/features_target_split.dart'; +import 'package:ml_algo/src/linear_optimizer/gradient_optimizer/learning_rate_generator/learning_rate_type.dart'; +import 'package:ml_algo/src/linear_optimizer/initial_coefficients_generator/initial_coefficients_type.dart'; +import 'package:ml_algo/src/linear_optimizer/linear_optimizer.dart'; +import 'package:ml_algo/src/linear_optimizer/linear_optimizer_factory.dart'; +import 'package:ml_algo/src/linear_optimizer/linear_optimizer_type.dart'; +import 'package:ml_algo/src/linear_optimizer/regularization_type.dart'; +import 'package:ml_dataframe/ml_dataframe.dart'; +import 'package:ml_linalg/dtype.dart'; +import 'package:ml_linalg/matrix.dart'; + +LinearOptimizer createSquaredCostOptimizer( + DataFrame observations, String targetName, { + LinearOptimizerType optimizerType, + int iterationsLimit, + double initialLearningRate, + double minCoefficientsUpdate, + double probabilityThreshold, + double lambda, + RegularizationType regularizationType, + int randomSeed, + int batchSize, + bool fitIntercept, + double interceptScale, + bool isFittingDataNormalized, + LearningRateType learningRateType, + InitialCoefficientsType initialCoefficientsType, + Matrix initialCoefficients, + DType dtype, + }) { + + final splits = featuresTargetSplit(observations, + targetNames: [targetName]).toList(); + + final points = splits[0].toMatrix(); + final labels = splits[1].toMatrix(); + + final dependencies = getDependencies(); + + final optimizerFactory = dependencies + .getDependency(); + + final costFunctionFactory = dependencies + .getDependency(); + + final costFunction = costFunctionFactory.createByType( + CostFunctionType.squared, + ); + + return optimizerFactory.createByType( + optimizerType, + addInterceptIf(fitIntercept, points, interceptScale), + labels, + costFunction: costFunction, + iterationLimit: iterationsLimit, + initialLearningRate: initialLearningRate, + minCoefficientsUpdate: minCoefficientsUpdate, + lambda: lambda, + regularizationType: regularizationType, + randomSeed: randomSeed, + batchSize: batchSize, + learningRateType: learningRateType, + initialCoefficientsType: initialCoefficientsType, + dtype: dtype, + isFittingDataNormalized: isFittingDataNormalized, + ); +} diff --git a/lib/src/regressor/coordinate_regressor.dart b/lib/src/regressor/coordinate_regressor.dart deleted file mode 100644 index a8fa0ecb..00000000 --- a/lib/src/regressor/coordinate_regressor.dart +++ /dev/null @@ -1,60 +0,0 @@ -import 'package:ml_algo/ml_algo.dart'; -import 'package:ml_algo/src/cost_function/squared.dart'; -import 'package:ml_algo/src/helpers/add_intercept_if.dart'; -import 'package:ml_algo/src/metric/factory.dart'; -import 'package:ml_algo/src/metric/metric_type.dart'; -import 'package:ml_algo/src/solver/linear/coordinate/coordinate.dart'; -import 'package:ml_algo/src/solver/linear/initial_weights_generator/initial_weights_type.dart'; -import 'package:ml_algo/src/utils/default_parameter_values.dart'; -import 'package:ml_linalg/dtype.dart'; -import 'package:ml_linalg/linalg.dart'; - -class CoordinateRegressor implements LinearRegressor { - CoordinateRegressor( - Matrix trainingFeatures, - Matrix trainingOutcomes, { - int iterationsLimit = DefaultParameterValues.iterationsLimit, - double minWeightsUpdate = DefaultParameterValues.minCoefficientsUpdate, - double lambda, - bool fitIntercept = false, - double interceptScale = 1.0, - DType dtype = DefaultParameterValues.dtype, - InitialWeightsType initialWeightsType = InitialWeightsType.zeroes, - Matrix initialWeights, - bool isTrainDataNormalized = false, - }) : - _fitIntercept = fitIntercept, - _interceptScale = interceptScale, - coefficients = CoordinateOptimizer( - addInterceptIf(fitIntercept, trainingFeatures, interceptScale), - trainingOutcomes, - initialWeightsType: initialWeightsType, - iterationsLimit: iterationsLimit, - minCoefficientsDiff: minWeightsUpdate, - costFunction: const SquaredCost(), - lambda: lambda, - dtype: dtype, - isTrainDataNormalized: isTrainDataNormalized, - ).findExtrema( - initialWeights: initialWeights, - isMinimizingObjective: true, - ).getRow(0); - - final bool _fitIntercept; - - final double _interceptScale; - - @override - final Vector coefficients; - - @override - double assess(Matrix features, Matrix origLabels, MetricType metricType) { - final metric = MetricFactory.createByType(metricType); - final prediction = predict(features); - return metric.getScore(prediction, origLabels); - } - - @override - Matrix predict(Matrix features) => - addInterceptIf(_fitIntercept, features, _interceptScale) * coefficients; -} diff --git a/lib/src/regressor/gradient_regressor.dart b/lib/src/regressor/gradient_regressor.dart deleted file mode 100644 index c2ab0896..00000000 --- a/lib/src/regressor/gradient_regressor.dart +++ /dev/null @@ -1,66 +0,0 @@ -import 'package:ml_algo/src/cost_function/squared.dart'; -import 'package:ml_algo/src/helpers/add_intercept_if.dart'; -import 'package:ml_algo/src/metric/factory.dart'; -import 'package:ml_algo/src/metric/metric_type.dart'; -import 'package:ml_algo/src/solver/linear/gradient/gradient.dart'; -import 'package:ml_algo/src/solver/linear/gradient/learning_rate_generator/learning_rate_type.dart'; -import 'package:ml_algo/src/solver/linear/initial_weights_generator/initial_weights_type.dart'; -import 'package:ml_algo/src/regressor/linear_regressor.dart'; -import 'package:ml_algo/src/utils/default_parameter_values.dart'; -import 'package:ml_linalg/dtype.dart'; -import 'package:ml_linalg/matrix.dart'; -import 'package:ml_linalg/vector.dart'; - -class GradientRegressor implements LinearRegressor { - GradientRegressor( - Matrix trainingFeatures, - Matrix trainingOutcomes, { - int iterationsLimit = DefaultParameterValues.iterationsLimit, - double initialLearningRate = DefaultParameterValues.initialLearningRate, - double minWeightsUpdate = DefaultParameterValues.minCoefficientsUpdate, - double lambda, - bool fitIntercept = false, - double interceptScale = 1.0, - int randomSeed, - int batchSize = 1, - DType dtype = DefaultParameterValues.dtype, - Matrix initialWeights, - LearningRateType learningRateType = LearningRateType.constant, - InitialWeightsType initialWeightsType = InitialWeightsType.zeroes, - }) : - _fitIntercept = fitIntercept, - _interceptScale = interceptScale, - coefficients = GradientOptimizer( - addInterceptIf(fitIntercept, trainingFeatures, interceptScale), - trainingOutcomes, - costFunction: SquaredCost(), - learningRateType: learningRateType, - initialWeightsType: initialWeightsType, - initialLearningRate: initialLearningRate, - minCoefficientsUpdate: minWeightsUpdate, - iterationLimit: iterationsLimit, - lambda: lambda, - batchSize: batchSize, - randomSeed: randomSeed, - ).findExtrema( - initialWeights: initialWeights?.transpose(), - isMinimizingObjective: true, - ).getColumn(0); - - final bool _fitIntercept; - final double _interceptScale; - - @override - final Vector coefficients; - - @override - double assess(Matrix features, Matrix origLabels, MetricType metricType) { - final metric = MetricFactory.createByType(metricType); - final prediction = predict(features); - return metric.getScore(prediction, origLabels); - } - - @override - Matrix predict(Matrix features) => - addInterceptIf(_fitIntercept, features, _interceptScale) * coefficients; -} diff --git a/lib/src/regressor/knn_regressor.dart b/lib/src/regressor/knn_regressor.dart index f0d18a8f..b1842276 100644 --- a/lib/src/regressor/knn_regressor.dart +++ b/lib/src/regressor/knn_regressor.dart @@ -1,71 +1,66 @@ -import 'package:ml_algo/src/algorithms/knn/kernel.dart'; -import 'package:ml_algo/src/algorithms/knn/kernel_function_factory.dart'; -import 'package:ml_algo/src/algorithms/knn/kernel_function_factory_impl.dart'; import 'package:ml_algo/src/algorithms/knn/kernel_type.dart'; -import 'package:ml_algo/src/algorithms/knn/knn.dart'; -import 'package:ml_algo/src/metric/factory.dart'; -import 'package:ml_algo/src/metric/metric_type.dart'; -import 'package:ml_algo/src/regressor/parameterless_regressor.dart'; -import 'package:ml_algo/src/utils/default_parameter_values.dart'; +import 'package:ml_algo/src/helpers/features_target_split.dart'; +import 'package:ml_algo/src/model_selection/assessable.dart'; +import 'package:ml_algo/src/predictor/predictor.dart'; +import 'package:ml_algo/src/regressor/knn_regressor_impl.dart'; +import 'package:ml_dataframe/ml_dataframe.dart'; import 'package:ml_linalg/distance.dart'; import 'package:ml_linalg/dtype.dart'; -import 'package:ml_linalg/matrix.dart'; -import 'package:ml_linalg/vector.dart'; -class KNNRegressor implements ParameterlessRegressor { - KNNRegressor(this._trainingFeatures, this._trainingOutcomes, { - int k, - Distance distance = Distance.euclidean, - FindKnnFn solverFn = findKNeighbours, - Kernel kernel = Kernel.uniform, - DType dtype = DefaultParameterValues.dtype, +/// A class that performs regression basing on `k nearest neighbours` algorithm +/// +/// K nearest neighbours algorithm is an algorithm that is targeted to search +/// most similar labelled observations (number of these observations equals `k`) +/// for the given unlabelled one. +/// +/// In order to make a prediction, or rather to set a label for a given new +/// observation, labels of found `k` observations are being summed up and +/// divided by `k`. +/// +/// To get a more precise result, one may use weighted average of found labels - +/// the farther a found observation from the target one, the lower the weight of +/// the observation is. To obtain these weights one may use a kernel function. +abstract class KnnRegressor implements Assessable, Predictor { + /// Parameters: + /// + /// [fittingData] Labelled observations, among which will be searched [k] + /// nearest neighbours for unlabelled observations. Must contain [targetName] + /// column. + /// + /// [targetName] A string, that serves as a name of the column, that contains + /// labels (or outcomes). + /// + /// [k] a number of nearest neighbours to be searched among [fittingData] + /// + /// [kernel] a type of a kernel function, that will be used to predict an + /// outcome for a new observation + /// + /// [distance] a distance type, that will be used to measure a distance + /// between two observation vectors + /// + /// [dtype] A data type for all the numeric values, used by the algorithm. Can + /// affect performance or accuracy of the computations. Default value is + /// [DType.float32] + factory KnnRegressor( + DataFrame fittingData, + String targetName, { + int k, + Kernel kernel = Kernel.uniform, + Distance distance = Distance.euclidean, + DType dtype = DType.float32, + }) { + final splits = featuresTargetSplit(fittingData, + targetNames: [targetName], + ).toList(); - KernelFunctionFactory kernelFnFactory = const KernelFunctionFactoryImpl(), - }) : - _k = k, - _distanceType = distance, - _solverFn = solverFn, - _dtype = dtype, - _kernelFn = kernelFnFactory.createByType(kernel) { - if (_trainingFeatures.rowsNum != _trainingOutcomes.rowsNum) { - throw Exception('Number of observations and number of outcomes have to be' - 'equal'); - } - if (_k > _trainingFeatures.rowsNum) { - throw Exception('Parameter k should be less than or equal to the number ' - 'of training observations'); - } - } - - final Matrix _trainingFeatures; - final Matrix _trainingOutcomes; - final Distance _distanceType; - final int _k; - final FindKnnFn _solverFn; - final KernelFn _kernelFn; - final DType _dtype; - - Vector get _zeroVector => _cachedZeroVector ??= Vector.zero( - _trainingOutcomes.columnsNum, dtype: _dtype); - Vector _cachedZeroVector; - - @override - Matrix predict(Matrix observations) => Matrix.fromRows( - _generateOutcomes(observations).toList(growable: false), dtype: _dtype); - - Iterable _generateOutcomes(Matrix observations) sync* { - for (final kNeighbours in _solverFn(_k, _trainingFeatures, _trainingOutcomes, - observations, distance: _distanceType)) { - yield kNeighbours - .fold(_zeroVector, - (sum, pair) => sum + pair.label * _kernelFn(pair.distance)) / _k; - } - } - - @override - double assess(Matrix features, Matrix origLabels, MetricType metricType) { - final metric = MetricFactory.createByType(metricType); - final prediction = predict(features); - return metric.getScore(prediction, origLabels); + return KnnRegressorImpl( + splits[0].toMatrix(), + splits[1].toMatrix(), + targetName, + k: k, + kernel: kernel, + distance: distance, + dtype: dtype, + ); } } diff --git a/lib/src/regressor/knn_regressor_impl.dart b/lib/src/regressor/knn_regressor_impl.dart new file mode 100644 index 00000000..09344af8 --- /dev/null +++ b/lib/src/regressor/knn_regressor_impl.dart @@ -0,0 +1,80 @@ +import 'package:ml_algo/src/algorithms/knn/kernel.dart'; +import 'package:ml_algo/src/algorithms/knn/kernel_function_factory.dart'; +import 'package:ml_algo/src/algorithms/knn/kernel_function_factory_impl.dart'; +import 'package:ml_algo/src/algorithms/knn/kernel_type.dart'; +import 'package:ml_algo/src/algorithms/knn/knn.dart'; +import 'package:ml_algo/src/predictor/assessable_predictor_mixin.dart'; +import 'package:ml_algo/src/regressor/knn_regressor.dart'; +import 'package:ml_dataframe/ml_dataframe.dart'; +import 'package:ml_linalg/distance.dart'; +import 'package:ml_linalg/dtype.dart'; +import 'package:ml_linalg/matrix.dart'; +import 'package:ml_linalg/vector.dart'; + +class KnnRegressorImpl with AssessablePredictorMixin implements KnnRegressor { + KnnRegressorImpl( + this._trainingFeatures, + this._trainingOutcomes, + this._targetName, { + int k, + Distance distance = Distance.euclidean, + FindKnnFn solverFn = findKNeighbours, + Kernel kernel = Kernel.uniform, + DType dtype = DType.float32, + + KernelFunctionFactory kernelFnFactory = + const KernelFunctionFactoryImpl(), + }) : + _k = k, + _distanceType = distance, + _solverFn = solverFn, + _dtype = dtype, + _kernelFn = kernelFnFactory.createByType(kernel) { + if (_trainingFeatures.rowsNum != _trainingOutcomes.rowsNum) { + throw Exception('Number of observations and number of outcomes have to be' + 'equal'); + } + if (_k > _trainingFeatures.rowsNum) { + throw Exception('Parameter k should be less than or equal to the number ' + 'of training observations'); + } + } + + final Matrix _trainingFeatures; + final Matrix _trainingOutcomes; + final String _targetName; + final Distance _distanceType; + final int _k; + final FindKnnFn _solverFn; + final KernelFn _kernelFn; + final DType _dtype; + + Vector get _zeroVector => _cachedZeroVector ??= Vector.zero( + _trainingOutcomes.columnsNum, dtype: _dtype); + Vector _cachedZeroVector; + + @override + DataFrame predict(DataFrame observations) { + final prediction = Matrix.fromRows( + _generateOutcomes(observations.toMatrix()) + .toList(growable: false), + dtype: _dtype, + ); + + return DataFrame.fromMatrix( + prediction, + header: [_targetName], + ); + } + + Iterable _generateOutcomes(Matrix observations) => + _solverFn( + _k, + _trainingFeatures, + _trainingOutcomes, + observations, + distance: _distanceType, + ).map((kNeighbours) => kNeighbours + .fold(_zeroVector, + (sum, pair) => sum + pair.label * _kernelFn(pair.distance)) / _k); +} diff --git a/lib/src/regressor/linear_regressor.dart b/lib/src/regressor/linear_regressor.dart index b05d4a00..13743143 100644 --- a/lib/src/regressor/linear_regressor.dart +++ b/lib/src/regressor/linear_regressor.dart @@ -1,137 +1,156 @@ +import 'package:ml_algo/src/linear_optimizer/gradient_optimizer/learning_rate_generator/learning_rate_type.dart'; +import 'package:ml_algo/src/linear_optimizer/initial_coefficients_generator/initial_coefficients_type.dart'; +import 'package:ml_algo/src/linear_optimizer/linear_optimizer_type.dart'; +import 'package:ml_algo/src/linear_optimizer/regularization_type.dart'; import 'package:ml_algo/src/model_selection/assessable.dart'; -import 'package:ml_algo/src/solver/linear/gradient/learning_rate_generator/learning_rate_type.dart'; -import 'package:ml_algo/src/solver/linear/initial_weights_generator/initial_weights_type.dart'; -import 'package:ml_algo/src/regressor/coordinate_regressor.dart'; -import 'package:ml_algo/src/regressor/gradient_regressor.dart'; -import 'package:ml_algo/src/regressor/regressor.dart'; +import 'package:ml_algo/src/predictor/predictor.dart'; +import 'package:ml_algo/src/regressor/_helpers/squared_cost_optimizer_factory.dart'; +import 'package:ml_algo/src/regressor/linear_regressor_impl.dart'; +import 'package:ml_dataframe/ml_dataframe.dart'; import 'package:ml_linalg/dtype.dart'; import 'package:ml_linalg/matrix.dart'; import 'package:ml_linalg/vector.dart'; -/// A factory for all the linear regressors. +/// A class that performs linear regression /// /// A typical linear regressor uses the equation of a line for multidimensional /// space to make a prediction. Each `x` in the equation has its own coefficient /// (weight) and the combination of these `x`-es and its coefficients gives the -/// `y` term. The latter is a thing, that the regressor should predict, and -/// as one knows all the `x` values (since it is the input for the algorithm), +/// `y` term. The latter is a value, that the regressor should predict, and +/// as all the `x` values are known (since it is the input for the algorithm), /// the regressor should find the best coefficients (weights) (they are unknown) /// to make a best prediction of `y` term. -abstract class LinearRegressor implements Regressor, Assessable { - /** - * Creates a gradient linear regressor. Uses gradient descent solver to - * find the optimal weights - * - * Parameters: - * - * [trainingFeatures] A matrix with observations, that will be used by the - * regressor to learn coefficients of the hyperplane - * - * [trainingOutcomes] A matrix with outcomes (dependant variables) for each - * observation from [trainingFeatures] - * - * [iterationsLimit] A number of fitting iterations. Uses as a condition of - * convergence in the solver. Default - * value is `100` - * - * [initialLearningRate] A value, defining velocity of the convergence of the - * gradient descent solver. - * - * [minWeightsUpdate] A minimum distance between weights vectors in two - * subsequent iterations. Uses as a condition of convergence in the solver. - * In other words, if difference is small, there is no reason to continue - * fitting. Default value is `1e-12` - * - * [lambda] A coefficient for regularization. For this particular regressor - * there is only L2 regularization type. - * - * [randomSeed] A seed, that will be passed to a random value generator, used - * by stochastic optimizers. Will be ignored, if the solver is not - * stochastic. Remember, each time you run the regressor with the same - * parameters, you will receive a different result. To avoid it, define - * [randomSeed] - * - * [batchSize] A size of data (in rows), that will be used for fitting per - * one iteration. - * - * [fitIntercept] Whether or not to fit intercept term. Default value is - * `false`. - * - * [interceptScale] A value, defining a size of the intercept term - * - * [learningRateType] A value, defining a strategy for the learning rate - * behaviour throughout the whole fitting process - * - * [dtype] A data type for all the numeric values, used by the algorithm. - * Can affect performance or accuracy of the computations. Default value is - * [Float32x4] - */ - factory LinearRegressor.gradient( - Matrix trainingFeatures, - Matrix trainingOutcomes, { - int iterationsLimit, - LearningRateType learningRateType, - InitialWeightsType initialWeightsType, - double initialLearningRate, - double minWeightsUpdate, - double lambda, - bool fitIntercept, - double interceptScale, - int randomSeed, - int batchSize, - Matrix initialWeights, - DType dtype, - }) = GradientRegressor; +abstract class LinearRegressor implements Assessable, Predictor { + /// Parameters: + /// + /// [fittingData] A [DataFrame] with observations, that will be used by the + /// regressor to learn coefficients of the predicting hyperplane. Must contain + /// [targetName] column. + /// + /// [targetName] A string, that serves as a name of the target column, that + /// contains labels or outcomes. + /// + /// [optimizerType] Defines an algorithm of optimization, that will be used + /// to find the best coefficients of a cost function. Also defines, which + /// regularization type (L1 or L2) one may use to learn a linear regressor. + /// By default - [LinearOptimizerType.gradient] + /// + /// [iterationsLimit] A number of fitting iterations. Uses as a condition of + /// convergence in the optimization algorithm. Default value is `100`. + /// + /// [initialLearningRate] A value, defining velocity of the convergence of the + /// gradient descent optimizer. Default value is `1e-3`. + /// + /// [minCoefficientsUpdate] A minimum distance between coefficient vectors in + /// two contiguous iterations. Uses as a condition of convergence in the + /// optimization algorithm. If difference between the two vectors is small + /// enough, there is no reason to continue fitting. Default value is `1e-12` + /// + /// [lambda] A coefficient of regularization. Uses to prevent the regressor's + /// overfitting. The more the value of [lambda], the more regular the + /// coefficients of the predicting hyperplane are. Extremely large [lambda] + /// may decrease the coefficients to nothing, otherwise too small [lambda] may + /// be a cause of too large absolute values of the coefficients. + /// + /// [regularizationType] A way the coefficients of the regressor will be + /// regularized to prevent a model overfitting. + /// + /// [randomSeed] A seed, that will be passed to a random value generator, + /// used by stochastic optimizers. Will be ignored, if the solver cannot be + /// stochastic. Remember, each time you run the stochastic regressor with the + /// same parameters but with unspecified [randomSeed], you will receive + /// different results. To avoid it, define [randomSeed] + /// + /// [batchSize] A size of data (in rows), that will be used for fitting per + /// one iteration. Applicable not for all optimizers. If gradient-based + /// optimizer is used and If [batchSize] == `1`, stochastic mode will be + /// activated; if `1` < [batchSize] < `total number of rows`, mini-batch mode + /// will be activated; if [batchSize] == `total number of rows`, full-batch + /// mode will be activated. + /// + /// [fitIntercept] Whether or not to fit intercept term. Default value is + /// `false`. Intercept in 2-dimensional space is a bias of the line (relative + /// to X-axis) to be learned by the regressor + /// + /// [interceptScale] A value, defining a size of the intercept. + /// + /// [isFittingDataNormalized] Defines, whether the [fittingData] normalized + /// or not. Normalization should be performed column-wise. Normalized data + /// may be needed for some optimizers (e.g., for + /// [LinearOptimizerType.coordinate]) + /// + /// [learningRateType] A value, defining a strategy for the learning rate + /// behaviour throughout the whole fitting process. + /// + /// [initialCoefficientsType] Defines the coefficients, that will be + /// autogenerated before the first iteration of optimization. By default, + /// all the autogenerated coefficients are equal to zeroes at the start. + /// If [initialCoefficients] are provided, the parameter will be ignored + /// + /// [initialCoefficients] Coefficients to be used in the first iteration of + /// optimization algorithm. [initialCoefficients] is a vector, length of which + /// must be equal to the number of features in [fittingData]: the number of + /// features is equal to the number of columns in [fittingData] minus 1 + /// (target column). + /// + /// [dtype] A data type for all the numeric values, used by the algorithm. Can + /// affect performance or accuracy of the computations. Default value is + /// [DType.float32] + factory LinearRegressor(DataFrame fittingData, String targetName, { + LinearOptimizerType optimizerType = LinearOptimizerType.gradient, + int iterationsLimit = 100, + LearningRateType learningRateType = LearningRateType.constant, + InitialCoefficientsType initialCoefficientsType = + InitialCoefficientsType.zeroes, + double initialLearningRate = 1e-3, + double minCoefficientsUpdate = 1e-12, + double lambda, + RegularizationType regularizationType, + bool fitIntercept = false, + double interceptScale = 1.0, + int randomSeed, + int batchSize = 1, + Matrix initialCoefficients, + bool isFittingDataNormalized = false, + DType dtype = DType.float32, + }) { + final optimizer = createSquaredCostOptimizer( + fittingData, + targetName, + optimizerType: optimizerType, + iterationsLimit: iterationsLimit, + initialLearningRate: initialLearningRate, + minCoefficientsUpdate: minCoefficientsUpdate, + lambda: lambda, + regularizationType: regularizationType, + randomSeed: randomSeed, + batchSize: batchSize, + learningRateType: learningRateType, + initialCoefficientsType: initialCoefficientsType, + fitIntercept: fitIntercept, + interceptScale: interceptScale, + isFittingDataNormalized: isFittingDataNormalized, + dtype: dtype, + ); - /** - * Creates a linear regressor with a coordinate descent solver to use L1 - * regularization. - * - * L1 regularization allows to select more important features, make less - * important as zeroes. - * - * Parameters: - * - * [trainingFeatures] A matrix with observations, that will be used by the - * regressor to learn coefficients of the hyperplane - * - * [trainingOutcomes] A matrix with outcomes (dependant variables) for each - * observation from [trainingFeatures] - * - * [iterationsLimit] A number of fitting iterations. Uses as a condition of - * convergence in the solver. Default value is `100` - * - * [minWeightsUpdate] A minimum distance between weights vectors in two - * subsequent iterations. Uses as a condition of convergence in the solver. - * In other words, if difference is small, there is no reason to continue - * fitting. Default value is `1e-12` - * - * [lambda] A coefficient for regularization. For this particular regressor - * there is only L1 regularization type. - * - * [fitIntercept] Whether or not to fit intercept term. Default value is - * `false`. - * - * [interceptScale] A value, defining a size of the intercept term - * - * [dtype] A data type for all the numeric values, used by the algorithm. - * Can affect performance or accuracy of the computations. Default value is - * [Float32x4] - */ - factory LinearRegressor.coordinate( - Matrix trainingFeatures, - Matrix trainingOutcomes, { - int iterationsLimit, - double minWeightsUpdate, - double lambda, - bool fitIntercept, - double interceptScale, - InitialWeightsType initialWeightsType, - DType dtype, - Matrix initialWeights, - bool isTrainDataNormalized, - }) = CoordinateRegressor; + final coefficients = optimizer.findExtrema( + initialCoefficients: initialCoefficients, + isMinimizingObjective: true, + ).getColumn(0); + + return LinearRegressorImpl( + coefficients, + targetName, + fitIntercept: fitIntercept, + interceptScale: interceptScale, + dtype: dtype, + ); + } /// Learned coefficients (or weights) for given features Vector get coefficients; + + bool get fitIntercept; + + double get interceptScale; } diff --git a/lib/src/regressor/linear_regressor_impl.dart b/lib/src/regressor/linear_regressor_impl.dart new file mode 100644 index 00000000..e5199149 --- /dev/null +++ b/lib/src/regressor/linear_regressor_impl.dart @@ -0,0 +1,47 @@ +import 'package:ml_algo/src/helpers/add_intercept_if.dart'; +import 'package:ml_algo/src/predictor/assessable_predictor_mixin.dart'; +import 'package:ml_algo/src/regressor/linear_regressor.dart'; +import 'package:ml_dataframe/ml_dataframe.dart'; +import 'package:ml_linalg/dtype.dart'; +import 'package:ml_linalg/matrix.dart'; +import 'package:ml_linalg/vector.dart'; + +class LinearRegressorImpl with AssessablePredictorMixin + implements LinearRegressor { + + LinearRegressorImpl(this.coefficients, this._targetName, { + bool fitIntercept = false, + double interceptScale = 1.0, + Matrix initialCoefficients, + this.dtype = DType.float32, + }) : + fitIntercept = fitIntercept, + interceptScale = interceptScale; + + final String _targetName; + + @override + final bool fitIntercept; + + @override + final double interceptScale; + + @override + final Vector coefficients; + + final DType dtype; + + @override + DataFrame predict(DataFrame features) { + final prediction = addInterceptIf( + fitIntercept, + features.toMatrix(), + interceptScale, + ) * coefficients; + + return DataFrame.fromMatrix( + prediction, + header: [_targetName], + ); + } +} diff --git a/lib/src/regressor/parameterless_regressor.dart b/lib/src/regressor/parameterless_regressor.dart deleted file mode 100644 index fb7b4b7c..00000000 --- a/lib/src/regressor/parameterless_regressor.dart +++ /dev/null @@ -1,27 +0,0 @@ -import 'package:ml_algo/src/algorithms/knn/kernel_type.dart'; -import 'package:ml_algo/src/model_selection/assessable.dart'; -import 'package:ml_algo/src/regressor/knn_regressor.dart'; -import 'package:ml_algo/src/regressor/regressor.dart'; -import 'package:ml_linalg/distance.dart'; -import 'package:ml_linalg/matrix.dart'; - -/// A factory for all the non parametric family of Machine Learning algorithms -abstract class ParameterlessRegressor implements Regressor, Assessable { - /// Creates an instance of KNN regressor - /// - /// KNN here means "K nearest neighbor" - /// - /// [k] a number of nearest neighbours - /// - /// [kernel] a type of kernel function, that will be used to find an outcome - /// for a new observation - /// - /// [distance] a distance type, that will be used to measure a distance - /// between two observation vectors - factory ParameterlessRegressor.knn(Matrix trainingFeatures, - Matrix trainingOutcomes, { - int k, - Kernel kernel, - Distance distance, - }) = KNNRegressor; -} diff --git a/lib/src/regressor/regressor.dart b/lib/src/regressor/regressor.dart deleted file mode 100644 index b63da06c..00000000 --- a/lib/src/regressor/regressor.dart +++ /dev/null @@ -1,6 +0,0 @@ -import 'package:ml_linalg/matrix.dart'; - -abstract class Regressor { - /// Returns prediction based on the model learned parameters - Matrix predict(Matrix features); -} diff --git a/lib/src/solver/linear/convergence_detector/convergence_detector_factory_impl.dart b/lib/src/solver/linear/convergence_detector/convergence_detector_factory_impl.dart deleted file mode 100644 index ffb654b3..00000000 --- a/lib/src/solver/linear/convergence_detector/convergence_detector_factory_impl.dart +++ /dev/null @@ -1,11 +0,0 @@ -import 'package:ml_algo/src/solver/linear/convergence_detector/convergence_detector.dart'; -import 'package:ml_algo/src/solver/linear/convergence_detector/convergence_detector_factory.dart'; -import 'package:ml_algo/src/solver/linear/convergence_detector/convergence_detector_impl.dart'; - -class ConvergenceDetectorFactoryImpl implements ConvergenceDetectorFactory { - const ConvergenceDetectorFactoryImpl(); - - @override - ConvergenceDetector create(double minUpdate, int iterationsLimit) => - ConvergenceDetectorImpl(minUpdate, iterationsLimit); -} diff --git a/lib/src/solver/linear/convergence_detector/convergence_detector_impl.dart b/lib/src/solver/linear/convergence_detector/convergence_detector_impl.dart deleted file mode 100644 index 6355a266..00000000 --- a/lib/src/solver/linear/convergence_detector/convergence_detector_impl.dart +++ /dev/null @@ -1,30 +0,0 @@ -import 'package:ml_algo/src/solver/linear/convergence_detector/convergence_detector.dart'; - -class ConvergenceDetectorImpl implements ConvergenceDetector { - ConvergenceDetectorImpl(this.minDiff, this.iterationsLimit) { - if (minDiff == null && iterationsLimit == null) { - throw Exception('Neither minimum coefficients diff, nor iteration limit' - 'are specified. Please, provide `minDiff` or `iterationsLimit` ' - 'parameters as a convergence criteria'); - } - } - - @override - final double minDiff; - - @override - final int iterationsLimit; - - @override - bool isConverged(double coefficientsDiff, int iteration) { - if (iterationsLimit != null && iteration >= iterationsLimit) { - return true; - } - - if (minDiff != null && coefficientsDiff <= minDiff) { - return true; - } - - return false; - } -} diff --git a/lib/src/solver/linear/gradient/learning_rate_generator/learning_rate_generator_factory.dart b/lib/src/solver/linear/gradient/learning_rate_generator/learning_rate_generator_factory.dart deleted file mode 100644 index 6a1d11ba..00000000 --- a/lib/src/solver/linear/gradient/learning_rate_generator/learning_rate_generator_factory.dart +++ /dev/null @@ -1,8 +0,0 @@ -import 'package:ml_algo/src/solver/linear/gradient/learning_rate_generator/learning_rate_generator.dart'; -import 'package:ml_algo/src/solver/linear/gradient/learning_rate_generator/learning_rate_type.dart'; - -abstract class LearningRateGeneratorFactory { - LearningRateGenerator decreasing(); - LearningRateGenerator constant(); - LearningRateGenerator fromType(LearningRateType type); -} diff --git a/lib/src/solver/linear/gradient/learning_rate_generator/learning_rate_generator_factory_impl.dart b/lib/src/solver/linear/gradient/learning_rate_generator/learning_rate_generator_factory_impl.dart deleted file mode 100644 index 466b100e..00000000 --- a/lib/src/solver/linear/gradient/learning_rate_generator/learning_rate_generator_factory_impl.dart +++ /dev/null @@ -1,27 +0,0 @@ -import 'package:ml_algo/src/solver/linear/gradient/learning_rate_generator/constant.dart'; -import 'package:ml_algo/src/solver/linear/gradient/learning_rate_generator/decreasing.dart'; -import 'package:ml_algo/src/solver/linear/gradient/learning_rate_generator/learning_rate_generator.dart'; -import 'package:ml_algo/src/solver/linear/gradient/learning_rate_generator/learning_rate_generator_factory.dart'; -import 'package:ml_algo/src/solver/linear/gradient/learning_rate_generator/learning_rate_type.dart'; - -class LearningRateGeneratorFactoryImpl implements LearningRateGeneratorFactory { - const LearningRateGeneratorFactoryImpl(); - - @override - LearningRateGenerator decreasing() => DecreasingLearningRateGenerator(); - - @override - LearningRateGenerator constant() => ConstantLearningRateGenerator(); - - @override - LearningRateGenerator fromType(LearningRateType type) { - switch (type) { - case LearningRateType.constant: - return constant(); - case LearningRateType.decreasing: - return decreasing(); - default: - throw UnimplementedError(); - } - } -} diff --git a/lib/src/solver/linear/gradient/learning_rate_generator/learning_rate_type.dart b/lib/src/solver/linear/gradient/learning_rate_generator/learning_rate_type.dart deleted file mode 100644 index 51f2d5b5..00000000 --- a/lib/src/solver/linear/gradient/learning_rate_generator/learning_rate_type.dart +++ /dev/null @@ -1,7 +0,0 @@ -/// A type of learning rate behaviour -/// -/// [LearningRateType.decreasing] Learning rate will decrease every iteration -/// -/// [LearningRateType.constant] Learning rate will be constant throughout the -/// whole fitting process -enum LearningRateType { decreasing, constant } diff --git a/lib/src/solver/linear/initial_weights_generator/initial_weights_generator_factory.dart b/lib/src/solver/linear/initial_weights_generator/initial_weights_generator_factory.dart deleted file mode 100644 index 05c3fa2b..00000000 --- a/lib/src/solver/linear/initial_weights_generator/initial_weights_generator_factory.dart +++ /dev/null @@ -1,8 +0,0 @@ -import 'package:ml_algo/src/solver/linear/initial_weights_generator/initial_weights_generator.dart'; -import 'package:ml_algo/src/solver/linear/initial_weights_generator/initial_weights_type.dart'; -import 'package:ml_linalg/dtype.dart'; - -abstract class InitialWeightsGeneratorFactory { - InitialWeightsGenerator zeroes(DType dtype); - InitialWeightsGenerator fromType(InitialWeightsType type, DType dtype); -} diff --git a/lib/src/solver/linear/initial_weights_generator/initial_weights_generator_factory_impl.dart b/lib/src/solver/linear/initial_weights_generator/initial_weights_generator_factory_impl.dart deleted file mode 100644 index 98233740..00000000 --- a/lib/src/solver/linear/initial_weights_generator/initial_weights_generator_factory_impl.dart +++ /dev/null @@ -1,23 +0,0 @@ -import 'package:ml_algo/src/solver/linear/initial_weights_generator/initial_weights_generator.dart'; -import 'package:ml_algo/src/solver/linear/initial_weights_generator/initial_weights_generator_factory.dart'; -import 'package:ml_algo/src/solver/linear/initial_weights_generator/initial_weights_type.dart'; -import 'package:ml_algo/src/solver/linear/initial_weights_generator/zero_weights_generator.dart'; -import 'package:ml_linalg/dtype.dart'; - -class InitialWeightsGeneratorFactoryImpl - implements InitialWeightsGeneratorFactory { - const InitialWeightsGeneratorFactoryImpl(); - - @override - InitialWeightsGenerator zeroes(DType dtype) => ZeroWeightsGenerator(dtype); - - @override - InitialWeightsGenerator fromType(InitialWeightsType type, DType dtype) { - switch (type) { - case InitialWeightsType.zeroes: - return zeroes(dtype); - default: - throw UnsupportedError('Unsupported initial weights type - $type'); - } - } -} diff --git a/lib/src/solver/linear/initial_weights_generator/initial_weights_type.dart b/lib/src/solver/linear/initial_weights_generator/initial_weights_type.dart deleted file mode 100644 index 4f4725de..00000000 --- a/lib/src/solver/linear/initial_weights_generator/initial_weights_type.dart +++ /dev/null @@ -1,3 +0,0 @@ -enum InitialWeightsType { - zeroes, -} diff --git a/lib/src/solver/linear/initial_weights_generator/zero_weights_generator.dart b/lib/src/solver/linear/initial_weights_generator/zero_weights_generator.dart deleted file mode 100644 index e0b8c38e..00000000 --- a/lib/src/solver/linear/initial_weights_generator/zero_weights_generator.dart +++ /dev/null @@ -1,12 +0,0 @@ -import 'package:ml_algo/src/solver/linear/initial_weights_generator/initial_weights_generator.dart'; -import 'package:ml_linalg/dtype.dart'; -import 'package:ml_linalg/linalg.dart'; - -class ZeroWeightsGenerator implements InitialWeightsGenerator { - ZeroWeightsGenerator(this.dtype); - - final DType dtype; - - @override - Vector generate(int length) => Vector.zero(length, dtype: dtype); -} diff --git a/lib/src/solver/linear/linear_optimizer.dart b/lib/src/solver/linear/linear_optimizer.dart deleted file mode 100644 index 4d01d01c..00000000 --- a/lib/src/solver/linear/linear_optimizer.dart +++ /dev/null @@ -1,11 +0,0 @@ -import 'package:ml_linalg/matrix.dart'; - -abstract class LinearOptimizer { - /// [initialWeights] initial weights (coefficients) to start optimization (e.g. random values) - /// - /// [isMinimizingObjective] should the solver find a maxima or minima - Matrix findExtrema({ - Matrix initialWeights, - bool isMinimizingObjective, - }); -} diff --git a/lib/src/solver/linear/linear_optimizer_factory.dart b/lib/src/solver/linear/linear_optimizer_factory.dart deleted file mode 100644 index aea94b7d..00000000 --- a/lib/src/solver/linear/linear_optimizer_factory.dart +++ /dev/null @@ -1,37 +0,0 @@ -import 'package:ml_algo/src/cost_function/cost_function.dart'; -import 'package:ml_algo/src/math/randomizer/randomizer_factory.dart'; -import 'package:ml_algo/src/solver/linear/gradient/learning_rate_generator/learning_rate_generator_factory.dart'; -import 'package:ml_algo/src/solver/linear/gradient/learning_rate_generator/learning_rate_type.dart'; -import 'package:ml_algo/src/solver/linear/initial_weights_generator/initial_weights_generator_factory.dart'; -import 'package:ml_algo/src/solver/linear/initial_weights_generator/initial_weights_type.dart'; -import 'package:ml_algo/src/solver/linear/linear_optimizer.dart'; -import 'package:ml_linalg/dtype.dart'; -import 'package:ml_linalg/matrix.dart'; - -abstract class LinearOptimizerFactory { - LinearOptimizer gradient(Matrix points, Matrix labels, { - DType dtype, - CostFunction costFunction, - RandomizerFactory randomizerFactory, - LearningRateGeneratorFactory learningRateGeneratorFactory, - InitialWeightsGeneratorFactory initialWeightsGeneratorFactory, - LearningRateType learningRateType, - InitialWeightsType initialWeightsType, - double initialLearningRate, - double minCoefficientsUpdate, - int iterationLimit, - double lambda, - int batchSize, - int randomSeed, - }); - - LinearOptimizer coordinate(Matrix points, Matrix labels, { - DType dtype, - InitialWeightsGeneratorFactory initialWeightsGeneratorFactory, - CostFunction costFunction, - double minCoefficientsDiff, - int iterationLimit, - double lambda, - InitialWeightsType initialWeightsType, - }); -} diff --git a/lib/src/solver/linear/linear_optimizer_factory_impl.dart b/lib/src/solver/linear/linear_optimizer_factory_impl.dart deleted file mode 100644 index 9533dc81..00000000 --- a/lib/src/solver/linear/linear_optimizer_factory_impl.dart +++ /dev/null @@ -1,76 +0,0 @@ -import 'package:ml_algo/src/cost_function/cost_function.dart'; -import 'package:ml_algo/src/math/randomizer/randomizer_factory.dart'; -import 'package:ml_algo/src/math/randomizer/randomizer_factory_impl.dart'; -import 'package:ml_algo/src/solver/linear/coordinate/coordinate.dart'; -import 'package:ml_algo/src/solver/linear/gradient/gradient.dart'; -import 'package:ml_algo/src/solver/linear/gradient/learning_rate_generator/learning_rate_generator_factory.dart'; -import 'package:ml_algo/src/solver/linear/gradient/learning_rate_generator/learning_rate_generator_factory_impl.dart'; -import 'package:ml_algo/src/solver/linear/gradient/learning_rate_generator/learning_rate_type.dart'; -import 'package:ml_algo/src/solver/linear/initial_weights_generator/initial_weights_generator_factory.dart'; -import 'package:ml_algo/src/solver/linear/initial_weights_generator/initial_weights_generator_factory_impl.dart'; -import 'package:ml_algo/src/solver/linear/initial_weights_generator/initial_weights_type.dart'; -import 'package:ml_algo/src/solver/linear/linear_optimizer.dart'; -import 'package:ml_algo/src/solver/linear/linear_optimizer_factory.dart'; -import 'package:ml_algo/src/utils/default_parameter_values.dart'; -import 'package:ml_linalg/dtype.dart'; -import 'package:ml_linalg/matrix.dart'; - -class LinearOptimizerFactoryImpl implements LinearOptimizerFactory { - const LinearOptimizerFactoryImpl(); - - @override - LinearOptimizer coordinate(Matrix points, Matrix labels, { - DType dtype = DefaultParameterValues.dtype, - InitialWeightsGeneratorFactory initialWeightsGeneratorFactory = - const InitialWeightsGeneratorFactoryImpl(), - CostFunction costFunction, - double minCoefficientsDiff, - int iterationLimit, - double lambda, - InitialWeightsType initialWeightsType, - }) => - CoordinateOptimizer( - points, labels, - dtype: dtype, - initialWeightsGeneratorFactory: initialWeightsGeneratorFactory, - costFunction: costFunction, - minCoefficientsDiff: minCoefficientsDiff, - iterationsLimit: iterationLimit, - lambda: lambda, - initialWeightsType: initialWeightsType, - ); - - @override - LinearOptimizer gradient(Matrix points, Matrix labels, { - DType dtype = DefaultParameterValues.dtype, - RandomizerFactory randomizerFactory = const RandomizerFactoryImpl(), - CostFunction costFunction, - LearningRateGeneratorFactory learningRateGeneratorFactory = - const LearningRateGeneratorFactoryImpl(), - InitialWeightsGeneratorFactory initialWeightsGeneratorFactory = - const InitialWeightsGeneratorFactoryImpl(), - LearningRateType learningRateType, - InitialWeightsType initialWeightsType, - double initialLearningRate, - double minCoefficientsUpdate, - int iterationLimit, - double lambda, - int batchSize, - int randomSeed, - }) => - GradientOptimizer( - points, labels, - randomizerFactory: randomizerFactory, - costFunction: costFunction, - learningRateGeneratorFactory: learningRateGeneratorFactory, - initialWeightsGeneratorFactory: initialWeightsGeneratorFactory, - learningRateType: learningRateType, - initialWeightsType: initialWeightsType, - initialLearningRate: initialLearningRate, - minCoefficientsUpdate: minCoefficientsUpdate, - iterationLimit: iterationLimit, - lambda: lambda, - batchSize: batchSize, - randomSeed: randomSeed, - ); -} diff --git a/lib/src/utils/default_parameter_values.dart b/lib/src/utils/default_parameter_values.dart deleted file mode 100644 index 68cb62f5..00000000 --- a/lib/src/utils/default_parameter_values.dart +++ /dev/null @@ -1,8 +0,0 @@ -import 'package:ml_linalg/dtype.dart'; - -abstract class DefaultParameterValues { - static const dtype = DType.float32; - static const iterationsLimit = 100; - static const minCoefficientsUpdate = 1e-12; - static const initialLearningRate = 1e-3; -} diff --git a/pubspec.lock b/pubspec.lock index 0444102a..f5286e5f 100644 --- a/pubspec.lock +++ b/pubspec.lock @@ -7,7 +7,7 @@ packages: name: analyzer url: "https://pub.dartlang.org" source: hosted - version: "0.38.2" + version: "0.38.4" args: dependency: transitive description: @@ -21,7 +21,7 @@ packages: name: async url: "https://pub.dartlang.org" source: hosted - version: "2.3.0" + version: "2.4.0" benchmark_harness: dependency: "direct dev" description: @@ -42,7 +42,7 @@ packages: name: build url: "https://pub.dartlang.org" source: hosted - version: "1.1.6" + version: "1.2.0" build_cli_annotations: dependency: transitive description: @@ -70,28 +70,28 @@ packages: name: build_resolvers url: "https://pub.dartlang.org" source: hosted - version: "1.0.7" + version: "1.1.1" build_runner: dependency: "direct dev" description: name: build_runner url: "https://pub.dartlang.org" source: hosted - version: "1.7.0" + version: "1.7.1" build_runner_core: dependency: transitive description: name: build_runner_core url: "https://pub.dartlang.org" source: hosted - version: "4.0.0" + version: "4.1.0" build_test: dependency: "direct dev" description: name: build_test url: "https://pub.dartlang.org" source: hosted - version: "0.10.8" + version: "0.10.9" built_collection: dependency: transitive description: @@ -154,14 +154,14 @@ packages: name: coverage url: "https://pub.dartlang.org" source: hosted - version: "0.13.2" + version: "0.13.3" coveralls: dependency: transitive description: name: coveralls url: "https://pub.dartlang.org" source: hosted - version: "5.4.0" + version: "5.6.0" crypto: dependency: transitive description: @@ -189,7 +189,7 @@ packages: name: dart_style url: "https://pub.dartlang.org" source: hosted - version: "1.2.10" + version: "1.3.1" fixnum: dependency: transitive description: @@ -203,7 +203,7 @@ packages: name: front_end url: "https://pub.dartlang.org" source: hosted - version: "0.1.24" + version: "0.1.26" glob: dependency: transitive description: @@ -238,7 +238,7 @@ packages: name: html url: "https://pub.dartlang.org" source: hosted - version: "0.14.0+2" + version: "0.14.0+3" http: dependency: transitive description: @@ -260,6 +260,13 @@ packages: url: "https://pub.dartlang.org" source: hosted version: "3.1.3" + injector: + dependency: "direct main" + description: + name: injector + url: "https://pub.dartlang.org" + source: hosted + version: "1.0.8" io: dependency: transitive description: @@ -287,7 +294,7 @@ packages: name: kernel url: "https://pub.dartlang.org" source: hosted - version: "0.3.24" + version: "0.3.26" lcov: dependency: transitive description: @@ -329,14 +336,14 @@ packages: name: ml_dataframe url: "https://pub.dartlang.org" source: hosted - version: "0.0.4" + version: "0.0.11" ml_linalg: dependency: "direct main" description: name: ml_linalg url: "https://pub.dartlang.org" source: hosted - version: "11.0.0" + version: "12.1.0" ml_tech: dependency: "direct dev" description: @@ -518,21 +525,21 @@ packages: name: test url: "https://pub.dartlang.org" source: hosted - version: "1.6.10" + version: "1.8.0" test_api: dependency: transitive description: name: test_api url: "https://pub.dartlang.org" source: hosted - version: "0.2.7" + version: "0.2.8" test_core: dependency: transitive description: name: test_core url: "https://pub.dartlang.org" source: hosted - version: "0.2.9+1" + version: "0.2.10" timing: dependency: transitive description: @@ -540,13 +547,6 @@ packages: url: "https://pub.dartlang.org" source: hosted version: "0.1.1+2" - tuple: - dependency: "direct main" - description: - name: tuple - url: "https://pub.dartlang.org" - source: hosted - version: "1.0.2" typed_data: dependency: transitive description: @@ -560,14 +560,7 @@ packages: name: vm_service url: "https://pub.dartlang.org" source: hosted - version: "2.0.0" - vm_service_lib: - dependency: transitive - description: - name: vm_service_lib - url: "https://pub.dartlang.org" - source: hosted - version: "3.22.2+1" + version: "1.2.0" watcher: dependency: transitive description: @@ -602,13 +595,13 @@ packages: name: xrange url: "https://pub.dartlang.org" source: hosted - version: "0.0.6" + version: "0.0.8" yaml: dependency: transitive description: name: yaml url: "https://pub.dartlang.org" source: hosted - version: "2.1.16" + version: "2.2.0" sdks: - dart: ">=2.4.0 <3.0.0" + dart: ">=2.5.0 <3.0.0" diff --git a/pubspec.yaml b/pubspec.yaml index a06e0510..10d4a66a 100644 --- a/pubspec.yaml +++ b/pubspec.yaml @@ -5,14 +5,14 @@ author: Ilia Gyrdymov homepage: https://github.com/gyrdym/ml_algo environment: - sdk: '>=2.3.0 <3.0.0' + sdk: '>=2.4.1 <3.0.0' dependencies: - ml_dataframe: ^0.0.4 - ml_linalg: ^11.0.0 + injector: ^1.0.8 + ml_dataframe: 0.0.11 + ml_linalg: ^12.1.0 quiver: ^2.0.2 - tuple: ^1.0.2 - xrange: 0.0.6 + xrange: 0.0.8 dev_dependencies: benchmark_harness: '>=1.0.0 <2.0.0' diff --git a/test/classifier/non_linear/decision_tree/decision_tree_classifier_impl_test.dart b/test/classifier/decision_tree_classifier_impl_test.dart similarity index 67% rename from test/classifier/non_linear/decision_tree/decision_tree_classifier_impl_test.dart rename to test/classifier/decision_tree_classifier_impl_test.dart index bf9be535..7b1d1458 100644 --- a/test/classifier/non_linear/decision_tree/decision_tree_classifier_impl_test.dart +++ b/test/classifier/decision_tree_classifier_impl_test.dart @@ -1,13 +1,14 @@ -import 'package:ml_algo/src/classifier/non_linear/decision_tree/decision_tree_classifier_impl.dart'; -import 'package:ml_algo/src/solver/non_linear/decision_tree/decision_tree_leaf_label.dart'; -import 'package:ml_algo/src/solver/non_linear/decision_tree/decision_tree_solver.dart'; +import 'package:ml_algo/src/classifier/decision_tree_classifier_impl.dart'; +import 'package:ml_algo/src/decision_tree_solver/decision_tree_leaf_label.dart'; +import 'package:ml_algo/src/decision_tree_solver/decision_tree_solver.dart'; +import 'package:ml_dataframe/ml_dataframe.dart'; import 'package:ml_linalg/matrix.dart'; import 'package:ml_linalg/vector.dart'; import 'package:ml_tech/unit_testing/matchers/iterable_2d_almost_equal_to.dart'; import 'package:mockito/mockito.dart'; import 'package:test/test.dart'; -import '../../../test_utils/mocks.dart'; +import '../mocks.dart'; void main() { group('DecisionTreeClassifierImpl', () { @@ -33,10 +34,14 @@ void main() { sample3: DecisionTreeLeafLabel(label3), }); - final classifier = DecisionTreeClassifierImpl(solverMock); - final predictedLabels = classifier.predictClasses(features); + final classifier = DecisionTreeClassifierImpl(solverMock, 'class_name'); + final predictedClasses = classifier.predict( + DataFrame.fromMatrix(features), + ); + + expect(predictedClasses.header, equals(['class_name'])); - expect(predictedLabels, equals([ + expect(predictedClasses.toMatrix(), equals([ [label1], [label2], [label3], @@ -46,10 +51,11 @@ void main() { test('should return an empty matrix if input features matrix is ' 'empty', () { final solverMock = DecisionTreeSolverMock(); - final classifier = DecisionTreeClassifierImpl(solverMock); - final predictedLabels = classifier.predictClasses(Matrix.fromRows([])); + final classifier = DecisionTreeClassifierImpl(solverMock, 'class_name'); + final predictedClasses = classifier.predict(DataFrame([[]])); - expect(predictedLabels, isEmpty); + expect(predictedClasses.header, isEmpty); + expect(predictedClasses.toMatrix(), isEmpty); }); test('should call appropriate method from `solver` when making ' @@ -74,11 +80,14 @@ void main() { sample3: label3, }); - final classifier = DecisionTreeClassifierImpl(solverMock); - final predictedLabels = classifier.predictProbabilities(features); + final classifier = DecisionTreeClassifierImpl(solverMock, 'class_name'); + final predictedLabels = classifier.predictProbabilities( + DataFrame.fromMatrix(features), + ); + expect(predictedLabels.header, equals(['class_name'])); expect( - predictedLabels, + predictedLabels.toMatrix(), iterable2dAlmostEqualTo([ [label1.probability], [label2.probability], diff --git a/test/classifier/non_linear/decision_tree/decision_tree_classifier_test.dart b/test/classifier/decision_tree_classifier_test.dart similarity index 62% rename from test/classifier/non_linear/decision_tree/decision_tree_classifier_test.dart rename to test/classifier/decision_tree_classifier_test.dart index ab2ce1c7..fe4cab35 100644 --- a/test/classifier/non_linear/decision_tree/decision_tree_classifier_test.dart +++ b/test/classifier/decision_tree_classifier_test.dart @@ -1,5 +1,5 @@ -import 'package:ml_algo/src/classifier/non_linear/decision_tree/decision_tree_classifier.dart'; -import 'package:ml_algo/src/classifier/non_linear/decision_tree/decision_tree_classifier_impl.dart'; +import 'package:ml_algo/src/classifier/decision_tree_classifier.dart'; +import 'package:ml_algo/src/classifier/decision_tree_classifier_impl.dart'; import 'package:ml_dataframe/ml_dataframe.dart'; import 'package:ml_linalg/linalg.dart'; import 'package:test/test.dart'; @@ -25,16 +25,20 @@ void main() { ]); group('greedy', () { - final classifier = DecisionTreeClassifier.majority(dataFrame, - targetId: 7, minError: 0.3, minSamplesCount: 1, maxDepth: 3); + final classifier = DecisionTreeClassifier(dataFrame, 'col_8', + minError: 0.3, minSamplesCount: 1, maxDepth: 3); test('should create classifier', () { expect(classifier, isA()); - expect(classifier.classLabels, isNull); }); test('should predict class labels', () { - expect(classifier.predictClasses(featuresForPrediction), + final prediction = classifier.predict( + DataFrame.fromMatrix(featuresForPrediction), + ); + + expect(prediction.header, equals(['col_8'])); + expect(prediction.toMatrix(), equals([ [0], [2], @@ -43,8 +47,13 @@ void main() { ])); }); - test('should predict probability of classes', () { - expect(classifier.predictProbabilities(featuresForPrediction), equals([ + test('should predict probabilities of classes', () { + final probabilities = classifier.predictProbabilities( + DataFrame.fromMatrix(featuresForPrediction), + ); + + expect(probabilities.header, equals(['col_8'])); + expect(probabilities.toMatrix(), equals([ [1], [1], [1], diff --git a/test/classifier/linear/logistic_regressor/logistic_regressor_test.dart b/test/classifier/linear/logistic_regressor/logistic_regressor_test.dart deleted file mode 100644 index f0152d15..00000000 --- a/test/classifier/linear/logistic_regressor/logistic_regressor_test.dart +++ /dev/null @@ -1,93 +0,0 @@ -import 'package:ml_algo/src/classifier/linear/logistic_regressor/gradient_logistic_regressor.dart'; -import 'package:ml_algo/src/solver/linear/gradient/learning_rate_generator/learning_rate_type.dart'; -import 'package:ml_algo/src/solver/linear/initial_weights_generator/initial_weights_type.dart'; -import 'package:ml_linalg/dtype.dart'; -import 'package:ml_linalg/matrix.dart'; -import 'package:mockito/mockito.dart'; -import 'package:test/test.dart'; - -import '../../../test_utils/mocks.dart'; - -void main() { - group('LogisticRegressor', () { - test('should initialize properly', () { - final observations = Matrix.fromList([[1.0]]); - final outcomes = Matrix.fromList([[0]]); - final optimizerMock = OptimizerMock(); - final optimizerFactoryMock = createGradientOptimizerFactoryMock( - observations, outcomes, optimizerMock); - - GradientLogisticRegressor( - observations, - outcomes, - dtype: DType.float32, - learningRateType: LearningRateType.constant, - initialWeightsType: InitialWeightsType.zeroes, - iterationsLimit: 100, - initialLearningRate: 0.01, - minWeightsUpdate: 0.001, - lambda: 0.1, - optimizerFactory: optimizerFactoryMock, - randomSeed: 123, - ); - - verify(optimizerFactoryMock.gradient( - argThat(equals([[1.0]])), - argThat(equals([[0]])), - dtype: DType.float32, - costFunction: anyNamed('costFunction'), - learningRateType: LearningRateType.constant, - initialWeightsType: InitialWeightsType.zeroes, - initialLearningRate: 0.01, - minCoefficientsUpdate: 0.001, - iterationLimit: 100, - lambda: 0.1, - batchSize: 1, - randomSeed: 123, - )).called(1); - }); - - test('should make calls of appropriate method when `fit` is called', () { - final observations = Matrix.fromList([ - [10.1, 10.2, 12.0, 13.4], - [3.1, 5.2, 6.0, 77.4], - ]); - final outcomes = Matrix.fromList([ - [1.0], - [0.0], - ]); - final optimizerMock = OptimizerMock(); - final optimizerFactoryMock = createGradientOptimizerFactoryMock( - observations, outcomes, optimizerMock); - - final initialWeights = Matrix.fromList([ - [10.0], - [20.0], - [30.0], - [40.0], - ]); - - GradientLogisticRegressor( - observations, - outcomes, - dtype: DType.float32, - learningRateType: LearningRateType.constant, - initialWeightsType: InitialWeightsType.zeroes, - iterationsLimit: 100, - initialLearningRate: 0.01, - minWeightsUpdate: 0.001, - lambda: 0.1, - optimizerFactory: optimizerFactoryMock, - initialWeights: initialWeights, - randomSeed: 123, - ); - - verify(optimizerMock.findExtrema( - initialWeights: argThat( - equals(initialWeights), - named: 'initialWeights'), - isMinimizingObjective: false)) - .called(1); - }); - }); -} diff --git a/test/classifier/linear/softmax_regressor/softmax_regressor_test.dart b/test/classifier/linear/softmax_regressor/softmax_regressor_test.dart deleted file mode 100644 index 1aaabf70..00000000 --- a/test/classifier/linear/softmax_regressor/softmax_regressor_test.dart +++ /dev/null @@ -1,144 +0,0 @@ -import 'package:ml_algo/ml_algo.dart'; -import 'package:ml_algo/src/classifier/linear/softmax_regressor/gradient_softmax_regressor.dart'; -import 'package:ml_algo/src/solver/linear/initial_weights_generator/initial_weights_type.dart'; -import 'package:ml_linalg/dtype.dart'; -import 'package:ml_linalg/matrix.dart'; -import 'package:ml_tech/unit_testing/matchers/iterable_2d_almost_equal_to.dart'; -import 'package:mockito/mockito.dart'; -import 'package:test/test.dart'; - -import '../../../test_utils/mocks.dart'; - -void main() { - group('SoftmaxRegressor', () { - final dtype = DType.float32; - - test('should initialize properly', () { - final observations = Matrix.fromList([[1.0]]); - final outcomes = Matrix.fromList([[0]]); - final optimizerMock = OptimizerMock(); - final optimizerFactoryMock = createGradientOptimizerFactoryMock( - observations, outcomes, optimizerMock); - - GradientSoftmaxRegressor( - observations, outcomes, - dtype: dtype, - learningRateType: LearningRateType.constant, - initialWeightsType: InitialWeightsType.zeroes, - iterationsLimit: 100, - initialLearningRate: 0.01, - minWeightsUpdate: 0.001, - lambda: 0.1, - optimizerFactory: optimizerFactoryMock, - randomSeed: 123, - ); - - verify(optimizerFactoryMock.gradient( - observations, - outcomes, - dtype: dtype, - costFunction: anyNamed('costFunction'), - learningRateType: LearningRateType.constant, - initialWeightsType: InitialWeightsType.zeroes, - initialLearningRate: 0.01, - minCoefficientsUpdate: 0.001, - iterationLimit: 100, - lambda: 0.1, - batchSize: 1, - randomSeed: 123, - )).called(1); - }); - - test('should call solver\'s `findExtrema` method with proper ' - 'parameters and consider intercept term', () { - final observations = Matrix.fromList([ - [10.1, 10.2, 12.0, 13.4], - [13.1, 15.2, 61.0, 27.2], - [30.1, 25.2, 62.0, 34.1], - [32.1, 35.2, 36.0, 41.5], - [35.1, 95.2, 56.0, 52.6], - [90.1, 20.2, 10.0, 12.1], - ]); - - final outcomes = Matrix.fromList([ - [1.0, 0.0, 0.0], - [0.0, 1.0, 0.0], - [0.0, 1.0, 0.0], - [1.0, 0.0, 0.0], - [1.0, 0.0, 0.0], - [1.0, 0.0, 0.0], - ]); - - final initialWeights = Matrix.fromList([ - [1.0], - [10.0], - [20.0], - [30.0], - [40.0], - ]); - - final optimizerMock = OptimizerMock(); - final optimizerFactoryMock = createGradientOptimizerFactoryMock( - argThat(iterable2dAlmostEqualTo([ - [2.0, 10.1, 10.2, 12.0, 13.4], - [2.0, 13.1, 15.2, 61.0, 27.2], - [2.0, 30.1, 25.2, 62.0, 34.1], - [2.0, 32.1, 35.2, 36.0, 41.5], - [2.0, 35.1, 95.2, 56.0, 52.6], - [2.0, 90.1, 20.2, 10.0, 12.1], - ], 1e-2)), outcomes, optimizerMock, - ); - - GradientSoftmaxRegressor( - observations, - outcomes, - dtype: dtype, - learningRateType: LearningRateType.constant, - initialWeightsType: InitialWeightsType.zeroes, - iterationsLimit: 100, - initialLearningRate: 0.01, - minWeightsUpdate: 0.001, - lambda: 0.1, - fitIntercept: true, - interceptScale: 2.0, - optimizerFactory: optimizerFactoryMock, - initialWeights: initialWeights, - randomSeed: 123, - ); - - verify(optimizerFactoryMock.gradient( - argThat(iterable2dAlmostEqualTo([ - [2.0, 10.1, 10.2, 12.0, 13.4], - [2.0, 13.1, 15.2, 61.0, 27.2], - [2.0, 30.1, 25.2, 62.0, 34.1], - [2.0, 32.1, 35.2, 36.0, 41.5], - [2.0, 35.1, 95.2, 56.0, 52.6], - [2.0, 90.1, 20.2, 10.0, 12.1], - ], 1e-2)), - argThat(equals([ - [1.0, 0.0, 0.0], - [0.0, 1.0, 0.0], - [0.0, 1.0, 0.0], - [1.0, 0.0, 0.0], - [1.0, 0.0, 0.0], - [1.0, 0.0, 0.0], - ])), - dtype: dtype, - costFunction: anyNamed('costFunction'), - learningRateType: LearningRateType.constant, - initialWeightsType: InitialWeightsType.zeroes, - initialLearningRate: 0.01, - minCoefficientsUpdate: 0.001, - iterationLimit: 100, - lambda: 0.1, - batchSize: 1, - randomSeed: 123, - )).called(1); - - verify(optimizerMock.findExtrema( - initialWeights: initialWeights, - isMinimizingObjective: false, - )).called(1); - }); - }); -} diff --git a/test/classifier/linear/logistic_regressor/logistic_regressor_integration_test.dart b/test/classifier/logistic_regressor_integration_test.dart similarity index 59% rename from test/classifier/linear/logistic_regressor/logistic_regressor_integration_test.dart rename to test/classifier/logistic_regressor_integration_test.dart index e12a4e78..c9498d5f 100644 --- a/test/classifier/linear/logistic_regressor/logistic_regressor_integration_test.dart +++ b/test/classifier/logistic_regressor_integration_test.dart @@ -1,75 +1,33 @@ -import 'package:ml_algo/src/classifier/linear/logistic_regressor/gradient_logistic_regressor.dart'; +import 'package:ml_algo/src/classifier/logistic_regressor.dart'; +import 'package:ml_algo/src/di/injector.dart'; +import 'package:ml_algo/src/linear_optimizer/gradient_optimizer/learning_rate_generator/learning_rate_type.dart'; import 'package:ml_algo/src/metric/metric_type.dart'; -import 'package:ml_algo/src/solver/linear/gradient/learning_rate_generator/learning_rate_type.dart'; +import 'package:ml_dataframe/ml_dataframe.dart'; import 'package:ml_linalg/matrix.dart'; import 'package:ml_tech/unit_testing/matchers/iterable_2d_almost_equal_to.dart'; import 'package:test/test.dart'; void main() { - final firstClass = [1.0]; - final secondClass = [0.0]; - final thirdClass = [0.0]; - group('Logistic regressor', () { - test('should extract class labels from the test_data', () { - final features = Matrix.fromList([ - [5.0, 7.0, 6.0], - [1.0, 2.0, 3.0], - [10.0, 12.0, 31.0], - [9.0, 8.0, 5.0], - [4.0, 0.0, 1.0], - [4.0, 0.0, 1.0], - [4.0, 0.0, 1.0], - ]); - final labels = Matrix.fromList([ - [3.0], - [1.0], - [3.0], - [2.0], - [2.0], - [0.0], - [0.0], - ]); + tearDownAll(() => injector = null); - final classifier = GradientLogisticRegressor( - features, labels, - iterationsLimit: 2, - learningRateType: LearningRateType.constant, - initialLearningRate: 1.0, - fitIntercept: false - ); + test('should fit given data', () { + final samples = DataFrame(>[ + [5.0, 7.0, 6.0, 1.0], + [1.0, 2.0, 3.0, 0.0], + [10.0, 12.0, 31.0, 0.0], + [9.0, 8.0, 5.0, 0.0], + [4.0, 0.0, 1.0, 1.0], + ], headerExists: false); - expect(classifier.classLabels, equals([ - [3.0], - [1.0], - [2.0], - [0.0], - ])); - }); - - test('should properly fit given test_data', () { - final features = Matrix.fromList([ - [5.0, 7.0, 6.0], - [1.0, 2.0, 3.0], - [10.0, 12.0, 31.0], - [9.0, 8.0, 5.0], - [4.0, 0.0, 1.0], - ]); - final labels = Matrix.fromList([ - firstClass, - secondClass, - secondClass, - thirdClass, - firstClass, - ]); - - final classifier = GradientLogisticRegressor( - features, labels, - iterationsLimit: 2, - learningRateType: LearningRateType.constant, - initialLearningRate: 1.0, - batchSize: 5, - fitIntercept: false + final classifier = LogisticRegressor( + samples, + 'col_3', + iterationsLimit: 2, + learningRateType: LearningRateType.constant, + initialLearningRate: 1.0, + batchSize: 5, + fitIntercept: false, ); expect(classifier.coefficientsByClasses, iterable2dAlmostEqualTo([ @@ -80,59 +38,53 @@ void main() { }); test('should make prediction', () { - final features = Matrix.fromList([ - [5.0, 7.0, 6.0], - [1.0, 2.0, 3.0], - [10.0, 12.0, 31.0], - [9.0, 8.0, 5.0], - [4.0, 0.0, 1.0], - ]); - - final labels = Matrix.fromList([ - firstClass, - secondClass, - secondClass, - thirdClass, - firstClass, - ]); + final samples = DataFrame(>[ + [5.0, 7.0, 6.0, 1.0], + [1.0, 2.0, 3.0, 0.0], + [10.0, 12.0, 31.0, 0.0], + [9.0, 8.0, 5.0, 0.0], + [4.0, 0.0, 1.0, 1.0], + ], headerExists: false); - final classifier = GradientLogisticRegressor( - features, labels, - iterationsLimit: 2, - learningRateType: LearningRateType.constant, - initialLearningRate: 1.0, - batchSize: 5, - fitIntercept: false + final classifier = LogisticRegressor( + samples, + 'col_3', + iterationsLimit: 2, + learningRateType: LearningRateType.constant, + initialLearningRate: 1.0, + batchSize: 5, + fitIntercept: false, ); final newFeatures = Matrix.fromList([ [2.0, 4.0, 1.0], ]); - final probabilities = classifier.predictProbabilities(newFeatures); - final classes = classifier.predictClasses(newFeatures); - expect(probabilities, equals([[0.01798621006309986]])); - expect(classes, equals([thirdClass])); + final probabilities = classifier.predictProbabilities( + DataFrame.fromMatrix(newFeatures), + ); + + final classes = classifier.predict( + DataFrame.fromMatrix(newFeatures), + ); + + expect(probabilities.header, equals(['col_3'])); + expect(probabilities.toMatrix(), equals([[0.01798621006309986]])); + expect(classes.header, equals(['col_3'])); + expect(classes.toMatrix(), equals([[0.0]])); }); test('should evaluate prediction quality, accuracy = 0', () { - final features = Matrix.fromList([ - [5.0, 7.0, 6.0], - [1.0, 2.0, 3.0], - [10.0, 12.0, 31.0], - [9.0, 8.0, 5.0], - [4.0, 0.0, 1.0], - ]); - final labels = Matrix.fromList([ - firstClass, - secondClass, - secondClass, - thirdClass, - firstClass, - ]); + final samples = DataFrame(>[ + [5.0, 7.0, 6.0, 1.0], + [1.0, 2.0, 3.0, 0.0], + [10.0, 12.0, 31.0, 0.0], + [9.0, 8.0, 5.0, 0.0], + [4.0, 0.0, 1.0, 1.0], + ], headerExists: false); - final classifier = GradientLogisticRegressor( - features, labels, + final classifier = LogisticRegressor( + samples, 'col_3', iterationsLimit: 2, learningRateType: LearningRateType.constant, initialLearningRate: 1.0, @@ -140,35 +92,27 @@ void main() { fitIntercept: false ); - final newFeatures = Matrix.fromList([ - [2.0, 4.0, 1.0], - ]); - final origLabels = Matrix.fromList([ - [1.0] - ]); - final score = - classifier.assess(newFeatures, origLabels, MetricType.accuracy); + final newSamples = DataFrame([ + [2.0, 4.0, 1.0, 1.0], + ], header: ['first', 'second', 'third', 'target'], headerExists: false); + + final score = classifier.assess(newSamples, ['target'], + MetricType.accuracy); + expect(score, equals(0.0)); }); test('should evaluate prediction quality, accuracy = 1', () { - final features = Matrix.fromList([ - [5.0, 7.0, 6.0], - [1.0, 2.0, 3.0], - [10.0, 12.0, 31.0], - [9.0, 8.0, 5.0], - [4.0, 0.0, 1.0], - ]); - final labels = Matrix.fromList([ - firstClass, - secondClass, - secondClass, - thirdClass, - firstClass, - ]); + final samples = DataFrame(>[ + [5.0, 7.0, 6.0, 1.0], + [1.0, 2.0, 3.0, 0.0], + [10.0, 12.0, 31.0, 0.0], + [9.0, 8.0, 5.0, 0.0], + [4.0, 0.0, 1.0, 1.0], + ], headerExists: false); - final classifier = GradientLogisticRegressor( - features, labels, + final classifier = LogisticRegressor( + samples, 'col_3', iterationsLimit: 2, learningRateType: LearningRateType.constant, initialLearningRate: 1.0, @@ -176,33 +120,29 @@ void main() { fitIntercept: false ); - final newFeatures = Matrix.fromList([ - [2.0, 4.0, 1.0], - ]); - final newLabels = Matrix.fromList([ - thirdClass, - ]); - final score = - classifier.assess(newFeatures, newLabels, MetricType.accuracy); + final newFeatures = DataFrame([ + [2, 4, 1, 0], + ], header: ['first', 'second', 'third', 'target'], headerExists: false); + + final score = classifier.assess(newFeatures, ['target'], + MetricType.accuracy); + expect(score, equals(1.0)); }); test('should consider intercept term', () { - final features = Matrix.fromList([ - [5.0, 7.0, 6.0], - [1.0, 2.0, 3.0], - ]); - final labels = Matrix.fromList([ - [1.0], - [0.0], - ]); - final classifier = GradientLogisticRegressor( - features, labels, - iterationsLimit: 1, - learningRateType: LearningRateType.constant, - initialLearningRate: 1.0, - batchSize: 2, - fitIntercept: true + final features = DataFrame(>[ + [5.0, 7.0, 6.0, 1.0], + [1.0, 2.0, 3.0, 0.0], + ], headerExists: false); + + final classifier = LogisticRegressor( + features, 'col_3', + iterationsLimit: 1, + learningRateType: LearningRateType.constant, + initialLearningRate: 1.0, + batchSize: 2, + fitIntercept: true, ); // as the intercept is required to be fitted, our test_data should look as follows: // @@ -267,25 +207,21 @@ void main() { test('should consider intercept scale if intercept term is going to be ' 'fitted', () { - final features = Matrix.fromList([ - [5.0, 7.0, 6.0], - [1.0, 2.0, 3.0], - [3.0, 4.0, 5.0], - ]); - final labels = Matrix.fromList([ - [1.0], - [0.0], - [1.0], - ]); + final samples = DataFrame(>[ + [5.0, 7.0, 6.0, 1.0], + [1.0, 2.0, 3.0, 0.0], + [3.0, 4.0, 5.0, 1.0], + ], headerExists: false); - final classifier = GradientLogisticRegressor( - features, labels, - iterationsLimit: 1, - learningRateType: LearningRateType.constant, - initialLearningRate: 1.0, - batchSize: 3, - fitIntercept: true, - interceptScale: 2.0 + final classifier = LogisticRegressor( + samples, + 'col_3', + iterationsLimit: 1, + learningRateType: LearningRateType.constant, + initialLearningRate: 1.0, + batchSize: 3, + fitIntercept: true, + interceptScale: 2.0, ); // as the intercept is required to be fitted, our test_data should look as follows: diff --git a/test/classifier/logistic_regressor_test.dart b/test/classifier/logistic_regressor_test.dart new file mode 100644 index 00000000..947cc6b0 --- /dev/null +++ b/test/classifier/logistic_regressor_test.dart @@ -0,0 +1,281 @@ +import 'package:injector/injector.dart'; +import 'package:ml_algo/src/classifier/logistic_regressor.dart'; +import 'package:ml_algo/src/cost_function/cost_function.dart'; +import 'package:ml_algo/src/cost_function/cost_function_factory.dart'; +import 'package:ml_algo/src/cost_function/cost_function_type.dart'; +import 'package:ml_algo/src/di/injector.dart'; +import 'package:ml_algo/src/linear_optimizer/gradient_optimizer/learning_rate_generator/learning_rate_type.dart'; +import 'package:ml_algo/src/linear_optimizer/initial_coefficients_generator/initial_coefficients_type.dart'; +import 'package:ml_algo/src/linear_optimizer/linear_optimizer.dart'; +import 'package:ml_algo/src/linear_optimizer/linear_optimizer_factory.dart'; +import 'package:ml_algo/src/linear_optimizer/linear_optimizer_type.dart'; +import 'package:ml_algo/src/linear_optimizer/regularization_type.dart'; +import 'package:ml_algo/src/link_function/link_function.dart'; +import 'package:ml_algo/src/link_function/link_function_factory.dart'; +import 'package:ml_algo/src/link_function/link_function_type.dart'; +import 'package:ml_dataframe/ml_dataframe.dart'; +import 'package:ml_linalg/dtype.dart'; +import 'package:ml_linalg/matrix.dart'; +import 'package:ml_linalg/vector.dart'; +import 'package:ml_tech/unit_testing/matchers/iterable_2d_almost_equal_to.dart'; +import 'package:mockito/mockito.dart'; +import 'package:test/test.dart'; + +import '../mocks.dart'; + +void main() { + group('LogisticRegressor', () { + final observations = DataFrame([ + [10.1, 10.2, 12.0, 13.4, 1], + [ 3.1, 5.2, 6.0, 77.4, 0], + ], headerExists: false); + + final initialCoefficients = Vector.fromList([ + 10, + 20, + 30, + 40, + 50, + ]); + + final learnedCoefficients = Matrix.fromList([ + [100], + [200], + [300], + [400], + [500], + ]); + + final negativeLabel = 100; + final positiveLabel = 200; + + LinkFunction linkFunctionMock; + LinkFunctionFactory linkFunctionFactoryMock; + + CostFunction costFunctionMock; + CostFunctionFactory costFunctionFactoryMock; + + LinearOptimizer optimizerMock; + LinearOptimizerFactory optimizerFactoryMock; + + setUp(() { + linkFunctionMock = LinkFunctionMock(); + linkFunctionFactoryMock = createLinkFunctionFactoryMock(linkFunctionMock); + + costFunctionMock = CostFunctionMock(); + costFunctionFactoryMock = createCostFunctionFactoryMock(costFunctionMock); + + optimizerMock = LinearOptimizerMock(); + optimizerFactoryMock = createLinearOptimizerFactoryMock(optimizerMock); + + injector = Injector() + ..registerSingleton( + (_) => linkFunctionFactoryMock) + ..registerDependency( + (_) => costFunctionFactoryMock) + ..registerSingleton( + (_) => optimizerFactoryMock); + + when(optimizerMock.findExtrema( + initialCoefficients: anyNamed('initialCoefficients'), + isMinimizingObjective: anyNamed('isMinimizingObjective'), + )).thenReturn(learnedCoefficients); + }); + + tearDownAll(() => injector = null); + + test('should throw an exception if a target column does not exist', () { + final targetColumnName = 'col_10'; + + final actual = () => LogisticRegressor( + observations, + targetColumnName, + ); + + expect(actual, throwsException); + }); + + test('should call link function factory twice in order to create inverse ' + 'logit link function', () { + LogisticRegressor( + observations, + 'col_4', + ); + + verify(linkFunctionFactoryMock.createByType( + LinkFunctionType.inverseLogit, + dtype: DType.float32, + )).called(2); + }); + + test('should call cost function factory in order to create ' + 'loglikelihood cost function', () { + LogisticRegressor( + observations, + 'col_4', + ); + + verify(costFunctionFactoryMock.createByType( + CostFunctionType.logLikelihood, + linkFunction: linkFunctionMock, + )).called(1); + }); + + test('should call linear optimizer factory and consider intercept term ' + 'while calling the factory', () { + LogisticRegressor( + observations, + 'col_4', + learningRateType: LearningRateType.decreasingAdaptive, + initialCoefficientsType: InitialCoefficientsType.zeroes, + iterationsLimit: 1000, + initialLearningRate: 0.01, + minCoefficientsUpdate: 0.001, + lambda: 0.1, + regularizationType: RegularizationType.L2, + initialCoefficients: initialCoefficients, + randomSeed: 123, + fitIntercept: true, + interceptScale: 2.0, + isFittingDataNormalized: true, + positiveLabel: positiveLabel, + negativeLabel: negativeLabel, + dtype: DType.float32, + ); + + verify(optimizerFactoryMock.createByType( + LinearOptimizerType.gradient, + argThat(iterable2dAlmostEqualTo([ + [2.0, 10.1, 10.2, 12.0, 13.4], + [2.0, 3.1, 5.2, 6.0, 77.4], + ])), + argThat(equals([ + [1.0], + [0.0], + ])), + dtype: DType.float32, + costFunction: costFunctionMock, + learningRateType: LearningRateType.decreasingAdaptive, + initialCoefficientsType: InitialCoefficientsType.zeroes, + initialLearningRate: 0.01, + minCoefficientsUpdate: 0.001, + iterationLimit: 1000, + lambda: 0.1, + regularizationType: RegularizationType.L2, + batchSize: 1, + randomSeed: 123, + isFittingDataNormalized: true, + )).called(1); + }); + + test('should find the extrema for fitting observations while ' + 'instantiating', () { + LogisticRegressor( + observations, + 'col_4', + initialCoefficients: initialCoefficients, + ); + + verify(optimizerMock.findExtrema( + initialCoefficients: argThat( + equals(Matrix.fromColumns([initialCoefficients])), + named: 'initialCoefficients', + ), + isMinimizingObjective: false, + )).called(1); + }); + + test('should predict classes basing on learned coefficients', () { + final classifier = LogisticRegressor( + observations, + 'col_4', + initialCoefficients: initialCoefficients, + fitIntercept: true, + interceptScale: 2.0, + positiveLabel: positiveLabel, + negativeLabel: negativeLabel, + ); + + final probabilities = Matrix.fromList([ + [0.2], + [0.3], + [0.6], + ]); + + when(linkFunctionMock.link(any)).thenReturn(probabilities); + + final features = Matrix.fromList([ + [55, 44, 33, 22], + [10, 88, 77, 11], + [12, 22, 39, 13], + ]); + + final featuresWithIntercept = Matrix.fromColumns([ + Vector.filled(3, 2), + ...features.columns, + ]); + + final classes = classifier.predict( + DataFrame.fromMatrix(features), + ); + + expect(classes.header, equals(['col_4'])); + + expect(classes.toMatrix(), equals([ + [negativeLabel], + [negativeLabel], + [positiveLabel], + ])); + + verify(linkFunctionMock.link(argThat(iterable2dAlmostEqualTo( + featuresWithIntercept * learnedCoefficients + )))).called(1); + }); + + test('should predict probabilities of classes basing on learned ' + 'coefficients', () { + final classifier = LogisticRegressor( + observations, + 'col_4', + initialCoefficients: initialCoefficients, + fitIntercept: true, + interceptScale: 2.0, + ); + + final probabilities = Matrix.fromList([ + [0.2], + [0.3], + [0.6], + ]); + + when(linkFunctionMock.link(any)).thenReturn(probabilities); + + final features = Matrix.fromList([ + [55, 44, 33, 22], + [10, 88, 77, 11], + [12, 22, 39, 13], + ]); + + final featuresWithIntercept = Matrix.fromColumns([ + Vector.filled(3, 2), + ...features.columns, + ]); + + final prediction = classifier.predictProbabilities( + DataFrame.fromMatrix(features), + ); + + expect(prediction.header, equals(['col_4'])); + + expect(prediction.toMatrix(), iterable2dAlmostEqualTo([ + [0.2], + [0.3], + [0.6], + ])); + + verify(linkFunctionMock.link(argThat(iterable2dAlmostEqualTo( + featuresWithIntercept * learnedCoefficients + )))).called(1); + }); + }); +} diff --git a/test/classifier/softmax_regressor_test.dart b/test/classifier/softmax_regressor_test.dart new file mode 100644 index 00000000..80ea0e69 --- /dev/null +++ b/test/classifier/softmax_regressor_test.dart @@ -0,0 +1,323 @@ +import 'package:injector/injector.dart'; +import 'package:ml_algo/ml_algo.dart'; +import 'package:ml_algo/src/cost_function/cost_function.dart'; +import 'package:ml_algo/src/cost_function/cost_function_factory.dart'; +import 'package:ml_algo/src/cost_function/cost_function_type.dart'; +import 'package:ml_algo/src/di/injector.dart'; +import 'package:ml_algo/src/linear_optimizer/initial_coefficients_generator/initial_coefficients_type.dart'; +import 'package:ml_algo/src/linear_optimizer/linear_optimizer.dart'; +import 'package:ml_algo/src/linear_optimizer/linear_optimizer_factory.dart'; +import 'package:ml_algo/src/linear_optimizer/linear_optimizer_type.dart'; +import 'package:ml_algo/src/linear_optimizer/regularization_type.dart'; +import 'package:ml_algo/src/link_function/link_function.dart'; +import 'package:ml_algo/src/link_function/link_function_factory.dart'; +import 'package:ml_algo/src/link_function/link_function_type.dart'; +import 'package:ml_dataframe/ml_dataframe.dart'; +import 'package:ml_linalg/dtype.dart'; +import 'package:ml_linalg/linalg.dart'; +import 'package:ml_linalg/matrix.dart'; +import 'package:ml_tech/unit_testing/matchers/iterable_2d_almost_equal_to.dart'; +import 'package:mockito/mockito.dart'; +import 'package:test/test.dart'; + +import '../mocks.dart'; + +void main() { + group('SoftmaxRegressor', () { + final features = Matrix.fromList([ + [10.1, 10.2, 12.0, 13.4], + [13.1, 15.2, 61.0, 27.2], + [30.1, 25.2, 62.0, 34.1], + [32.1, 35.2, 36.0, 41.5], + [35.1, 95.2, 56.0, 52.6], + [90.1, 20.2, 10.0, 12.1], + ]); + + final outcomes = Matrix.fromList([ + [1.0, 0.0, 0.0], + [0.0, 0.0, 1.0], + [0.0, 1.0, 0.0], + [1.0, 0.0, 0.0], + [0.0, 0.0, 1.0], + [1.0, 0.0, 0.0], + ]); + + final observations = DataFrame.fromMatrix( + Matrix.fromColumns([ + ...features.columns, + ...outcomes.columns, + ], dtype: DType.float32), + header: ['a', 'b', 'c', 'd', 'target_1', 'target_2', 'target_3'], + ); + + final initialCoefficients = Matrix.fromList([ + [1.0], + [10.0], + [20.0], + [30.0], + [40.0], + ]); + + final learnedCoefficients = Matrix.fromList([ + [1, 2, 3], + [1, 2, 3], + [1, 2, 3], + [1, 2, 3], + [1, 2, 3], + ]); + + final negativeLabel = 10; + final positiveLabel = 20; + + LinkFunction linkFunctionMock; + LinkFunctionFactory linkFunctionFactoryMock; + + CostFunction costFunctionMock; + CostFunctionFactory costFunctionFactoryMock; + + LinearOptimizer optimizerMock; + LinearOptimizerFactory optimizerFactoryMock; + + SoftmaxRegressor classifier; + + setUp(() { + linkFunctionMock = LinkFunctionMock(); + linkFunctionFactoryMock = createLinkFunctionFactoryMock(linkFunctionMock); + + costFunctionMock = CostFunctionMock(); + costFunctionFactoryMock = createCostFunctionFactoryMock(costFunctionMock); + + optimizerMock = LinearOptimizerMock(); + optimizerFactoryMock = createLinearOptimizerFactoryMock(optimizerMock); + + injector = Injector() + ..registerSingleton((_) => linkFunctionFactoryMock) + ..registerDependency((_) => costFunctionFactoryMock) + ..registerSingleton((_) => optimizerFactoryMock); + + when(optimizerMock.findExtrema( + initialCoefficients: anyNamed('initialCoefficients'), + isMinimizingObjective: anyNamed('isMinimizingObjective'), + )).thenReturn(learnedCoefficients); + }); + + tearDownAll(() => injector = null); + + test('should throw an exception if some of target columns does not ' + 'exist', () { + final targetColumnNames = ['target_1', 'some', 'unknown', 'columns']; + + final actual = () => SoftmaxRegressor( + observations, + targetColumnNames, + ); + + expect(actual, throwsException); + }); + + test('should throw an exception if target columns number is less than ' + 'two, since the SoftmaxRegressor supports only multiclass ' + 'classification with one-hot (or other similar method) encoded ' + 'features', () { + + final targetColumnNames = + ['target_1']; + + final actual = () => SoftmaxRegressor( + observations, + targetColumnNames, + ); + + expect(actual, throwsException); + }); + + test('should call link function factory twice in order to create softmax ' + 'link function', () { + SoftmaxRegressor( + observations, + ['target_1', 'target_2', 'target_3'], + ); + + verify(linkFunctionFactoryMock.createByType( + LinkFunctionType.softmax, + dtype: DType.float32, + )).called(2); + }); + + test('should call cost function factory in order to create ' + 'loglikelihood cost function', () { + SoftmaxRegressor( + observations, + ['target_1', 'target_2', 'target_3'], + ); + + verify(costFunctionFactoryMock.createByType( + CostFunctionType.logLikelihood, + linkFunction: linkFunctionMock, + )).called(1); + }); + + test('should call linear optimizer factory and consider intercept term ' + 'while calling the factory', () { + SoftmaxRegressor( + observations, + ['target_1', 'target_2', 'target_3'], + optimizerType: LinearOptimizerType.gradient, + learningRateType: LearningRateType.constant, + initialCoefficientsType: InitialCoefficientsType.zeroes, + iterationsLimit: 100, + initialLearningRate: 0.01, + minCoefficientsUpdate: 0.001, + lambda: 0.1, + regularizationType: RegularizationType.L2, + fitIntercept: true, + interceptScale: 2.0, + initialCoefficients: initialCoefficients, + randomSeed: 123, + negativeLabel: negativeLabel, + positiveLabel: positiveLabel, + dtype: DType.float32, + ); + + verify(optimizerFactoryMock.createByType( + LinearOptimizerType.gradient, + argThat(iterable2dAlmostEqualTo([ + [2.0, 10.1, 10.2, 12.0, 13.4], + [2.0, 13.1, 15.2, 61.0, 27.2], + [2.0, 30.1, 25.2, 62.0, 34.1], + [2.0, 32.1, 35.2, 36.0, 41.5], + [2.0, 35.1, 95.2, 56.0, 52.6], + [2.0, 90.1, 20.2, 10.0, 12.1], + ], 1e-2)), + argThat(equals([ + [1.0, 0.0, 0.0], + [0.0, 0.0, 1.0], + [0.0, 1.0, 0.0], + [1.0, 0.0, 0.0], + [0.0, 0.0, 1.0], + [1.0, 0.0, 0.0], + ])), + dtype: DType.float32, + costFunction: costFunctionMock, + learningRateType: LearningRateType.constant, + initialCoefficientsType: InitialCoefficientsType.zeroes, + initialLearningRate: 0.01, + minCoefficientsUpdate: 0.001, + iterationLimit: 100, + lambda: 0.1, + regularizationType: RegularizationType.L2, + batchSize: 1, + randomSeed: 123, + )).called(1); + }); + + test('should find the extrema for fitting observations while ' + 'instantiating', () { + SoftmaxRegressor( + observations, + ['target_1', 'target_2', 'target_3'], + initialCoefficients: initialCoefficients, + ); + + verify(optimizerMock.findExtrema( + initialCoefficients: initialCoefficients, + isMinimizingObjective: false, + )).called(1); + }); + + test('should predict classes basing on learned coefficients', () { + final classifier = SoftmaxRegressor( + observations, + ['target_1', 'target_2', 'target_3'], + fitIntercept: true, + interceptScale: 2.0, + initialCoefficients: initialCoefficients, + negativeLabel: negativeLabel, + positiveLabel: positiveLabel, + ); + + final probabilities = Matrix.fromList([ + [0.2, 0.7, 0.1], + [0.3, 0.2, 0.5], + [0.6, 0.2, 0.2], + ]); + + when(linkFunctionMock.link(any)).thenReturn(probabilities); + + final features = Matrix.fromList([ + [55, 44, 33, 22], + [10, 88, 77, 11], + [12, 22, 39, 13], + ]); + + final featuresWithIntercept = Matrix.fromColumns([ + Vector.filled(3, 2), + ...features.columns, + ]); + + final classes = classifier.predict( + DataFrame.fromMatrix(features), + ); + + expect(classes.header, equals(['target_1', 'target_2', 'target_3'])); + + expect(classes.toMatrix(), equals([ + [negativeLabel, positiveLabel, negativeLabel], + [negativeLabel, negativeLabel, positiveLabel], + [positiveLabel, negativeLabel, negativeLabel], + ])); + + verify(linkFunctionMock.link(argThat(iterable2dAlmostEqualTo( + featuresWithIntercept * learnedCoefficients + )))).called(1); + }); + + test('should predict probabilities of classes basing on learned ' + 'coefficients', () { + final classifier = SoftmaxRegressor( + observations, + ['target_1', 'target_2', 'target_3'], + fitIntercept: true, + interceptScale: 2.0, + initialCoefficients: initialCoefficients, + negativeLabel: negativeLabel, + positiveLabel: positiveLabel, + ); + + final probabilities = Matrix.fromList([ + [0.2, 0.7, 0.1], + [0.3, 0.2, 0.5], + [0.6, 0.2, 0.2], + ]); + + when(linkFunctionMock.link(any)).thenReturn(probabilities); + + final features = Matrix.fromList([ + [55, 44, 33, 22], + [10, 88, 77, 11], + [12, 22, 39, 13], + ]); + + final featuresWithIntercept = Matrix.fromColumns([ + Vector.filled(3, 2), + ...features.columns, + ]); + + final prediction = classifier.predictProbabilities( + DataFrame.fromMatrix(features), + ); + + expect(prediction.header, equals(['target_1', 'target_2', 'target_3'])); + + expect(prediction.toMatrix(), iterable2dAlmostEqualTo([ + [0.2, 0.7, 0.1], + [0.3, 0.2, 0.5], + [0.6, 0.2, 0.2], + ])); + + verify(linkFunctionMock.link(argThat(iterable2dAlmostEqualTo( + featuresWithIntercept * learnedCoefficients + )))).called(1); + }); + }); +} diff --git a/test/classifier/test_all.dart b/test/classifier/test_all.dart deleted file mode 100644 index 2351f75c..00000000 --- a/test/classifier/test_all.dart +++ /dev/null @@ -1,13 +0,0 @@ -import 'linear/logistic_regressor/logistic_regressor_integration_test.dart' as logistic_regressor_integration_test; -import 'linear/logistic_regressor/logistic_regressor_test.dart' as logistic_regressor_test; -import 'linear/softmax_regressor/softmax_regressor_test.dart' as softmax_regressor_test; -import 'non_linear/decision_tree/decision_tree_classifier_impl_test.dart' as decision_tree_classifier_impl_test; -import 'non_linear/decision_tree/decision_tree_classifier_test.dart' as decision_tree_classifier_test; - -void main() { - decision_tree_classifier_impl_test.main(); - decision_tree_classifier_test.main(); - logistic_regressor_integration_test.main(); - logistic_regressor_test.main(); - softmax_regressor_test.main(); -} diff --git a/test/cost_function/cost_function_factory_impl_test.dart b/test/cost_function/cost_function_factory_impl_test.dart new file mode 100644 index 00000000..17abfc85 --- /dev/null +++ b/test/cost_function/cost_function_factory_impl_test.dart @@ -0,0 +1,39 @@ +import 'package:ml_algo/src/cost_function/cost_function_factory_impl.dart'; +import 'package:ml_algo/src/cost_function/cost_function_type.dart'; +import 'package:ml_algo/src/cost_function/log_likelihood.dart'; +import 'package:ml_algo/src/cost_function/squared.dart'; +import 'package:ml_algo/src/link_function/link_function_factory_impl.dart'; +import 'package:ml_algo/src/link_function/link_function_type.dart'; +import 'package:test/test.dart'; + +void main() { + group('CostFunctionFactoryImpl', () { + final factory = const CostFunctionFactoryImpl(); + + test('should create a squared cost function', () { + final costFn = factory.createByType(CostFunctionType.squared); + expect(costFn, isA()); + }); + + test('should create a loglikelihood cost function considering passed ' + 'link function', () { + final linkFn = const LinkFunctionFactoryImpl() + .createByType(LinkFunctionType.softmax); + + final costFn = factory.createByType( + CostFunctionType.logLikelihood, + linkFunction: linkFn, + ); + + expect(costFn, isA()); + }); + + test('should throw an exception if no link function provided for ' + 'loglikelihood function', () { + expect( + () => factory.createByType(CostFunctionType.logLikelihood), + throwsException, + ); + }); + }); +} diff --git a/test/cost_function/cost_function_test.dart b/test/cost_function/cost_function_test.dart index 7fcafa29..ac6f33f5 100644 --- a/test/cost_function/cost_function_test.dart +++ b/test/cost_function/cost_function_test.dart @@ -4,13 +4,13 @@ import 'package:ml_linalg/linalg.dart'; import 'package:mockito/mockito.dart'; import 'package:test/test.dart'; -import '../test_utils/mocks.dart'; +import '../mocks.dart'; void main() { group('SquaredCost', () { final squaredCost = const SquaredCost(); - test('should return a proper gradient vector', () { + test('should return a gradient vector of quadratic error function', () { // The formula in matrix notation: // -2 * X^t * (y - X*w) // where X^t - transposed X matrix @@ -52,7 +52,8 @@ void main() { [1.0], ])); - test('should return a proper gradient vector', () { + test('should return a gradient vector of log likelihood error ' + 'function', () { // The formula in matrix notation: // X^t * (indicator(Y, 1) - P(y=+1|X,W)) // where X^t - transposed X matrix @@ -87,5 +88,19 @@ void main() { expect(actual, equals(expected)); }); + + test('should return error cost value', () { + expect( + () => logLikelihoodCost.getCost(null, null), + throwsUnimplementedError, + ); + }); + + test('should return a subgradient vector', () { + expect( + () => logLikelihoodCost.getSubGradient(null, null, null, null), + throwsUnimplementedError, + ); + }); }); } diff --git a/test/cross_validator/cross_validator_impl_test.dart b/test/cross_validator/cross_validator_impl_test.dart deleted file mode 100644 index dd4df0c6..00000000 --- a/test/cross_validator/cross_validator_impl_test.dart +++ /dev/null @@ -1,81 +0,0 @@ -import 'dart:typed_data'; - -import 'package:ml_algo/src/metric/metric_type.dart'; -import 'package:ml_algo/src/model_selection/cross_validator/cross_validator_impl.dart'; -import 'package:ml_algo/src/model_selection/data_splitter/splitter.dart'; -import 'package:ml_linalg/dtype.dart'; -import 'package:ml_linalg/matrix.dart'; -import 'package:mockito/mockito.dart'; -import 'package:test/test.dart'; - -import '../test_utils/mocks.dart'; - -Splitter createSplitter(Iterable> indices) { - final splitter = SplitterMock(); - when(splitter.split(any)).thenReturn(indices); - return splitter; -} - -void main() { - group('CrossValidatorImpl', () { - test('should perform validation of a model on given test indices of' - 'observations', () { - final allObservations = Matrix.fromList([ - [330, 930, 130], - [630, 830, 230], - [730, 730, 330], - [830, 630, 430], - [930, 530, 530], - [130, 430, 630], - [230, 330, 730], - [430, 230, 830], - [530, 130, 930], - ]); - final allOutcomes = Matrix.fromList([ - [100],[200],[300],[400],[500],[600],[700],[800],[900], - ]); - final metric = MetricType.mape; - final splitter = createSplitter([[0,2,4],[6, 8]]); - final predictor = PredictorMock(); - final validator = CrossValidatorImpl(DType.float32, splitter); - - var score = 20.0; - when(predictor.assess(any, any, any)) - .thenAnswer((Invocation inv) => score = score + 10); - - final actual = validator.evaluate((observations, outcomes) => predictor, - allObservations, allOutcomes, metric); - - expect(actual, 35); - - verify(predictor.assess(argThat(equals([ - [330, 930, 130], - [730, 730, 330], - [930, 530, 530], - ])), argThat(equals([[100], [300], [500]])), metric)).called(1); - - verify(predictor.assess(argThat(equals([ - [230, 330, 730], - [530, 130, 930], - ])), argThat(equals([[700], [900]])), metric)).called(1); - }); - - test('should throw an exception if observations number and outcomes number ' - 'mismatch', () { - final allObservations = Matrix.fromList([ - [330, 930, 130], - [630, 830, 230], - ]); - final allOutcomes = Matrix.fromList([ - [100], - ]); - final metric = MetricType.mape; - final splitter = SplitterMock(); - final predictor = PredictorMock(); - final validator = CrossValidatorImpl(DType.float32, splitter); - - expect(() => validator.evaluate((observations, outcomes) => predictor, - allObservations, allOutcomes, metric), throwsException); - }); - }); -} diff --git a/test/solver/non_linear/decision_tree/assessor/majority_split_assesor_test.dart b/test/decision_tree_solver/assessor/majority_split_assesor_test.dart similarity index 97% rename from test/solver/non_linear/decision_tree/assessor/majority_split_assesor_test.dart rename to test/decision_tree_solver/assessor/majority_split_assesor_test.dart index b72495fb..11ddae04 100644 --- a/test/solver/non_linear/decision_tree/assessor/majority_split_assesor_test.dart +++ b/test/decision_tree_solver/assessor/majority_split_assesor_test.dart @@ -1,4 +1,4 @@ -import 'package:ml_algo/src/solver/non_linear/decision_tree/split_assessor/majority_split_assessor.dart'; +import 'package:ml_algo/src/decision_tree_solver/split_assessor/majority_split_assessor.dart'; import 'package:ml_linalg/matrix.dart'; import 'package:test/test.dart'; diff --git a/test/solver/non_linear/decision_tree/decision_tree_node_test.dart b/test/decision_tree_solver/decision_tree_node_test.dart similarity index 92% rename from test/solver/non_linear/decision_tree/decision_tree_node_test.dart rename to test/decision_tree_solver/decision_tree_node_test.dart index e0892d16..0ac89d87 100644 --- a/test/solver/non_linear/decision_tree/decision_tree_node_test.dart +++ b/test/decision_tree_solver/decision_tree_node_test.dart @@ -1,4 +1,4 @@ -import 'package:ml_algo/src/solver/non_linear/decision_tree/decision_tree_node.dart'; +import 'package:ml_algo/src/decision_tree_solver/decision_tree_node.dart'; import 'package:ml_linalg/vector.dart'; import 'package:ml_tech/unit_testing/readers/json.dart'; import 'package:test/test.dart'; @@ -60,7 +60,7 @@ void main() { child13, ], null); - final snapshotFileName = 'test/solver/non_linear/decision_tree/' + final snapshotFileName = 'test/decision_tree_solver/' 'decision_tree_node_test.json'; final actual = root.serialize(); final expected = await readJSON(snapshotFileName); diff --git a/test/solver/non_linear/decision_tree/decision_tree_node_test.json b/test/decision_tree_solver/decision_tree_node_test.json similarity index 100% rename from test/solver/non_linear/decision_tree/decision_tree_node_test.json rename to test/decision_tree_solver/decision_tree_node_test.json diff --git a/test/solver/non_linear/decision_tree/decision_tree_solver_integration_test.dart b/test/decision_tree_solver/decision_tree_solver_integration_test.dart similarity index 80% rename from test/solver/non_linear/decision_tree/decision_tree_solver_integration_test.dart rename to test/decision_tree_solver/decision_tree_solver_integration_test.dart index 95180af4..31ff925d 100644 --- a/test/solver/non_linear/decision_tree/decision_tree_solver_integration_test.dart +++ b/test/decision_tree_solver/decision_tree_solver_integration_test.dart @@ -1,6 +1,4 @@ -import 'dart:math'; - -import 'package:ml_algo/src/solver/non_linear/decision_tree/solver_factory/greedy_solver.dart'; +import 'package:ml_algo/src/decision_tree_solver/solver_factory/greedy_solver.dart'; import 'package:ml_dataframe/ml_dataframe.dart'; import 'package:ml_tech/unit_testing/readers/json.dart'; import 'package:test/test.dart'; @@ -20,9 +18,9 @@ void main() { ]); test('should build a decision tree structure', () async { - final snapshotFileName = 'test/solver/non_linear/decision_tree/' + final snapshotFileName = 'test/decision_tree_solver/' 'decision_tree_solver_integration_test.json'; - final solver = createGreedySolver(dataFrame, 7, null, 0.3, 1, 3); + final solver = createGreedySolver(dataFrame, 'col_8', 0.3, 1, 3); final actual = solver.root.serialize(); final expected = await readJSON(snapshotFileName); diff --git a/test/solver/non_linear/decision_tree/decision_tree_solver_integration_test.json b/test/decision_tree_solver/decision_tree_solver_integration_test.json similarity index 100% rename from test/solver/non_linear/decision_tree/decision_tree_solver_integration_test.json rename to test/decision_tree_solver/decision_tree_solver_integration_test.json diff --git a/test/solver/non_linear/decision_tree/leaf_detector/leaf_detector_impl_test.dart b/test/decision_tree_solver/leaf_detector/leaf_detector_impl_test.dart similarity index 97% rename from test/solver/non_linear/decision_tree/leaf_detector/leaf_detector_impl_test.dart rename to test/decision_tree_solver/leaf_detector/leaf_detector_impl_test.dart index bf7603af..06508530 100644 --- a/test/solver/non_linear/decision_tree/leaf_detector/leaf_detector_impl_test.dart +++ b/test/decision_tree_solver/leaf_detector/leaf_detector_impl_test.dart @@ -1,10 +1,10 @@ -import 'package:ml_algo/src/solver/non_linear/decision_tree/leaf_detector/leaf_detector_impl.dart'; +import 'package:ml_algo/src/decision_tree_solver/leaf_detector/leaf_detector_impl.dart'; import 'package:ml_linalg/matrix.dart'; import 'package:mockito/mockito.dart'; import 'package:quiver/iterables.dart'; import 'package:test/test.dart'; -import '../../../../test_utils/mocks.dart'; +import '../../mocks.dart'; void main() { group('LeafDetectorImpl', () { diff --git a/test/solver/non_linear/decision_tree/leaf_label_factory/majority_leaf_label_factory_test.dart b/test/decision_tree_solver/leaf_label_factory/majority_leaf_label_factory_test.dart similarity index 94% rename from test/solver/non_linear/decision_tree/leaf_label_factory/majority_leaf_label_factory_test.dart rename to test/decision_tree_solver/leaf_label_factory/majority_leaf_label_factory_test.dart index fc0cd5dc..13524207 100644 --- a/test/solver/non_linear/decision_tree/leaf_label_factory/majority_leaf_label_factory_test.dart +++ b/test/decision_tree_solver/leaf_label_factory/majority_leaf_label_factory_test.dart @@ -1,12 +1,12 @@ import 'dart:collection'; import 'package:ml_algo/src/common/sequence_elements_distribution_calculator/distribution_calculator.dart'; -import 'package:ml_algo/src/solver/non_linear/decision_tree/leaf_label_factory/majority_leaf_label_factory.dart'; +import 'package:ml_algo/src/decision_tree_solver/leaf_label_factory/majority_leaf_label_factory.dart'; import 'package:ml_linalg/matrix.dart'; import 'package:mockito/mockito.dart'; import 'package:test/test.dart'; -import '../../../../test_utils/mocks.dart'; +import '../../mocks.dart'; void main() { group('MajorityLeafLabelFactory', () { diff --git a/test/solver/non_linear/decision_tree/split_selector/greedy_split_selector_test.dart b/test/decision_tree_solver/split_selector/greedy_split_selector_test.dart similarity index 96% rename from test/solver/non_linear/decision_tree/split_selector/greedy_split_selector_test.dart rename to test/decision_tree_solver/split_selector/greedy_split_selector_test.dart index 00670b55..6e03585d 100644 --- a/test/solver/non_linear/decision_tree/split_selector/greedy_split_selector_test.dart +++ b/test/decision_tree_solver/split_selector/greedy_split_selector_test.dart @@ -1,9 +1,9 @@ -import 'package:ml_algo/src/solver/non_linear/decision_tree/split_selector/greedy_split_selector.dart'; +import 'package:ml_algo/src/decision_tree_solver/split_selector/greedy_split_selector.dart'; import 'package:ml_linalg/matrix.dart'; import 'package:mockito/mockito.dart'; import 'package:test/test.dart'; -import '../../../../test_utils/mocks.dart'; +import '../../mocks.dart'; void main() { group('GreedySplitSelector', () { diff --git a/test/solver/non_linear/decision_tree/splitter/greedy_splitter_test.dart b/test/decision_tree_solver/splitter/greedy_splitter_test.dart similarity index 82% rename from test/solver/non_linear/decision_tree/splitter/greedy_splitter_test.dart rename to test/decision_tree_solver/splitter/greedy_splitter_test.dart index 45b95c24..28019aad 100644 --- a/test/solver/non_linear/decision_tree/splitter/greedy_splitter_test.dart +++ b/test/decision_tree_solver/splitter/greedy_splitter_test.dart @@ -1,12 +1,12 @@ -import 'package:ml_algo/src/solver/non_linear/decision_tree/decision_tree_node.dart'; -import 'package:ml_algo/src/solver/non_linear/decision_tree/splitter/greedy_splitter.dart'; -import 'package:ml_algo/src/solver/non_linear/decision_tree/splitter/nominal_splitter/nominal_splitter.dart'; -import 'package:ml_algo/src/solver/non_linear/decision_tree/splitter/numerical_splitter/numerical_splitter.dart'; +import 'package:ml_algo/src/decision_tree_solver/decision_tree_node.dart'; +import 'package:ml_algo/src/decision_tree_solver/splitter/greedy_splitter.dart'; +import 'package:ml_algo/src/decision_tree_solver/splitter/nominal_splitter/nominal_splitter.dart'; +import 'package:ml_algo/src/decision_tree_solver/splitter/numerical_splitter/numerical_splitter.dart'; import 'package:ml_linalg/matrix.dart'; import 'package:mockito/mockito.dart'; import 'package:test/test.dart'; -import '../../../../test_utils/mocks.dart'; +import '../../mocks.dart'; void main() { group('GreedySplitter', () { @@ -164,6 +164,47 @@ void main() { splittingValues)).called(1); }); + test('should skip the same values while iterating through the sorted ' + 'rows', () { + final samples = Matrix.fromList([ + [11], + [11], + [11], + [20], + ]); + final splittingColumnIdx = 0; + final targetColumnIdx = 1; + + final numericalSplitter = NumericalSplitterMock(); + final assessor = SplitAssessorMock(); + + when(numericalSplitter.split(samples, splittingColumnIdx, any)) + .thenReturn({}); + when(assessor.getError(any, targetColumnIdx)).thenReturn(0.5); + + final splitter = GreedySplitter(assessor, numericalSplitter, null); + + final split = splitter.split( + samples, + splittingColumnIdx, + targetColumnIdx, + null, + ); + expect(split.values, equals([])); + + final verificationResult = verify( + numericalSplitter.split( + samples, + splittingColumnIdx, + captureThat(isNotNull), + ), + ); + + verificationResult.called(1); + + expect(verificationResult.captured[0], 15.5); + }); + test('should throw an error if negative splitting index is given', () { final samples = Matrix.fromList([ [11, 22, 1, 30], diff --git a/test/solver/non_linear/decision_tree/splitter/nominal_splitter/nominal_splitter_impl_test.dart b/test/decision_tree_solver/splitter/nominal_splitter/nominal_splitter_impl_test.dart similarity index 97% rename from test/solver/non_linear/decision_tree/splitter/nominal_splitter/nominal_splitter_impl_test.dart rename to test/decision_tree_solver/splitter/nominal_splitter/nominal_splitter_impl_test.dart index c1c59902..5fd70f3e 100644 --- a/test/solver/non_linear/decision_tree/splitter/nominal_splitter/nominal_splitter_impl_test.dart +++ b/test/decision_tree_solver/splitter/nominal_splitter/nominal_splitter_impl_test.dart @@ -1,4 +1,4 @@ -import 'package:ml_algo/src/solver/non_linear/decision_tree/splitter/nominal_splitter/nominal_splitter_impl.dart'; +import 'package:ml_algo/src/decision_tree_solver/splitter/nominal_splitter/nominal_splitter_impl.dart'; import 'package:ml_linalg/matrix.dart'; import 'package:ml_linalg/vector.dart'; import 'package:test/test.dart'; diff --git a/test/solver/non_linear/decision_tree/splitter/numerical_splitter/numerical_splitter_impl_test.dart b/test/decision_tree_solver/splitter/numerical_splitter/numerical_splitter_impl_test.dart similarity index 98% rename from test/solver/non_linear/decision_tree/splitter/numerical_splitter/numerical_splitter_impl_test.dart rename to test/decision_tree_solver/splitter/numerical_splitter/numerical_splitter_impl_test.dart index 21a3e11a..76723c0c 100644 --- a/test/solver/non_linear/decision_tree/splitter/numerical_splitter/numerical_splitter_impl_test.dart +++ b/test/decision_tree_solver/splitter/numerical_splitter/numerical_splitter_impl_test.dart @@ -1,4 +1,4 @@ -import 'package:ml_algo/src/solver/non_linear/decision_tree/splitter/numerical_splitter/numerical_splitter_impl.dart'; +import 'package:ml_algo/src/decision_tree_solver/splitter/numerical_splitter/numerical_splitter_impl.dart'; import 'package:ml_linalg/matrix.dart'; import 'package:ml_linalg/vector.dart'; import 'package:test/test.dart'; diff --git a/test/solver/non_linear/decision_tree/test_utils.dart b/test/decision_tree_solver/test_utils.dart similarity index 85% rename from test/solver/non_linear/decision_tree/test_utils.dart rename to test/decision_tree_solver/test_utils.dart index 8b69d0bc..38bfd59e 100644 --- a/test/solver/non_linear/decision_tree/test_utils.dart +++ b/test/decision_tree_solver/test_utils.dart @@ -1,5 +1,5 @@ -import 'package:ml_algo/src/solver/non_linear/decision_tree/decision_tree_leaf_label.dart'; -import 'package:ml_algo/src/solver/non_linear/decision_tree/decision_tree_node.dart'; +import 'package:ml_algo/src/decision_tree_solver/decision_tree_leaf_label.dart'; +import 'package:ml_algo/src/decision_tree_solver/decision_tree_node.dart'; import 'package:ml_linalg/vector.dart'; import 'package:test/test.dart'; diff --git a/test/helpers/add_intercept.dart b/test/helpers/add_intercept_test.dart similarity index 100% rename from test/helpers/add_intercept.dart rename to test/helpers/add_intercept_test.dart diff --git a/test/helpers/features_target_split_test.dart b/test/helpers/features_target_split_test.dart new file mode 100644 index 00000000..cd47fd8e --- /dev/null +++ b/test/helpers/features_target_split_test.dart @@ -0,0 +1,101 @@ +import 'package:ml_algo/src/helpers/features_target_split.dart'; +import 'package:ml_dataframe/ml_dataframe.dart'; +import 'package:test/test.dart'; + +void main() { + group('featuresTargetSplit', () { + test('should split given dataset by target index into two parts - features ' + 'dataframe and target dataframe', () { + final dataset = DataFrame(>[ + [ 0, 1, 2, 4, 5], + [10, 20, 30, 40, 50], + [66, 77, 88, 99, 11], + [11, 22, 33, 44, 55], + ], headerExists: false); + + final splits = featuresTargetSplit(dataset, targetIndices: [3]).toList(); + + expect(splits[0].toMatrix(), equals([ + [ 0, 1, 2, 5], + [10, 20, 30, 50], + [66, 77, 88, 11], + [11, 22, 33, 55], + ])); + + expect(splits[1].toMatrix(), equals([ + [ 4], + [40], + [99], + [44], + ])); + }); + + test('should split given dataset by target indices into two parts - ' + 'features dataframe and target dataframe', () { + final dataset = DataFrame(>[ + [ 0, 1, 2, 4, 5], + [10, 20, 30, 40, 50], + [66, 77, 88, 99, 11], + [11, 22, 33, 44, 55], + ], headerExists: false); + + final splits = featuresTargetSplit(dataset, targetIndices: [0, 3]) + .toList(); + + expect(splits[0].toMatrix(), equals([ + [ 1, 2, 5], + [20, 30, 50], + [77, 88, 11], + [22, 33, 55], + ])); + + expect(splits[1].toMatrix(), equals([ + [ 0, 4], + [10, 40], + [66, 99], + [11, 44], + ])); + }); + + test('should split given dataset just by unique target indices', () { + final dataset = DataFrame(>[ + [ 0, 1, 2, 4, 5], + [10, 20, 30, 40, 50], + [66, 77, 88, 99, 11], + [11, 22, 33, 44, 55], + ], headerExists: false); + + final splits = featuresTargetSplit( + dataset, targetIndices: [0, 3, 0, 0, 3]).toList(); + + expect(splits[0].toMatrix(), equals([ + [ 1, 2, 5], + [20, 30, 50], + [77, 88, 11], + [22, 33, 55], + ])); + + expect(splits[1].toMatrix(), equals([ + [ 0, 4], + [10, 40], + [66, 99], + [11, 44], + ])); + }); + + test('should throw an error if there is an outranged index while accessing ' + 'the target split', () { + final dataset = DataFrame(>[ + [ 0, 1, 2, 4, 5], + [10, 20, 30, 40, 50], + [66, 77, 88, 99, 11], + [11, 22, 33, 44, 55], + ], headerExists: false); + + expect( + () => featuresTargetSplit(dataset, targetIndices: [0, 30]), + throwsRangeError, + ); + }); + }); +} diff --git a/test/solver/linear/convergence_detector/convergence_detector_impl_test.dart b/test/linear_optimizer/convergence_detector/convergence_detector_impl_test.dart similarity index 95% rename from test/solver/linear/convergence_detector/convergence_detector_impl_test.dart rename to test/linear_optimizer/convergence_detector/convergence_detector_impl_test.dart index 7e74056e..94d7a6fd 100644 --- a/test/solver/linear/convergence_detector/convergence_detector_impl_test.dart +++ b/test/linear_optimizer/convergence_detector/convergence_detector_impl_test.dart @@ -1,4 +1,4 @@ -import 'package:ml_algo/src/solver/linear/convergence_detector/convergence_detector_impl.dart'; +import 'package:ml_algo/src/linear_optimizer/convergence_detector/convergence_detector_impl.dart'; import 'package:test/test.dart'; void main() { diff --git a/test/solver/linear/coordinate/coordinate_optimizer_integration_test.dart b/test/linear_optimizer/coordinate_optimizer/coordinate_descent_optimizer_integration_test.dart similarity index 62% rename from test/solver/linear/coordinate/coordinate_optimizer_integration_test.dart rename to test/linear_optimizer/coordinate_optimizer/coordinate_descent_optimizer_integration_test.dart index 28b91e66..c169594f 100644 --- a/test/solver/linear/coordinate/coordinate_optimizer_integration_test.dart +++ b/test/linear_optimizer/coordinate_optimizer/coordinate_descent_optimizer_integration_test.dart @@ -1,6 +1,7 @@ import 'package:ml_algo/src/cost_function/squared.dart'; -import 'package:ml_algo/src/solver/linear/coordinate/coordinate.dart'; -import 'package:ml_algo/src/solver/linear/initial_weights_generator/initial_weights_type.dart'; +import 'package:ml_algo/src/di/injector.dart'; +import 'package:ml_algo/src/linear_optimizer/coordinate_optimizer/coordinate_descent_optimizer.dart'; +import 'package:ml_algo/src/linear_optimizer/initial_coefficients_generator/initial_coefficients_type.dart'; import 'package:ml_linalg/linalg.dart'; import 'package:ml_tech/unit_testing/matchers/iterable_almost_equal_to.dart'; import 'package:test/test.dart'; @@ -20,7 +21,7 @@ void main() { final point3 = [70.0, 80.0, 90.0]; final point4 = [20.0, 30.0, 10.0]; - CoordinateOptimizer optimizer; + CoordinateDescentOptimizer optimizer; Matrix data; Matrix labels; @@ -32,22 +33,24 @@ void main() { [20.0], [40.0] ]); - optimizer = CoordinateOptimizer( + optimizer = CoordinateDescentOptimizer( data, labels, - initialWeightsType: InitialWeightsType.zeroes, + initialWeightsType: InitialCoefficientsType.zeroes, costFunction: const SquaredCost(), - minCoefficientsDiff: 1e-5, + minCoefficientsUpdate: 1e-5, iterationsLimit: iterationsNumber, lambda: lambda); }); + tearDownAll(() => injector = null); + /// (The test case explanation)[https://github.com/gyrdym/ml_algo/wiki/Coordinate-descent-optimizer-(unregularized-case)-should-find-optimal-weights-for-the-given-data] - test('should find optimal weights for the given data', () { - final weights = optimizer.findExtrema(); + test('should find optimal coefficients for the given data', () { + final coefficients = optimizer.findExtrema(); final expected = [-81796400.0, -81295300.0, -85285400.0]; - expect(weights.rowsNum, 1); - expect(weights.columnsNum, 3); - expect(weights.getRow(0), iterableAlmostEqualTo(expected, 5.0)); + expect(coefficients.rowsNum, 3); + expect(coefficients.columnsNum, 1); + expect(coefficients.getColumn(0), iterableAlmostEqualTo(expected, 5.0)); }); }); @@ -59,7 +62,7 @@ void main() { final point2 = [20.0, 30.0, 40.0]; final point3 = [70.0, 80.0, 90.0]; - CoordinateOptimizer optimizer; + CoordinateDescentOptimizer optimizer; Matrix data; Matrix labels; @@ -70,23 +73,25 @@ void main() { [3.0], [2.0], ]); - optimizer = CoordinateOptimizer( - data, labels, - costFunction: const SquaredCost(), - isTrainDataNormalized: true, - minCoefficientsDiff: 1e-5, - iterationsLimit: iterationsNumber, - initialWeightsType: InitialWeightsType.zeroes, - lambda: lambda); + optimizer = CoordinateDescentOptimizer( + data, + labels, + costFunction: const SquaredCost(), + isFittingDataNormalized: true, + minCoefficientsUpdate: 1e-5, + iterationsLimit: iterationsNumber, + initialWeightsType: InitialCoefficientsType.zeroes, + lambda: lambda, + ); }); /// (The test case explanation)[https://github.com/gyrdym/ml_algo/wiki/Coordinate-descent-optimizer-(regularized-case)-should-find-optimal-weights-for-the-given-data] - test('should find optimal weights for the given data', () { + test('should find optimal coefficients for the given data', () { // actually, points in this example are not normalized - final weights = optimizer + final coefficients = optimizer .findExtrema() - .getRow(0); - expect(weights, equals([-4381770, -4493700, -4073630])); + .getColumn(0); + expect(coefficients, equals([-4381770, -4493700, -4073630])); }); }); } diff --git a/test/linear_optimizer/gradient/gradient_optimizer_test.dart b/test/linear_optimizer/gradient/gradient_optimizer_test.dart new file mode 100644 index 00000000..93a9d40f --- /dev/null +++ b/test/linear_optimizer/gradient/gradient_optimizer_test.dart @@ -0,0 +1,432 @@ +import 'package:injector/injector.dart'; +import 'package:ml_algo/src/cost_function/cost_function.dart'; +import 'package:ml_algo/src/di/injector.dart'; +import 'package:ml_algo/src/linear_optimizer/convergence_detector/convergence_detector.dart'; +import 'package:ml_algo/src/linear_optimizer/convergence_detector/convergence_detector_factory.dart'; +import 'package:ml_algo/src/linear_optimizer/gradient_optimizer/gradient_optimizer.dart'; +import 'package:ml_algo/src/linear_optimizer/gradient_optimizer/learning_rate_generator/learning_rate_generator.dart'; +import 'package:ml_algo/src/linear_optimizer/gradient_optimizer/learning_rate_generator/learning_rate_generator_factory.dart'; +import 'package:ml_algo/src/linear_optimizer/initial_coefficients_generator/initial_coefficients_generator.dart'; +import 'package:ml_algo/src/linear_optimizer/initial_coefficients_generator/initial_coefficients_generator_factory.dart'; +import 'package:ml_algo/src/linear_optimizer/linear_optimizer.dart'; +import 'package:ml_algo/src/math/randomizer/randomizer.dart'; +import 'package:ml_algo/src/math/randomizer/randomizer_factory.dart'; +import 'package:ml_linalg/linalg.dart'; +import 'package:ml_tech/unit_testing/matchers/iterable_2d_almost_equal_to.dart'; +import 'package:mockito/mockito.dart'; +import 'package:test/test.dart'; + +import '../../mocks.dart'; + +void main() { + group('Gradient descent optimizer', () { + final costFunction = CostFunctionMock(); + + Randomizer randomizerMock; + RandomizerFactory randomizerFactoryMock; + + LearningRateGenerator learningRateGeneratorMock; + LearningRateGeneratorFactory learningRateGeneratorFactoryMock; + + InitialCoefficientsGenerator initialWeightsGeneratorMock; + InitialCoefficientsGeneratorFactory initialWeightsGeneratorFactoryMock; + + ConvergenceDetector convergenceDetectorMock; + ConvergenceDetectorFactory convergenceDetectorFactoryMock; + + setUp(() { + randomizerMock = RandomizerMock(); + randomizerFactoryMock = createRandomizerFactoryMock(randomizerMock); + + learningRateGeneratorMock = createLearningRateGenerator(); + learningRateGeneratorFactoryMock = + createLearningRateGeneratorFactoryMock(learningRateGeneratorMock); + + initialWeightsGeneratorMock = createInitialWeightsGenerator(); + initialWeightsGeneratorFactoryMock = + createInitialWeightsGeneratorFactoryMock(initialWeightsGeneratorMock); + + convergenceDetectorMock = ConvergenceDetectorMock(); + convergenceDetectorFactoryMock = + createConvergenceDetectorFactoryMock(convergenceDetectorMock); + + injector = Injector() + ..registerDependency( + (_) => learningRateGeneratorFactoryMock) + ..registerDependency( + (_) => initialWeightsGeneratorFactoryMock) + ..registerDependency( + (_) => convergenceDetectorFactoryMock) + ..registerDependency( + (_) => randomizerFactoryMock); + }); + + tearDownAll(() => injector = null); + + test('should process `batchSize` parameter when the latter is equal to ' + '`1` (stochastic case)', () { + final points = getPoints(); + final labels = Matrix.fromList([ + [10.0], + [20.0], + [30.0], + [40.0], + ]); + final x = [ + [10.0, 20.0, 30.0] + ]; + final y = [[30.0]]; + final w1 = [ + [0.0], + [0.0], + [0.0], + ]; + final w2 = [ + [-20.0], + [-20.0], + [-20.0], + ]; + final w3 = [ + [-40.0], + [-40.0], + [-40.0], + ]; + final gradient = Matrix.fromList([ + [10.0], + [10.0], + [10.0] + ]); + final interval = [2, 3]; + + mockGetGradient(costFunction, x: x, w: w1, y: y, gradient: gradient); + mockGetGradient(costFunction, x: x, w: w2, y: y, gradient: gradient); + mockGetGradient(costFunction, x: x, w: w3, y: y, gradient: gradient); + + when(randomizerMock.getIntegerInterval(0, 4, intervalLength: 1)) + .thenReturn(interval); + + testOptimizer( + points, + labels, + costFunction, + convergenceDetectorMock, + (optimizer) { + optimizer.findExtrema(); + verifyGetGradientCall(costFunction, x: x, w: w1, y: y, calls: 1); + verifyGetGradientCall(costFunction, x: x, w: w2, y: y, calls: 1); + verifyGetGradientCall(costFunction, x: x, w: w3, y: y, calls: 1); + }, + iterations: 3, + batchSize: 1 + ); + }); + + test('should process `batchSize` parameter when the latter is equal to `2` ' + '(mini batch case, total number of points is 4)', () { + final points = getPoints(); + final labels = Matrix.fromList([ + [10.0], + [20.0], + [30.0], + [40.0], + ]); + final batch = [ + [5.0, 10.0, 15.0], + [1.0, 2.0, 3.0] + ]; + final y = [ + [10.0], + [20.0], + ]; + final grad = [ + [10.0], + [10.0], + [10.0] + ]; + + when(randomizerMock.getIntegerInterval(0, 4, intervalLength: 2)) + .thenReturn([0, 2]); + + when(costFunction.getGradient( + argThat(equals(batch)), any, argThat(equals(y)))) + .thenReturn(Matrix.fromList(grad)); + + testOptimizer( + points, + labels, + costFunction, + convergenceDetectorMock, + (optimizer) { + optimizer.findExtrema(); + verify(costFunction.getGradient( + argThat(equals(batch)), any, argThat(equals(y)))) + .called(3); // 3 iterations + }, + iterations: 3, + batchSize: 2, + ); + }); + + test('should process `batchSize` parameter when the latter is equal to `4` ' + '(batch case, total number of points is 4)', () { + final points = getPoints(); + final labels = Matrix.fromList([ + [10.0], + [20.0], + [30.0], + [40.0], + ]); + + when(randomizerMock.getIntegerInterval(0, 4, intervalLength: 4)) + .thenReturn([0, 4]); + when(costFunction.getGradient( + argThat(equals(points)), any, argThat(equals(labels)))) + .thenReturn(Matrix.fromList([ + [10.0], + [10.0], + [10.0] + ])); + + testOptimizer( + points, + labels, + costFunction, + convergenceDetectorMock, + (optimizer) { + optimizer.findExtrema(); + verify(costFunction.getGradient( + argThat(equals(points)), any, argThat(equals(labels)))) + .called(3); // 3 iterations + }, + iterations: 3, + batchSize: 4); + }); + + test('should reduce the batch size parameter to make it equal to the number ' + 'of given points', () { + final iterationLimit = 3; + final points = getPoints(); + final labels = Matrix.fromList([ + [10.0], + [20.0], + [30.0], + [40.0], + ]); + final interval = [0, 4]; + final grad = [ + [10.0], + [10.0], + [10.0], + ]; + + when(randomizerMock.getIntegerInterval(0, 4, intervalLength: 4)) + .thenReturn(interval); + when(costFunction.getGradient(any, any, any)) + .thenReturn(Matrix.fromList(grad)); + + testOptimizer( + points, + labels, + costFunction, + convergenceDetectorMock, + (optimizer) { + optimizer.findExtrema(); + verify(costFunction.getGradient( + argThat(equals(points)), any, argThat(equals(labels)))) + .called(iterationLimit); + verify(learningRateGeneratorMock.getNextValue()).called(iterationLimit); + }, + batchSize: 4, + iterations: iterationLimit, + ); + }); + + /// (Explanation of the test case)[https://github.com/gyrdym/ml_algo/wiki/Gradient-descent-optimizer-should-find-optimal-coefficient-values] + test('should find optimal coefficient values', () { + final points = Matrix.fromList([ + [1.0, 2.0, 3.0], + [4.0, 5.0, 6.0], + ]); + final labels = Matrix.fromList([ + [7.0], + [8.0], + ]); + + when(randomizerMock.getIntegerInterval(0, 2, intervalLength: 2)) + .thenReturn([0, 2]); + when(costFunction.getGradient( + argThat(equals(points)), any, argThat(equals(labels)))) + .thenReturn(Matrix.fromList([ + [8.0], + [8.0], + [8.0] + ])); + + testOptimizer( + points, + labels, + costFunction, + convergenceDetectorMock, + (optimizer) { + final optimalCoefficients = optimizer.findExtrema(); + expect(optimalCoefficients, equals([ + [-48.0], + [-48.0], + [-48.0] + ])); + expect(optimalCoefficients.columnsNum, 1); + expect(optimalCoefficients.rowsNum, 3); + }, + iterations: 3, + batchSize: 2, + ); + }); + + /// (Explanation of the test case)[https://github.com/gyrdym/ml_algo/wiki/Gradient-descent-optimizer-should-find-optimal-coefficient-values-and-regularize-it] + test('should find optimal coefficient values and regularize it', () { + final points = Matrix.fromList([ + [1.0, 2.0, 3.0], + [4.0, 5.0, 6.0], + ]); + final labels = Matrix.fromList([ + [7.0], + [8.0] + ]); + final gradient = Matrix.fromList([ + [8.0], + [8.0], + [8.0], + ]); + + when(randomizerMock.getIntegerInterval(0, 2, intervalLength: 2)) + .thenReturn([0, 2]); + when(costFunction.getGradient( + argThat(equals(points)), any, argThat(equals(labels)))) + .thenReturn(gradient); + + testOptimizer( + points, + labels, + costFunction, + convergenceDetectorMock, + (optimizer) { + final optimalCoefficients = optimizer.findExtrema(); + expect(optimalCoefficients, equals([ + [-23728.0], + [-23728.0], + [-23728.0] + ])); + expect(optimalCoefficients.columnsNum, 1); + expect(optimalCoefficients.rowsNum, 3); + }, + iterations: 3, + batchSize: 2, + lambda: 10.0, + ); + }); + + test('should consider `learningRate` parameter', () { + final initialLearningRate = 10.0; + final iterations = 3; + testOptimizer( + Matrix.fromList([[]]), + Matrix.fromList([[]]), + costFunction, + convergenceDetectorMock, + (optimizer) { + verify(learningRateGeneratorMock.init(initialLearningRate)).called(1); + }, + iterations: iterations, + batchSize: 1, + lambda: 0.0, + eta: initialLearningRate, + verifyConvergenceDetectorCall: false, + ); + }); + }); +} + + + +Matrix getPoints() => Matrix.fromList([ + [ 5, 10, 15], + [ 1, 2, 3], + [ 10, 20, 30], + [100, 200, 300], +]); + +void verifyGetGradientCall(CostFunction mock, + {Iterable> x, + Iterable> w, + Iterable> y, + int calls}) { + verify(mock.getGradient( + argThat(equals(x)), + argThat(equals(w)), + argThat(equals(y)), + )).called(calls); +} + +void testOptimizer( + Matrix points, + Matrix labels, + CostFunction costFunction, + ConvergenceDetector convergenceDetectorMock, + void callback(LinearOptimizer optimizer), { + int iterations, + int batchSize = 1, + double minCoeffUpdate = 1e-100, + double lambda = 0.0, + double eta, + bool verifyConvergenceDetectorCall = true, + }) { + + when(convergenceDetectorMock.isConverged( + any, argThat(inInclusiveRange(0, iterations - 1)))) + .thenReturn(false); + when(convergenceDetectorMock.isConverged(any, iterations)).thenReturn(true); + + final optimizer = GradientOptimizer( + points, + labels, + costFunction: costFunction, + initialLearningRate: eta, + minCoefficientsUpdate: minCoeffUpdate, + iterationLimit: iterations, + lambda: lambda, + batchSize: batchSize, + ); + + callback(optimizer); + + if (verifyConvergenceDetectorCall) { + verify(convergenceDetectorMock.isConverged(any, any)) + .called(iterations + 1); + } +} + +void mockGetGradient(CostFunction mock, { + Iterable> x, + Iterable> w, + Iterable> y, + Matrix gradient +}) { + when(mock.getGradient( + x == null ? any : argThat(iterable2dAlmostEqualTo(x)), + w == null ? any : argThat(iterable2dAlmostEqualTo(w)), + y == null ? any : argThat(iterable2dAlmostEqualTo(y)), + )).thenReturn(gradient ?? Matrix.fromList([[]])); +} + +LearningRateGenerator createLearningRateGenerator() { + final mock = LearningRateGeneratorMock(); + when(mock.getNextValue()).thenReturn(2.0); + return mock; +} + +InitialCoefficientsGenerator createInitialWeightsGenerator() { + final mock = InitialWeightsGeneratorMock(); + when(mock.generate(3)).thenReturn(Vector.fromList([0.0, 0.0, 0.0])); + return mock; +} + diff --git a/test/mocks.dart b/test/mocks.dart new file mode 100644 index 00000000..88b64f98 --- /dev/null +++ b/test/mocks.dart @@ -0,0 +1,161 @@ +import 'package:ml_algo/src/common/sequence_elements_distribution_calculator/distribution_calculator.dart'; +import 'package:ml_algo/src/cost_function/cost_function.dart'; +import 'package:ml_algo/src/cost_function/cost_function_factory.dart'; +import 'package:ml_algo/src/decision_tree_solver/decision_tree_node.dart'; +import 'package:ml_algo/src/decision_tree_solver/decision_tree_solver.dart'; +import 'package:ml_algo/src/decision_tree_solver/leaf_detector/leaf_detector.dart'; +import 'package:ml_algo/src/decision_tree_solver/leaf_label_factory/leaf_label_factory.dart'; +import 'package:ml_algo/src/decision_tree_solver/split_assessor/split_assessor.dart'; +import 'package:ml_algo/src/decision_tree_solver/split_selector/split_selector.dart'; +import 'package:ml_algo/src/decision_tree_solver/splitter/nominal_splitter/nominal_splitter.dart'; +import 'package:ml_algo/src/decision_tree_solver/splitter/numerical_splitter/numerical_splitter.dart'; +import 'package:ml_algo/src/decision_tree_solver/splitter/splitter.dart' as decision_tree_splitter; +import 'package:ml_algo/src/linear_optimizer/convergence_detector/convergence_detector.dart'; +import 'package:ml_algo/src/linear_optimizer/convergence_detector/convergence_detector_factory.dart'; +import 'package:ml_algo/src/linear_optimizer/gradient_optimizer/learning_rate_generator/learning_rate_generator.dart'; +import 'package:ml_algo/src/linear_optimizer/gradient_optimizer/learning_rate_generator/learning_rate_generator_factory.dart'; +import 'package:ml_algo/src/linear_optimizer/initial_coefficients_generator/initial_coefficients_generator.dart'; +import 'package:ml_algo/src/linear_optimizer/initial_coefficients_generator/initial_coefficients_generator_factory.dart'; +import 'package:ml_algo/src/linear_optimizer/linear_optimizer.dart'; +import 'package:ml_algo/src/linear_optimizer/linear_optimizer_factory.dart'; +import 'package:ml_algo/src/link_function/link_function.dart'; +import 'package:ml_algo/src/link_function/link_function_factory.dart'; +import 'package:ml_algo/src/math/randomizer/randomizer.dart'; +import 'package:ml_algo/src/math/randomizer/randomizer_factory.dart'; +import 'package:ml_algo/src/model_selection/assessable.dart'; +import 'package:ml_algo/src/model_selection/data_splitter/splitter.dart'; +import 'package:mockito/mockito.dart'; +import 'package:test/test.dart'; + +class RandomizerFactoryMock extends Mock implements RandomizerFactory {} + +class RandomizerMock extends Mock implements Randomizer {} + +class CostFunctionMock extends Mock implements CostFunction {} + +class CostFunctionFactoryMock extends Mock implements CostFunctionFactory {} + +class LearningRateGeneratorFactoryMock extends Mock + implements LearningRateGeneratorFactory {} + +class LearningRateGeneratorMock extends Mock implements + LearningRateGenerator {} + +class InitialWeightsGeneratorFactoryMock extends Mock + implements InitialCoefficientsGeneratorFactory {} + +class InitialWeightsGeneratorMock extends Mock + implements InitialCoefficientsGenerator {} + +class LinkFunctionMock extends Mock implements LinkFunction {} + +class LinkFunctionFactoryMock extends Mock implements LinkFunctionFactory {} + +class LinearOptimizerFactoryMock extends Mock + implements LinearOptimizerFactory {} + +class LinearOptimizerMock extends Mock implements LinearOptimizer {} + +class ConvergenceDetectorFactoryMock extends Mock + implements ConvergenceDetectorFactory {} + +class ConvergenceDetectorMock extends Mock implements ConvergenceDetector {} + +class SplitterMock extends Mock implements Splitter {} + +class AssessableMock extends Mock implements Assessable {} + +class SplitAssessorMock extends Mock implements SplitAssessor {} + +class DecisionTreeSplitterMock extends Mock implements + decision_tree_splitter.Splitter {} + +class NumericalSplitterMock extends Mock implements NumericalSplitter {} + +class NominalSplitterMock extends Mock implements NominalSplitter {} + +class DistributionCalculatorMock extends Mock implements + SequenceElementsDistributionCalculator {} + +class LeafDetectorMock extends Mock implements LeafDetector {} + +class LeafLabelFactoryMock extends Mock implements + DecisionTreeLeafLabelFactory {} + +class SplitSelectorMock extends Mock implements SplitSelector {} + +class DecisionTreeNodeMock extends Mock implements DecisionTreeNode {} + +class DecisionTreeSolverMock extends Mock implements DecisionTreeSolver {} + +LearningRateGeneratorFactoryMock createLearningRateGeneratorFactoryMock( + LearningRateGenerator generator) { + final factory = LearningRateGeneratorFactoryMock(); + when(factory.fromType(any)).thenReturn(generator); + return factory; +} + +RandomizerFactory createRandomizerFactoryMock(Randomizer randomizer) { + final factory = RandomizerFactoryMock(); + when(factory.create(any)).thenReturn(randomizer); + return factory; +} + +InitialCoefficientsGeneratorFactory createInitialWeightsGeneratorFactoryMock( + InitialCoefficientsGenerator generator) { + final factory = InitialWeightsGeneratorFactoryMock(); + when(factory.fromType(any, any)).thenReturn(generator); + return factory; +} + +LinkFunctionFactory createLinkFunctionFactoryMock( + LinkFunction linkFunctionMock) { + final factory = LinkFunctionFactoryMock(); + + when(factory.createByType(argThat(isNotNull), dtype: anyNamed('dtype'))) + .thenReturn(linkFunctionMock); + + return factory; +} + +CostFunctionFactory createCostFunctionFactoryMock( + CostFunction costFunctionMock) { + final costFunctionFactory = CostFunctionFactoryMock(); + + when(costFunctionFactory.createByType(argThat(isNotNull), + linkFunction: anyNamed('linkFunction'))).thenReturn(costFunctionMock); + + return costFunctionFactory; +} + +ConvergenceDetectorFactory createConvergenceDetectorFactoryMock( + ConvergenceDetector detector) { + final convergenceDetectorFactory = ConvergenceDetectorFactoryMock(); + when(convergenceDetectorFactory.create(any, any)).thenReturn(detector); + return convergenceDetectorFactory; +} + +LinearOptimizerFactory createLinearOptimizerFactoryMock( + LinearOptimizer optimizer) { + final factory = LinearOptimizerFactoryMock(); + + when(factory.createByType( + any, + any, + any, + dtype: anyNamed('dtype'), + costFunction: anyNamed('costFunction'), + learningRateType: anyNamed('learningRateType'), + initialCoefficientsType: anyNamed('initialCoefficientsType'), + initialLearningRate: anyNamed('initialLearningRate'), + minCoefficientsUpdate: anyNamed('minCoefficientsUpdate'), + iterationLimit: anyNamed('iterationLimit'), + lambda: anyNamed('lambda'), + regularizationType: anyNamed('regularizationType'), + batchSize: anyNamed('batchSize'), + randomSeed: anyNamed('randomSeed'), + isFittingDataNormalized: anyNamed('isFittingDataNormalized') + )).thenReturn(optimizer); + + return factory; +} diff --git a/test/model_selection/cross_validator/cross_validator_impl_test.dart b/test/model_selection/cross_validator/cross_validator_impl_test.dart new file mode 100644 index 00000000..2257317f --- /dev/null +++ b/test/model_selection/cross_validator/cross_validator_impl_test.dart @@ -0,0 +1,70 @@ +import 'package:ml_algo/src/metric/metric_type.dart'; +import 'package:ml_algo/src/model_selection/cross_validator/cross_validator_impl.dart'; +import 'package:ml_algo/src/model_selection/data_splitter/splitter.dart'; +import 'package:ml_dataframe/ml_dataframe.dart'; +import 'package:ml_linalg/dtype.dart'; +import 'package:mockito/mockito.dart'; +import 'package:test/test.dart'; + +import '../../mocks.dart'; + +Splitter createSplitter(Iterable> indices) { + final splitter = SplitterMock(); + when(splitter.split(any)).thenReturn(indices); + return splitter; +} + +void main() { + group('CrossValidatorImpl', () { + test('should perform validation of a model on given test indices of' + 'observations', () { + final allObservations = DataFrame(>[ + [330, 930, 130, 100], + [630, 830, 230, 200], + [730, 730, 330, 300], + [830, 630, 430, 400], + [930, 530, 530, 500], + [130, 430, 630, 600], + [230, 330, 730, 700], + [430, 230, 830, 800], + [530, 130, 930, 900], + ], header: ['first', 'second', 'third', 'target'], headerExists: false); + + final metric = MetricType.mape; + final splitter = createSplitter([[0,2,4],[6, 8]]); + final predictor = AssessableMock(); + final validator = CrossValidatorImpl(allObservations, + ['target'], splitter, DType.float32); + + var score = 20.0; + when(predictor.assess(any, any, any)) + .thenAnswer((Invocation inv) => score = score + 10); + + final actual = validator + .evaluate((observations, outcomes) => predictor, metric); + + expect(actual, 35); + + final verificationResult = verify( + predictor.assess( + captureThat(isNotNull), + argThat(equals(['target'])), + metric, + )); + final firstAssessCallArgs = verificationResult.captured; + + expect((firstAssessCallArgs[0] as DataFrame).rows, equals([ + [330, 930, 130, 100], + [730, 730, 330, 300], + [930, 530, 530, 500], + ])); + + expect((firstAssessCallArgs[1] as DataFrame).rows, equals([ + [230, 330, 730, 700], + [530, 130, 930, 900], + ])); + + verificationResult.called(2); + }); + }); +} diff --git a/test/data_splitter/k_fold_splitter_test.dart b/test/model_selection/data_splitter/k_fold_splitter_test.dart similarity index 100% rename from test/data_splitter/k_fold_splitter_test.dart rename to test/model_selection/data_splitter/k_fold_splitter_test.dart diff --git a/test/data_splitter/lpo_splitter_test.dart b/test/model_selection/data_splitter/lpo_splitter_test.dart similarity index 100% rename from test/data_splitter/lpo_splitter_test.dart rename to test/model_selection/data_splitter/lpo_splitter_test.dart diff --git a/test/regressor/knn_regressor_test.dart b/test/regressor/knn_regressor_impl_test.dart similarity index 56% rename from test/regressor/knn_regressor_test.dart rename to test/regressor/knn_regressor_impl_test.dart index 8003fbd9..c14b7428 100644 --- a/test/regressor/knn_regressor_test.dart +++ b/test/regressor/knn_regressor_impl_test.dart @@ -1,5 +1,6 @@ import 'package:ml_algo/src/algorithms/knn/neigbour.dart'; -import 'package:ml_algo/src/regressor/knn_regressor.dart'; +import 'package:ml_algo/src/regressor/knn_regressor_impl.dart'; +import 'package:ml_dataframe/ml_dataframe.dart'; import 'package:ml_linalg/distance.dart'; import 'package:ml_linalg/matrix.dart'; import 'package:ml_linalg/vector.dart'; @@ -9,7 +10,7 @@ void main() { final fakeKNeighbours = >>[[ Neighbour(1.0, Vector.fromList([1.0]))]]; - group('KNNRegressor', () { + group('KnnRegressorImpl', () { test('should consider k parameter', () { final solverFn = (int k, Matrix trainObservations, Matrix outcomes, Matrix observations, {Distance distance}) { @@ -17,13 +18,16 @@ void main() { return fakeKNeighbours; }; - KNNRegressor( + KnnRegressorImpl( Matrix.fromList([[1.0]]), Matrix.fromList([[1.0]]), + 'class_name', k: 1, distance: Distance.cosine, - solverFn: solverFn - )..predict(Matrix.fromList([[1.0]])); + solverFn: solverFn, + )..predict( + DataFrame([[1]], headerExists: false), + ); }); test('should pass train observations to the solver function', () { @@ -36,14 +40,22 @@ void main() { return fakeKNeighbours; }; - KNNRegressor(Matrix.fromList([ - [1.0, 2.0, 3.0], - [4.0, 5.0, 6.0], - ]), Matrix.fromList([ - [1.0], - [2.0] - ]), k: 2, distance: Distance.cosine, solverFn: solverFn) - ..predict(Matrix.fromList([[10.0, 20.0, 30.0]])); + KnnRegressorImpl( + Matrix.fromList([ + [1.0, 2.0, 3.0], + [4.0, 5.0, 6.0], + ]), + Matrix.fromList([ + [1.0], + [2.0], + ]), + 'class_name', + k: 2, + distance: Distance.cosine, + solverFn: solverFn, + )..predict( + DataFrame([[10, 20, 30]], headerExists: false), + ); }); test('should pass train outcomes to the solver function', () { @@ -51,19 +63,25 @@ void main() { Matrix observations, {Distance distance}) { expect(trainOutcomes, equals([ [1.0], - [2.0] + [2.0], ])); return fakeKNeighbours; }; - KNNRegressor(Matrix.fromList([ - [1.0, 2.0, 3.0], - [4.0, 5.0, 6.0], - ]), Matrix.fromList([ - [1.0], - [2.0] - ]), k: 2, distance: Distance.cosine, solverFn: solverFn) - ..predict(Matrix.fromList([[10.0, 20.0, 30.0]])); + KnnRegressorImpl( + Matrix.fromList([ + [1.0, 2.0, 3.0], + [4.0, 5.0, 6.0], + ]), + Matrix.fromList([ + [1.0], + [2.0] + ]), + 'class_name', + k: 2, + distance: Distance.cosine, + solverFn: solverFn, + )..predict(DataFrame([[10, 20, 30]], headerExists: false)); }); test('should pass observations to the solver function', () { @@ -73,14 +91,20 @@ void main() { return fakeKNeighbours; }; - KNNRegressor(Matrix.fromList([ - [1.0, 2.0, 3.0], - [4.0, 5.0, 6.0], - ]), Matrix.fromList([ - [1.0], - [2.0] - ]), k: 2, distance: Distance.cosine, solverFn: solverFn) - ..predict(Matrix.fromList([[10.0, 20.0, 30.0]])); + KnnRegressorImpl( + Matrix.fromList([ + [1.0, 2.0, 3.0], + [4.0, 5.0, 6.0], + ]), + Matrix.fromList([ + [1.0], + [2.0] + ]), + 'class_name', + k: 2, + distance: Distance.cosine, + solverFn: solverFn, + )..predict(DataFrame([[10, 20, 30]], headerExists: false)); }); test('should consider distance type', () { @@ -90,13 +114,14 @@ void main() { return fakeKNeighbours; }; - KNNRegressor( + KnnRegressorImpl( Matrix.fromList([[1.0]]), Matrix.fromList([[1.0]]), + 'class_name', k: 1, distance: Distance.cosine, - solverFn: solverFn) - ..predict(Matrix.fromList([[1.0]])); + solverFn: solverFn, + )..predict(DataFrame([[1]], headerExists: false)); }); test('should throw an exception if number of training observations and ' @@ -104,10 +129,17 @@ void main() { final solverFn = (int k, Matrix trainObservations, Matrix outcomes, Matrix observations, {Distance distance}) => fakeKNeighbours; expect( - () => KNNRegressor(Matrix.fromList([[1.0, 2,0]]), Matrix.fromList([ - [1.0], - [3.0], - ]), k: 1, distance: Distance.cosine, solverFn: solverFn), + () => KnnRegressorImpl( + Matrix.fromList([[1.0, 2,0]]), + Matrix.fromList([ + [1.0], + [3.0], + ]), + 'class_name', + k: 1, + distance: Distance.cosine, + solverFn: solverFn, + ), throwsException, ); }); @@ -117,8 +149,14 @@ void main() { final solverFn = (int k, Matrix trainObservations, Matrix outcomes, Matrix observations, {Distance distance}) => fakeKNeighbours; expect( - () => KNNRegressor(Matrix.fromList([[1.0, 2,0]]), Matrix.fromList([ - [1.0],]), k: 3, distance: Distance.cosine, solverFn: solverFn), + () => KnnRegressorImpl( + Matrix.fromList([[1.0, 2,0]]), + Matrix.fromList([[1.0]]), + 'class_name', + k: 3, + distance: Distance.cosine, + solverFn: solverFn, + ), throwsException, ); }); diff --git a/test/regressor/knn_regressor_integration_test.dart b/test/regressor/knn_regressor_integration_test.dart index 4b3ed575..deb37a53 100644 --- a/test/regressor/knn_regressor_integration_test.dart +++ b/test/regressor/knn_regressor_integration_test.dart @@ -1,59 +1,64 @@ import 'package:ml_algo/ml_algo.dart'; +import 'package:ml_dataframe/ml_dataframe.dart'; import 'package:ml_linalg/matrix.dart'; import 'package:test/test.dart'; void main() { - group('KNNRegressor (integration)', () { - test('should predict values with help of uniform kernel', () { + group('KnnRegressor', () { + test('should predict values using uniform kernel', () { final k = 2; - final features = Matrix.fromList([ - [20, 20, 20, 20, 20], - [30, 30, 30, 30, 30], - [15, 15, 15, 15, 15], - [25, 25, 25, 25, 25], - [10, 10, 10, 10, 10], - ]); - final outcomes = Matrix.fromList([ - [1.0], - [2.0], - [3.0], - [4.0], - [5.0], - ]); + final data = DataFrame(>[ + [20, 20, 20, 20, 20, 1], + [30, 30, 30, 30, 30, 2], + [15, 15, 15, 15, 15, 3], + [25, 25, 25, 25, 25, 4], + [10, 10, 10, 10, 10, 5], + ], + header: ['first', 'second', 'third', 'fourth', 'fifth', 'target'], + headerExists: false); + final testFeatures = Matrix.fromList([ [9.0, 9.0, 9.0, 9.0, 9.0], ]); - final regressor = ParameterlessRegressor.knn(features, - outcomes, k: k); - final actual = regressor.predict(testFeatures); - expect(actual, equals([[4.0]])); + final regressor = KnnRegressor(data, 'target', k: k); + + final actual = regressor.predict( + DataFrame.fromMatrix(testFeatures), + ); + + expect(actual.header, equals(['target'])); + expect(actual.toMatrix(), equals([[4.0]])); }); - test('should predict values with help of epanechnikov kernel', () { + test('should predict values using epanechnikov kernel', () { final k = 2; - final features = Matrix.fromList([ - [20, 20, 20, 20, 20], - [30, 30, 30, 30, 30], - [15, 15, 15, 15, 15], - [25, 25, 25, 25, 25], - [10, 10, 10, 10, 10], - ]); - final outcomes = Matrix.fromList([ - [1.0], - [2.0], - [3.0], - [4.0], - [5.0], - ]); + final data = DataFrame(>[ + [20, 20, 20, 20, 20, 1], + [30, 30, 30, 30, 30, 2], + [15, 15, 15, 15, 15, 3], + [25, 25, 25, 25, 25, 4], + [10, 10, 10, 10, 10, 5], + ], + header: ['first', 'second', 'third', 'fourth', 'fifth', 'target'], + headerExists: false + ); + final testFeatures = Matrix.fromList([ [9.0, 9.0, 9.0, 9.0, 9.0], ]); - final regressor = ParameterlessRegressor.knn(features, - outcomes, k: k, kernel: Kernel.epanechnikov); - final actual = regressor.predict(testFeatures); - expect(actual, equals([[-208.875]])); + final regressor = KnnRegressor(data, 'target', + k: k, + kernel: Kernel.epanechnikov, + ); + + final actual = regressor.predict( + DataFrame.fromMatrix(testFeatures), + ); + + expect(actual.header, equals(['target'])); + expect(actual.toMatrix(), equals([[-208.875]])); }); }); } diff --git a/test/regressor/linear_regressor_test.dart b/test/regressor/linear_regressor_test.dart new file mode 100644 index 00000000..80608e3c --- /dev/null +++ b/test/regressor/linear_regressor_test.dart @@ -0,0 +1,176 @@ +import 'package:injector/injector.dart'; +import 'package:ml_algo/ml_algo.dart'; +import 'package:ml_algo/src/cost_function/cost_function.dart'; +import 'package:ml_algo/src/cost_function/cost_function_factory.dart'; +import 'package:ml_algo/src/cost_function/cost_function_type.dart'; +import 'package:ml_algo/src/di/injector.dart'; +import 'package:ml_algo/src/linear_optimizer/initial_coefficients_generator/initial_coefficients_type.dart'; +import 'package:ml_algo/src/linear_optimizer/linear_optimizer.dart'; +import 'package:ml_algo/src/linear_optimizer/linear_optimizer_factory.dart'; +import 'package:ml_algo/src/linear_optimizer/linear_optimizer_type.dart'; +import 'package:ml_algo/src/linear_optimizer/regularization_type.dart'; +import 'package:ml_dataframe/ml_dataframe.dart'; +import 'package:ml_linalg/dtype.dart'; +import 'package:ml_linalg/linalg.dart'; +import 'package:ml_tech/unit_testing/matchers/iterable_2d_almost_equal_to.dart'; +import 'package:mockito/mockito.dart'; +import 'package:test/test.dart'; + +import '../mocks.dart'; + +void main() { + group('LinearRegressor', () { + final initialCoefficients = Matrix.fromList([ + [1], + [2], + [3], + [4], + [5], + ]); + + final learnedCoefficients = Matrix.fromColumns([ + Vector.fromList([55, 66, 77, 88, 99]), + ]); + + final observations = DataFrame( + [ + [10, 20, 30, 40, 200], + [11, 22, 33, 44, 500], + ], + header: ['feature_1', 'feature_2', 'feature_3', 'feature_4', 'target'], + headerExists: false, + ); + + CostFunction costFunctionMock; + CostFunctionFactory costFunctionFactoryMock; + + LinearOptimizer linearOptimizerMock; + LinearOptimizerFactory linearOptimizerFactoryMock; + + setUp(() { + costFunctionMock = CostFunctionMock(); + costFunctionFactoryMock = createCostFunctionFactoryMock(costFunctionMock); + + linearOptimizerMock = LinearOptimizerMock(); + linearOptimizerFactoryMock = createLinearOptimizerFactoryMock( + linearOptimizerMock); + + injector = Injector() + ..registerDependency( + (_) => costFunctionFactoryMock) + ..registerDependency( + (_) => linearOptimizerFactoryMock); + + when(linearOptimizerMock.findExtrema( + initialCoefficients: anyNamed('initialCoefficients'), + isMinimizingObjective: anyNamed('isMinimizingObjective'), + )).thenReturn(learnedCoefficients); + }); + + tearDownAll(() => injector = null); + + test('should throw an error if the target column does not exist', () { + final targetColumn = 'absent_column'; + expect( + () => LinearRegressor(observations, targetColumn), + throwsException, + ); + }); + + test('should call cost function factory in order to create ' + 'squared cost function instance', () { + LinearRegressor(observations, 'target'); + + verify(costFunctionFactoryMock.createByType( + CostFunctionType.squared, + )).called(1); + }); + + test('should call linear optimizer factory and consider intercept term ' + 'while calling the factory', () { + LinearRegressor(observations, 'target', + optimizerType: LinearOptimizerType.coordinate, + iterationsLimit: 1000, + initialLearningRate: 5, + minCoefficientsUpdate: 1000, + lambda: 20.0, + regularizationType: RegularizationType.L1, + randomSeed: 200, + batchSize: 100, + fitIntercept: true, + interceptScale: 3.0, + learningRateType: LearningRateType.decreasingAdaptive, + initialCoefficientsType: InitialCoefficientsType.zeroes, + initialCoefficients: initialCoefficients, + dtype: DType.float32, + ); + + verify(linearOptimizerFactoryMock.createByType( + LinearOptimizerType.coordinate, + argThat(iterable2dAlmostEqualTo([ + [3.0, 10, 20, 30, 40], + [3.0, 11, 22, 33, 44], + ])), + argThat(equals([ + [200], + [500], + ])), + dtype: DType.float32, + costFunction: costFunctionMock, + learningRateType: LearningRateType.decreasingAdaptive, + initialCoefficientsType: InitialCoefficientsType.zeroes, + initialLearningRate: 5, + minCoefficientsUpdate: 1000, + iterationLimit: 1000, + lambda: 20.0, + regularizationType: RegularizationType.L1, + batchSize: 100, + randomSeed: 200, + isFittingDataNormalized: false, + )).called(1); + }); + + test('should find the extrema for fitting observations while ' + 'instantiating', () { + LinearRegressor(observations, 'target', + initialCoefficients: initialCoefficients, + ); + + verify(linearOptimizerMock.findExtrema( + initialCoefficients: initialCoefficients, + isMinimizingObjective: true, + )).called(1); + }); + + test('should predict values basing on learned coefficients', () { + final predictor = LinearRegressor( + observations, + 'target', + initialCoefficients: initialCoefficients, + fitIntercept: true, + interceptScale: 2.0, + ); + + final features = Matrix.fromList([ + [55, 44, 33, 22], + [10, 88, 77, 11], + [12, 22, 39, 13], + ]); + + final featuresWithIntercept = Matrix.fromColumns([ + Vector.filled(3, 2), + ...features.columns, + ]); + + final prediction = predictor.predict( + DataFrame.fromMatrix(features), + ); + + expect(prediction.header, equals(['target'])); + + expect(prediction.toMatrix(), equals( + featuresWithIntercept * learnedCoefficients, + )); + }); + }); +} diff --git a/test/solver/linear/gradient/gradient_common.dart b/test/solver/linear/gradient/gradient_common.dart deleted file mode 100644 index fb21fbb7..00000000 --- a/test/solver/linear/gradient/gradient_common.dart +++ /dev/null @@ -1,116 +0,0 @@ -import 'package:ml_algo/src/cost_function/cost_function.dart'; -import 'package:ml_algo/src/math/randomizer/randomizer.dart'; -import 'package:ml_algo/src/solver/linear/convergence_detector/convergence_detector.dart'; -import 'package:ml_algo/src/solver/linear/gradient/gradient.dart'; -import 'package:ml_algo/src/solver/linear/gradient/learning_rate_generator/learning_rate_generator.dart'; -import 'package:ml_algo/src/solver/linear/initial_weights_generator/initial_weights_generator.dart'; -import 'package:ml_algo/src/solver/linear/linear_optimizer.dart'; -import 'package:ml_linalg/matrix.dart'; -import 'package:ml_linalg/vector.dart'; -import 'package:ml_tech/unit_testing/matchers/iterable_2d_almost_equal_to.dart'; -import 'package:mockito/mockito.dart'; -import 'package:test/test.dart'; - -import '../../../test_utils/mocks.dart'; - -Randomizer randomizerMock = RandomizerMock(); -LearningRateGenerator learningRateGeneratorMock; -InitialWeightsGenerator initialWeightsGeneratorMock; -ConvergenceDetector convergenceDetectorMock = ConvergenceDetectorMock(); - -LearningRateGenerator createLearningRateGenerator() { - final mock = LearningRateGeneratorMock(); - when(mock.getNextValue()).thenReturn(2.0); - return mock; -} - -InitialWeightsGenerator createInitialWeightsGenerator() { - final mock = InitialWeightsGeneratorMock(); - when(mock.generate(3)).thenReturn(Vector.fromList([0.0, 0.0, 0.0])); - return mock; -} - -GradientOptimizer createOptimizer(Matrix points, Matrix labels, - CostFunction costFunction, { - double eta, - double minCoeffUpdate, - int iterationsLimit, - double lambda, - int batchSize -}) { - learningRateGeneratorMock = createLearningRateGenerator(); - initialWeightsGeneratorMock = createInitialWeightsGenerator(); - - final randomizerFactoryMock = RandomizerFactoryMock(); - final learningRateGeneratorFactoryMock = LearningRateGeneratorFactoryMock(); - final initialWeightsGeneratorFactoryMock = - InitialWeightsGeneratorFactoryMock(); - final convergenceDetectorFactoryMock = ConvergenceDetectorFactoryMock(); - - when(randomizerFactoryMock.create(any)).thenReturn(randomizerMock); - when(learningRateGeneratorFactoryMock.fromType(any)) - .thenReturn(learningRateGeneratorMock); - when(initialWeightsGeneratorFactoryMock.fromType(any, any)) - .thenReturn(initialWeightsGeneratorMock); - when(convergenceDetectorFactoryMock.create(any, any)) - .thenReturn(convergenceDetectorMock); - - return GradientOptimizer( - points, labels, - randomizerFactory: randomizerFactoryMock, - costFunction: costFunction, - learningRateGeneratorFactory: learningRateGeneratorFactoryMock, - initialWeightsGeneratorFactory: initialWeightsGeneratorFactoryMock, - convergenceDetectorFactory: convergenceDetectorFactoryMock, - initialLearningRate: eta, - minCoefficientsUpdate: minCoeffUpdate, - iterationLimit: iterationsLimit, - lambda: lambda, - batchSize: batchSize); -} - -void mockGetGradient(CostFunction mock, { - Iterable> x, - Iterable> w, - Iterable> y, - Matrix gradient -}) { - when(mock.getGradient( - x == null ? any : argThat(iterable2dAlmostEqualTo(x)), - w == null ? any : argThat(iterable2dAlmostEqualTo(w)), - y == null ? any : argThat(iterable2dAlmostEqualTo(y)), - )).thenReturn(gradient ?? Matrix.fromList([[]])); -} - -void testOptimizer( - Matrix points, Matrix labels, CostFunction costFunction, - void callback(LinearOptimizer optimizer), { - int iterations, - int batchSize = 1, - double minCoeffUpdate = 1e-100, - double lambda = 0.0, - double eta, - bool verifyConvergenceDetectorCall = true, -}) { - when(convergenceDetectorMock.isConverged( - any, argThat(inInclusiveRange(0, iterations - 1)))) - .thenReturn(false); - when(convergenceDetectorMock.isConverged(any, iterations)).thenReturn(true); - - final optimizer = createOptimizer( - points, - labels, - costFunction, - minCoeffUpdate: minCoeffUpdate, - iterationsLimit: iterations, - lambda: lambda, - eta: eta, - batchSize: batchSize); - - callback(optimizer); - - if (verifyConvergenceDetectorCall) { - verify(convergenceDetectorMock.isConverged(any, any)) - .called(iterations + 1); - } -} diff --git a/test/solver/linear/gradient/gradient_optimizer_integration_test.dart b/test/solver/linear/gradient/gradient_optimizer_integration_test.dart deleted file mode 100644 index 3d1b4a8c..00000000 --- a/test/solver/linear/gradient/gradient_optimizer_integration_test.dart +++ /dev/null @@ -1,22 +0,0 @@ -// -// Implement the following scenario: -// -// coeffs = [0, 0, 0] -// -// [1, 2, 3] [7] -// [4, 5, 6] [8] -// -// iteration 1: -// eta = 2 -// -// c_1 = c_1 - 2 * eta * x_1_1 * (7 - coeffs . x_1) = 0 - 2 * 2 * 1 * (7 - 0) = 0 - 4 * 7 = -28 -// c_2 = c_2 - 2 * eta * x_1_2 * (7 - coeffs . x_1) = 0 - 2 * 2 * 2 * (7 - 0) = 0 - 8 * 7 = -56 -// c_3 = c_3 - 2 * eta * x_1_3 * (7 - coeffs . x_1) = 0 - 2 * 2 * 3 * (7 - 0) = 0 - 12 * 7 = -84 -// -// c_1 = c_1 - 2 * eta * x_2_1 * (8 - coeffs . x_2) = 0 - 2 * 2 * 4 * (8 - 0) = 0 - 16 * 8 = -128 -// c_2 = c_2 - 2 * eta * x_2_2 * (8 - coeffs . x_2) = 0 - 2 * 2 * 5 * (8 - 0) = 0 - 20 * 8 = - 160 -// c_3 = c_3 - 2 * eta * x_2_3 * (8 - coeffs . x_2) = 0 - 2 * 2 * 6 * (8 - 0) = 0 - 24 * 8 = -192 -// -// coeffs = [-28, -56, -84] + [-128, -160, -192] = [-156, -216, -276] - -void main() {} diff --git a/test/solver/linear/gradient/gradient_optimizer_test.dart b/test/solver/linear/gradient/gradient_optimizer_test.dart deleted file mode 100644 index 480f2ef8..00000000 --- a/test/solver/linear/gradient/gradient_optimizer_test.dart +++ /dev/null @@ -1,260 +0,0 @@ -import 'package:ml_algo/src/cost_function/cost_function.dart'; -import 'package:ml_linalg/linalg.dart'; -import 'package:mockito/mockito.dart'; -import 'package:test/test.dart'; - -import '../../../test_utils/mocks.dart'; -import 'gradient_common.dart'; - -Matrix getPoints() => Matrix.fromList([ - [5.0, 10.0, 15.0], - [1.0, 2.0, 3.0], - [10.0, 20.0, 30.0], - [100.0, 200.0, 300.0], - ]); - -void verifyGetGradientCall(CostFunction mock, - {Iterable> x, - Iterable> w, - Iterable> y, - int calls}) { - verify(mock.getGradient( - argThat(equals(x)), - argThat(equals(w)), - argThat(equals(y)), - )).called(calls); -} - -void main() { - group('Gradient descent solver', () { - final costFunction = CostFunctionMock(); - - test('should properly process `batchSize` parameter when the latter is ' - 'equal to `1` (stochastic case)', () { - final points = getPoints(); - final labels = Matrix.fromList([ - [10.0], - [20.0], - [30.0], - [40.0], - ]); - final x = [ - [10.0, 20.0, 30.0] - ]; - final y = [[30.0]]; - final w1 = [ - [0.0], - [0.0], - [0.0], - ]; - final w2 = [ - [-20.0], - [-20.0], - [-20.0], - ]; - final w3 = [ - [-40.0], - [-40.0], - [-40.0], - ]; - final grad = Matrix.fromList([ - [10.0], - [10.0], - [10.0] - ]); - final interval = [2, 3]; - - mockGetGradient(costFunction, x: x, w: w1, y: y, gradient: grad); - mockGetGradient(costFunction, x: x, w: w2, y: y, gradient: grad); - mockGetGradient(costFunction, x: x, w: w3, y: y, gradient: grad); - - when(randomizerMock.getIntegerInterval(0, 4, intervalLength: 1)) - .thenReturn(interval); - - testOptimizer(points, labels, costFunction, (optimizer) { - optimizer.findExtrema(); - verifyGetGradientCall(costFunction, x: x, w: w1, y: y, calls: 1); - verifyGetGradientCall(costFunction, x: x, w: w2, y: y, calls: 1); - verifyGetGradientCall(costFunction, x: x, w: w3, y: y, calls: 1); - }, iterations: 3, batchSize: 1); - }); - - test('should properly process `batchSize` parameter when the latter is ' - 'equal to `2` (mini batch case)', () { - final points = getPoints(); - final labels = Matrix.fromList([ - [10.0], - [20.0], - [30.0], - [40.0], - ]); - final batch = [ - [5.0, 10.0, 15.0], - [1.0, 2.0, 3.0] - ]; - final y = [ - [10.0], - [20.0], - ]; - final grad = [ - [10.0], - [10.0], - [10.0] - ]; - - when(randomizerMock.getIntegerInterval(0, 4, intervalLength: 2)) - .thenReturn([0, 2]); - - when(costFunction.getGradient( - argThat(equals(batch)), any, argThat(equals(y)))) - .thenReturn(Matrix.fromList(grad)); - - testOptimizer(points, labels, costFunction, (optimizer) { - optimizer.findExtrema(); - verify(costFunction.getGradient( - argThat(equals(batch)), any, argThat(equals(y)))) - .called(3); // 3 iterations - }, iterations: 3, batchSize: 2); - }); - - test('should properly process `batchSize` parameter when the latter is ' - 'equal to `4` (batch case)', () { - final points = getPoints(); - final labels = Matrix.fromList([ - [10.0], - [20.0], - [30.0], - [40.0], - ]); - - when(randomizerMock.getIntegerInterval(0, 4, intervalLength: 4)) - .thenReturn([0, 4]); - when(costFunction.getGradient( - argThat(equals(points)), any, argThat(equals(labels)))) - .thenReturn(Matrix.fromList([ - [10.0], - [10.0], - [10.0] - ])); - - testOptimizer(points, labels, costFunction, (optimizer) { - optimizer.findExtrema(); - verify(costFunction.getGradient( - argThat(equals(points)), any, argThat(equals(labels)))) - .called(3); // 3 iterations - }, iterations: 3, batchSize: 4); - }); - - test('should cut the batch size parameter to make it equal to the number ' - 'of given points', () { - final iterationLimit = 3; - final points = getPoints(); - final labels = Matrix.fromList([ - [10.0], - [20.0], - [30.0], - [40.0], - ]); - final interval = [0, 4]; - final grad = [ - [10.0], - [10.0], - [10.0], - ]; - - when(randomizerMock.getIntegerInterval(0, 4, intervalLength: 4)) - .thenReturn(interval); - when(costFunction.getGradient(any, any, any)) - .thenReturn(Matrix.fromList(grad)); - - testOptimizer(points, labels, costFunction, (optimizer) { - optimizer.findExtrema(); - verify(costFunction.getGradient( - argThat(equals(points)), any, argThat(equals(labels)))) - .called(iterationLimit); - verify(learningRateGeneratorMock.getNextValue()).called(iterationLimit); - }, batchSize: 4, iterations: iterationLimit); - }); - - /// (Explanation of the test case)[https://github.com/gyrdym/ml_algo/wiki/Gradient-descent-optimizer-should-find-optimal-coefficient-values] - test('should find optimal coefficient values', () { - final points = Matrix.fromList([ - [1.0, 2.0, 3.0], - [4.0, 5.0, 6.0], - ]); - final labels = Matrix.fromList([ - [7.0], - [8.0], - ]); - - when(randomizerMock.getIntegerInterval(0, 2, intervalLength: 2)) - .thenReturn([0, 2]); - when(costFunction.getGradient( - argThat(equals(points)), any, argThat(equals(labels)))) - .thenReturn(Matrix.fromList([ - [8.0], - [8.0], - [8.0] - ])); - - testOptimizer(points, labels, costFunction, (optimizer) { - final optimalCoefficients = optimizer.findExtrema(); - expect(optimalCoefficients, equals([ - [-48.0], - [-48.0], - [-48.0] - ])); - expect(optimalCoefficients.columnsNum, 1); - expect(optimalCoefficients.rowsNum, 3); - }, iterations: 3, batchSize: 2); - }); - - /// (Explanation of the test case)[https://github.com/gyrdym/ml_algo/wiki/Gradient-descent-optimizer-should-find-optimal-coefficient-values-and-regularize-it] - test('should find optimal coefficient values and regularize it', () { - final points = Matrix.fromList([ - [1.0, 2.0, 3.0], - [4.0, 5.0, 6.0], - ]); - final labels = Matrix.fromList([ - [7.0], - [8.0] - ]); - final gradient = Matrix.fromList([ - [8.0], - [8.0], - [8.0], - ]); - - when(randomizerMock.getIntegerInterval(0, 2, intervalLength: 2)) - .thenReturn([0, 2]); - when(costFunction.getGradient( - argThat(equals(points)), any, argThat(equals(labels)))) - .thenReturn(gradient); - - testOptimizer(points, labels, costFunction, (optimizer) { - final optimalCoefficients = optimizer.findExtrema(); - expect(optimalCoefficients, equals([ - [-23728.0], - [-23728.0], - [-23728.0] - ])); - expect(optimalCoefficients.columnsNum, 1); - expect(optimalCoefficients.rowsNum, 3); - }, iterations: 3, batchSize: 2, lambda: 10.0); - }); - - test('should consider `learningRate` parameter', () { - final initialLearningRate = 10.0; - final iterations = 3; - testOptimizer(Matrix.fromList([[]]), Matrix.fromList([[]]), costFunction, - (optimizer) { - verify(learningRateGeneratorMock.init(initialLearningRate)).called(1); - }, - iterations: iterations, - batchSize: 1, - lambda: 0.0, - eta: initialLearningRate, - verifyConvergenceDetectorCall: false); - }); - }); -} diff --git a/test/test_utils/mocks.dart b/test/test_utils/mocks.dart deleted file mode 100644 index fc799590..00000000 --- a/test/test_utils/mocks.dart +++ /dev/null @@ -1,142 +0,0 @@ -import 'package:ml_algo/src/common/sequence_elements_distribution_calculator/distribution_calculator.dart'; -import 'package:ml_algo/src/cost_function/cost_function.dart'; -import 'package:ml_algo/src/link_function/link_function.dart'; -import 'package:ml_algo/src/math/randomizer/randomizer.dart'; -import 'package:ml_algo/src/math/randomizer/randomizer_factory.dart'; -import 'package:ml_algo/src/model_selection/assessable.dart'; -import 'package:ml_algo/src/model_selection/data_splitter/splitter.dart'; -import 'package:ml_algo/src/solver/linear/convergence_detector/convergence_detector.dart'; -import 'package:ml_algo/src/solver/linear/convergence_detector/convergence_detector_factory.dart'; -import 'package:ml_algo/src/solver/linear/gradient/learning_rate_generator/learning_rate_generator.dart'; -import 'package:ml_algo/src/solver/linear/gradient/learning_rate_generator/learning_rate_generator_factory.dart'; -import 'package:ml_algo/src/solver/linear/gradient/learning_rate_generator/learning_rate_type.dart'; -import 'package:ml_algo/src/solver/linear/initial_weights_generator/initial_weights_generator.dart'; -import 'package:ml_algo/src/solver/linear/initial_weights_generator/initial_weights_generator_factory.dart'; -import 'package:ml_algo/src/solver/linear/initial_weights_generator/initial_weights_type.dart'; -import 'package:ml_algo/src/solver/linear/linear_optimizer.dart'; -import 'package:ml_algo/src/solver/linear/linear_optimizer_factory.dart'; -import 'package:ml_algo/src/solver/non_linear/decision_tree/decision_tree_node.dart'; -import 'package:ml_algo/src/solver/non_linear/decision_tree/decision_tree_solver.dart'; -import 'package:ml_algo/src/solver/non_linear/decision_tree/leaf_detector/leaf_detector.dart'; -import 'package:ml_algo/src/solver/non_linear/decision_tree/leaf_label_factory/leaf_label_factory.dart'; -import 'package:ml_algo/src/solver/non_linear/decision_tree/split_assessor/split_assessor.dart'; -import 'package:ml_algo/src/solver/non_linear/decision_tree/split_selector/split_selector.dart'; -import 'package:ml_algo/src/solver/non_linear/decision_tree/splitter/nominal_splitter/nominal_splitter.dart'; -import 'package:ml_algo/src/solver/non_linear/decision_tree/splitter/numerical_splitter/numerical_splitter.dart'; -import 'package:ml_algo/src/solver/non_linear/decision_tree/splitter/splitter.dart' as decision_tree_splitter; -import 'package:ml_linalg/matrix.dart'; -import 'package:mockito/mockito.dart'; - -class RandomizerFactoryMock extends Mock implements RandomizerFactory {} - -class RandomizerMock extends Mock implements Randomizer {} - -class CostFunctionMock extends Mock implements CostFunction {} - -class LearningRateGeneratorFactoryMock extends Mock - implements LearningRateGeneratorFactory {} - -class LearningRateGeneratorMock extends Mock implements - LearningRateGenerator {} - -class InitialWeightsGeneratorFactoryMock extends Mock - implements InitialWeightsGeneratorFactory {} - -class InitialWeightsGeneratorMock extends Mock - implements InitialWeightsGenerator {} - -class LinkFunctionMock extends Mock implements LinkFunction {} - -class OptimizerFactoryMock extends Mock implements LinearOptimizerFactory {} - -class OptimizerMock extends Mock implements LinearOptimizer {} - -class ConvergenceDetectorFactoryMock extends Mock - implements ConvergenceDetectorFactory {} - -class ConvergenceDetectorMock extends Mock implements ConvergenceDetector {} - -class SplitterMock extends Mock implements Splitter {} - -class PredictorMock extends Mock implements Assessable {} - -class SplitAssessorMock extends Mock implements SplitAssessor {} - -class DecisionTreeSplitterMock extends Mock implements - decision_tree_splitter.Splitter {} - -class NumericalSplitterMock extends Mock implements NumericalSplitter {} - -class NominalSplitterMock extends Mock implements NominalSplitter {} - -class DistributionCalculatorMock extends Mock implements - SequenceElementsDistributionCalculator {} - -class LeafDetectorMock extends Mock implements LeafDetector {} - -class LeafLabelFactoryMock extends Mock implements - DecisionTreeLeafLabelFactory {} - -class SplitSelectorMock extends Mock implements SplitSelector {} - -class DecisionTreeNodeMock extends Mock implements DecisionTreeNode {} - -class DecisionTreeSolverMock extends Mock implements DecisionTreeSolver {} - -LearningRateGeneratorFactoryMock createLearningRateGeneratorFactoryMock({ - Map generators, -}) { - final factory = LearningRateGeneratorFactoryMock(); - generators.forEach((LearningRateType type, LearningRateGenerator generator) { - when(factory.fromType(type)).thenReturn(generator); - }); - return factory; -} - -RandomizerFactoryMock createRandomizerFactoryMock({ - Map randomizers, -}) { - final factory = RandomizerFactoryMock(); - randomizers.forEach((int seed, Randomizer randomizer) { - when(factory.create(seed)).thenReturn(randomizer); - }); - return factory; -} - -InitialWeightsGeneratorFactoryMock createInitialWeightsGeneratorFactoryMock({ - Map generators, -}) { - final factory = InitialWeightsGeneratorFactoryMock(); - generators - .forEach((InitialWeightsType type, InitialWeightsGenerator generator) { - when(factory.fromType(type, any)).thenReturn(generator); - }); - return factory; -} - -OptimizerFactoryMock createGradientOptimizerFactoryMock( - Matrix points, - Matrix labels, - LinearOptimizer optimizer, -) { - final factory = OptimizerFactoryMock(); - when(factory.gradient( - points, - labels, - dtype: anyNamed('dtype'), - randomizerFactory: anyNamed('randomizerFactory'), - costFunction: anyNamed('costFunction'), - learningRateGeneratorFactory: anyNamed('learningRateGeneratorFactory'), - initialWeightsGeneratorFactory: - anyNamed('initialWeightsGeneratorFactory'), - learningRateType: anyNamed('learningRateType'), - initialWeightsType: anyNamed('initialWeightsType'), - initialLearningRate: anyNamed('initialLearningRate'), - minCoefficientsUpdate: anyNamed('minCoefficientsUpdate'), - iterationLimit: anyNamed('iterationLimit'), - lambda: anyNamed('lambda'), - batchSize: anyNamed('batchSize'), - randomSeed: anyNamed('randomSeed'), - )).thenReturn(optimizer); - return factory; -}