Project including data pre-processing algo. We aim to include scaling, centering, normalization, binarization methods.
To install the project, go to the Playground (Ctrl+OW) in your Pharo image and execute the following Metacello script (select it and press Do-it button or Ctrl+D):
Metacello new
baseline: 'AIDataPreProcessing';
repository: 'github://pharo-ai/data-preprocessing/src';
load.If you want to add a dependency on this project to your project, include the following lines into your baseline method:
spec
baseline: 'AIDataPreProcessing'
with: [ spec repository: 'github://pharo-ai/data-preprocessing/src' ].If you are new to baselines and Metacello, check out this wonderful Baselines tutorial on Pharo Wiki.
It is possible to encode a 2D collection with numerical categories easily like this:
| collection |
collection := #( #( 'Female' 3 ) #( 'Male' 1 ) #( 'Female' 2 ) ).
AIOrdinalEncoder new
fitAndTransform: collection. "#(#(2 3) #(1 1) #(2 2))"For more details check the documentation bellow.
It is possible to normalize your collections like this:
normalizer := AIStandardizationNormalizer new.
numbers := #(10 -3 4 2 -7 1000 0.1 -4.05).
nomalizer normalize: numbers. "#(-0.3261 -0.3628 -0.343 -0.3487 -0.3741 2.475 -0.3541 -0.3658)"For more details check the documentation bellow.
AIOrdinalEncoder is a simple encoder whose responsibility is to associate a number to each unique value of a 2D collection. (Can be a DataFrame)
I can be fitted with a collection to compute the categories to use and then transform another collection (possibily the same one). All values of the collection to transform must be present in the collection used to fit the datas or an AIMissingCategory exception will be raised.
I can be use like this:
| collection |
collection := #( #( 'Female' 3 ) #( 'Male' 1 ) #( 'Female' 2 ) ).
AIOrdinalEncoder new
fit: collection;
transform: collection. "#(#(2 3) #(1 1) #(2 2))"Or
| collection |
collection := #( #( 'Female' 3 ) #( 'Male' 1 ) #( 'Female' 2 ) ).
AIOrdinalEncoder new
fitAndTransform: collection. "#(#(2 3) #(1 1) #(2 2))"I can also be used on a DataFrame in the same way:
| collection |
collection := DataFrame withRows: #( #( 'Female' 3 ) #( 'Male' 1 ) #( 'Female' 2 ) ).
AIOrdinalEncoder new
fitAndTransform: collection. "#(#(2 3) #(1 1) #(2 2))"The user can also give directly the categories to use like this:
| collection |
collection := #( #( 'Female' 3 ) #( 'Male' 1 ) #( 'Female' 2 ) ).
AIOrdinalEncoder new
categories: #( #( 'Female' 'Male' ) #( 3 1 2 ) );
transform: collection. "#(#(1 1) #(2 2) #(1 3))"In that case, the index of each elements will be used as a category.
Normalization is a technique often applied as part of data preparation for machine learning. The goal of normalization is to change the values of numeric columns in the dataset to use a common scale, without distorting differences in the ranges of values or losing information.
For example, consider that you have two collections, ages and salaries:
ages := #(25 19 30 32 41 50 24).
salaries := #(1600 1000 2500 2400 5000 3500 2500).Those collections are on a very different scale. The differences in salaries have larger magnitude than differences in age. Which can confuse some machine learning algorithms and force them to "think" that if the difference salaries is 600 (euros) and the difference in age is 6 (years), then salary difference is 100 times greater than age difference. Such algorithms require data to be normalized - for example, both ages and salaries can be transformed to a scale of [0, 1].
There are different types of normalization. In this repository, we implement two most commonly used strategies: Min-Max Normalization and Standardization. You can easily define your own strategy by adding a subclass of AINormalizer.
Min-Max or Rescaling is the type of normalization, every element of the numeric collection is transformed to a scale of [0, 1]:
x'[i] = (x[i] - x min) / (x max - x min)
Standardization is the type of normalization, every element of the numeric collection is by scaled to be centered around the mean with a unit standard deviation:
x'[i] = (x[i] - x mean) / x std
You can normalize any numeric collection by calling the normalized method on it:
numbers := #(10 -3 4 2 -7 1000 0.1 -4.05).
numbers normalized. "#(0.0169 0.004 0.0109 0.0089 0.0 1.0 0.007 0.0029)"By default, it will use the AIMinMaxNormalizer. If you want to use a different normalization strategy, you can call normalizedUsing: on a collection:
normalizer := AIStandardizationNormalizer new.
numbers normalizedUsing: normalizer. "#(-0.3261 -0.3628 -0.343 -0.3487 -0.3741 2.475 -0.3541 -0.3658)"Or you can also give the collection to normalize to the normalizer:
normalizer := AIStandardizationNormalizer new.
nomalizer normalize: numbers. "#(-0.3261 -0.3628 -0.343 -0.3487 -0.3741 2.475 -0.3541 -0.3658)"For the two normalization strategies that are defined in this package, we provide alias methods:
numbers rescaled.
"is the same as"
numbers normalizedUsing: AIMinMaxNormalizer new.numbers standardized.
"is the same as"
numbers normalizedUsing: AIStandardizationNormalizer new.Each normalizer remembers the parameters of the original collection (e.g., min/max or mean/std) and can use them to restore the normalized collection to its original state:
numbers := #(10 -3 4 2 -7 1000 0.1 -4.05).
normalizer := AIMinMaxNormalizer new.
normalizedNumbers := normalizer normalize: numbers. "#(0.0169 0.004 0.0109 0.0089 0.0 1.0 0.007 0.0029)"
restoredNumbers := normalizer restore: normalizedNumbers. "#(10 -3 4 2 -7 1000 0.1 -4.05)"Normalization is implemented using a strategy design pattern. The AI-Normalization defines an abstract class AINormalizer which has two abstract methods AINormalizer class >> normalize: aCollection and AINormalizer class >> restore: aCollection. To define a normalization strategy, please implement a subclass of AINormalizer and provide your own definitions of normalize: and restore: methods. Keep in mind that those methods must not modify the given collection but return a new one.
To normalize a collection using your own strategy, call:
normalizer := YourCustomNormalizer new.
numbers normalizedUsing: normalizer.