Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[GSOC]DatasetMapper & Imputer #694

Merged
merged 47 commits into from Jul 25, 2016
Merged

[GSOC]DatasetMapper & Imputer #694

Changes from 1 commit
Commits
Show all changes
47 commits
Select commit Hold shift + click to select a range
87c05a5
concept work for imputer
keon Jun 1, 2016
2e4b1a8
Merge branch 'master' of github.com:keonkim/mlpack into imputer
keon Jun 6, 2016
631e59e
do not to use NaN by default, let the user specify
keon Jun 6, 2016
391006e
Merge branch 'master' of github.com:keonkim/mlpack into imputer
keon Jun 6, 2016
6a1fb81
add template to datasetinfo and add imputer class
keon Jun 12, 2016
b0c5224
clean datasetinfo class and rename files
keon Jun 13, 2016
de35241
implement basic imputation strategies
keon Jun 13, 2016
2d38604
modify imputer_main and clean logs
keon Jun 13, 2016
bb045b8
add parameter verification for imputer_main
keon Jun 13, 2016
1295f4b
add custom strategy to impute_main
keon Jun 13, 2016
5a517c2
add datatype change in IncrementPolicy
keon Jun 14, 2016
94b7a5c
update types used in datasetinfo
keon Jun 14, 2016
ebed68f
initialize imputer with parameters
keon Jun 14, 2016
db78f39
remove datatype in dataset_info
keon Jun 15, 2016
7c60b97
Merge branch 'master' of github.com:keonkim/mlpack into imputer
keon Jun 15, 2016
da4e409
add test for imputer
keon Jun 15, 2016
d8618ec
restructure, add listwise deletion & imputer tests
keon Jun 18, 2016
3b8ffd0
fix transpose problem
keon Jun 27, 2016
90a5cd2
Merge pull request #7 from mlpack/master
keon Jun 27, 2016
32c8a73
merge
keon Jun 27, 2016
e09d9bc
updates and fixes on imputation methods
keon Jun 28, 2016
87d8d46
update data::load to accept different mappertypes
keon Jul 1, 2016
de0b2db
update data::load to accept different policies
keon Jul 1, 2016
bc187ca
add imputer doc
keon Jul 1, 2016
a340f69
debug median imputation and listwise deletion
keon Jul 2, 2016
21d94c0
remove duplicate code in load function
keon Jul 2, 2016
a92afaa
delete load overload
keon Jul 3, 2016
bace8b2
modify MapToNumerical to work with MissingPolicy
keon Jul 4, 2016
896a018
MissingPolicy uses NaN instead of numbers
keon Jul 4, 2016
1a908c2
fix reference issue in DatasetMapper
keon Jul 4, 2016
2edbc40
Move MapToNumerical(MapTokens) to Policy class
keon Jul 5, 2016
d881cb7
make policy and imputation api more consistent
keon Jul 5, 2016
a881831
numerical values can be set as missing values
keon Jul 6, 2016
63268a3
add comments and use more proper names
keon Jul 7, 2016
2eb6754
modify custom impute interface and rename variables
keon Jul 10, 2016
6d43aa3
add input-only overloads to imputation methods
keon Jul 10, 2016
fedc5e0
update median imputation to exclude missing values
keon Jul 11, 2016
787fd82
optimize imputation methods with output overloads
keon Jul 18, 2016
a0b7d59
expressive comments in imputation_test
keon Jul 18, 2016
9a6dce7
shorten imputation tests
keon Jul 18, 2016
c3aeba1
optimize preprocess imputer executable
keon Jul 18, 2016
028c217
fix bugs in imputation test
keon Jul 18, 2016
03e19a4
add more comments and delete impute_test.csv
keon Jul 22, 2016
ef4536b
Merge pull request #8 from mlpack/master
keon Jul 22, 2016
6e2c1ff
Merge branch 'master' of github.com:keonkim/mlpack into imputer
keon Jul 22, 2016
5eb9abd
fix PARAM statements in imputer
keon Jul 22, 2016
d043235
delete Impute() overloads that produce output matrix
keon Jul 23, 2016
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Jump to
Jump to file
Failed to load files.
Diff view
Diff view
63 changes: 17 additions & 46 deletions src/mlpack/tests/imputation_test.cpp
View file
Open in desktop
Expand Up @@ -24,6 +24,19 @@ using namespace mlpack::data;
using namespace std;

BOOST_AUTO_TEST_SUITE(ImputationTest);
/**
* Check if two matrixes are equal.
*/
void CheckEqual(const arma::mat& lhs, const arma::mat& rhs)
{
BOOST_REQUIRE(lhs.n_rows == rhs.n_rows);
BOOST_REQUIRE(lhs.n_cols == rhs.n_cols);
for(size_t i = 0; i != lhs.n_elem; ++i)
{
BOOST_REQUIRE_CLOSE(lhs[i], rhs[i], 1e-5);
}
}

/**
* 1. Make sure a CSV is loaded correctly with mappings using MissingPolicy.
* 2. Try Imputer object with CustomImputation method to impute data "a".
Expand Down Expand Up @@ -136,19 +149,7 @@ BOOST_AUTO_TEST_CASE(CustomImputationTest)

// overwrite to the input
imputer.Impute(input, mappedValue, 0/*dimension*/, true);
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think we can reduce some duplicate codes

void CheckEqual(arma::mat const &lhs, arma::mat const &rhs)
{
  BOOST_REQUIRE(lhs.n_rows == rhs.n_rows);
  BOOST_REQUIRE(lhs.n_cols == rhs.n_cols);
  for(size_t i = 0; i != lhs.n_elem; ++i)
  {
    BOOST_REQUIRE_CLOSE(lhs[i], rhs[i], 1e-5);
  }
}

//.......

// overwrite to the input
imputer.Impute(input, mappedValue, 0/*dimension*/, columnMajor);
CheckEqual(input, output);


BOOST_REQUIRE_CLOSE(input(0, 0), 3.0, 1e-5);
BOOST_REQUIRE_CLOSE(input(0, 1), 99.0, 1e-5);
BOOST_REQUIRE_CLOSE(input(0, 2), 2.0, 1e-5);
BOOST_REQUIRE_CLOSE(input(0, 3), 99.0, 1e-5);
BOOST_REQUIRE_CLOSE(input(1, 0), 5.0, 1e-5);
BOOST_REQUIRE_CLOSE(input(1, 1), 6.0, 1e-5);
BOOST_REQUIRE_CLOSE(input(1, 2), 0.0, 1e-5);
BOOST_REQUIRE_CLOSE(input(1, 3), 6.0, 1e-5);
BOOST_REQUIRE_CLOSE(input(2, 0), 9.0, 1e-5);
BOOST_REQUIRE_CLOSE(input(2, 1), 8.0, 1e-5);
BOOST_REQUIRE_CLOSE(input(2, 2), 4.0, 1e-5);
BOOST_REQUIRE_CLOSE(input(2, 3), 8.0, 1e-5);
CheckEqual(input, output);
}

/**
Expand Down Expand Up @@ -200,19 +201,7 @@ BOOST_AUTO_TEST_CASE(MeanImputationTest)

// overwrite to the input
imputer.Impute(input, mappedValue, 0/*dimension*/, true);

BOOST_REQUIRE_CLOSE(input(0, 0), 3.0, 1e-5);
BOOST_REQUIRE_CLOSE(input(0, 1), 2.5, 1e-5);
BOOST_REQUIRE_CLOSE(input(0, 2), 2.0, 1e-5);
BOOST_REQUIRE_CLOSE(input(0, 3), 2.5, 1e-5);
BOOST_REQUIRE_CLOSE(input(1, 0), 5.0, 1e-5);
BOOST_REQUIRE_CLOSE(input(1, 1), 6.0, 1e-5);
BOOST_REQUIRE_CLOSE(input(1, 2), 0.0, 1e-5);
BOOST_REQUIRE_CLOSE(input(1, 3), 6.0, 1e-5);
BOOST_REQUIRE_CLOSE(input(2, 0), 9.0, 1e-5);
BOOST_REQUIRE_CLOSE(input(2, 1), 8.0, 1e-5);
BOOST_REQUIRE_CLOSE(input(2, 2), 4.0, 1e-5);
BOOST_REQUIRE_CLOSE(input(2, 3), 8.0, 1e-5);
CheckEqual(input, output);
}

/**
Expand Down Expand Up @@ -263,19 +252,7 @@ BOOST_AUTO_TEST_CASE(MedianImputationTest)

// overwrite to the input
imputer.Impute(input, mappedValue, 1/*dimension*/, true);

BOOST_REQUIRE_CLOSE(input(0, 0), 3.0, 1e-5);
BOOST_REQUIRE_CLOSE(input(0, 1), 0.0, 1e-5);
BOOST_REQUIRE_CLOSE(input(0, 2), 2.0, 1e-5);
BOOST_REQUIRE_CLOSE(input(0, 3), 0.0, 1e-5);
BOOST_REQUIRE_CLOSE(input(1, 0), 5.0, 1e-5);
BOOST_REQUIRE_CLOSE(input(1, 1), 6.0, 1e-5);
BOOST_REQUIRE_CLOSE(input(1, 2), 6.0, 1e-5);
BOOST_REQUIRE_CLOSE(input(1, 3), 6.0, 1e-5);
BOOST_REQUIRE_CLOSE(input(2, 0), 9.0, 1e-5);
BOOST_REQUIRE_CLOSE(input(2, 1), 8.0, 1e-5);
BOOST_REQUIRE_CLOSE(input(2, 2), 4.0, 1e-5);
BOOST_REQUIRE_CLOSE(input(2, 3), 8.0, 1e-5);
CheckEqual(input, output);
}

/**
Expand Down Expand Up @@ -317,13 +294,7 @@ BOOST_AUTO_TEST_CASE(ListwiseDeletionTest)

// overwrite to the input
imputer.Impute(input, mappedValue, 0, true); // column wise

BOOST_REQUIRE_CLOSE(input(0, 0), 3.0, 1e-5);
BOOST_REQUIRE_CLOSE(input(0, 1), 2.0, 1e-5);
BOOST_REQUIRE_CLOSE(input(1, 0), 5.0, 1e-5);
BOOST_REQUIRE_CLOSE(input(1, 1), 0.0, 1e-5);
BOOST_REQUIRE_CLOSE(input(2, 0), 9.0, 1e-5);
BOOST_REQUIRE_CLOSE(input(2, 1), 4.0, 1e-5);
CheckEqual(input, output);
}

BOOST_AUTO_TEST_SUITE_END();