Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[GSOC]DatasetMapper & Imputer #694

Merged
merged 47 commits into from Jul 25, 2016
Merged
Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
47 commits
Select commit Hold shift + click to select a range
87c05a5
concept work for imputer
keon Jun 1, 2016
2e4b1a8
Merge branch 'master' of github.com:keonkim/mlpack into imputer
keon Jun 6, 2016
631e59e
do not to use NaN by default, let the user specify
keon Jun 6, 2016
391006e
Merge branch 'master' of github.com:keonkim/mlpack into imputer
keon Jun 6, 2016
6a1fb81
add template to datasetinfo and add imputer class
keon Jun 12, 2016
b0c5224
clean datasetinfo class and rename files
keon Jun 13, 2016
de35241
implement basic imputation strategies
keon Jun 13, 2016
2d38604
modify imputer_main and clean logs
keon Jun 13, 2016
bb045b8
add parameter verification for imputer_main
keon Jun 13, 2016
1295f4b
add custom strategy to impute_main
keon Jun 13, 2016
5a517c2
add datatype change in IncrementPolicy
keon Jun 14, 2016
94b7a5c
update types used in datasetinfo
keon Jun 14, 2016
ebed68f
initialize imputer with parameters
keon Jun 14, 2016
db78f39
remove datatype in dataset_info
keon Jun 15, 2016
7c60b97
Merge branch 'master' of github.com:keonkim/mlpack into imputer
keon Jun 15, 2016
da4e409
add test for imputer
keon Jun 15, 2016
d8618ec
restructure, add listwise deletion & imputer tests
keon Jun 18, 2016
3b8ffd0
fix transpose problem
keon Jun 27, 2016
90a5cd2
Merge pull request #7 from mlpack/master
keon Jun 27, 2016
32c8a73
merge
keon Jun 27, 2016
e09d9bc
updates and fixes on imputation methods
keon Jun 28, 2016
87d8d46
update data::load to accept different mappertypes
keon Jul 1, 2016
de0b2db
update data::load to accept different policies
keon Jul 1, 2016
bc187ca
add imputer doc
keon Jul 1, 2016
a340f69
debug median imputation and listwise deletion
keon Jul 2, 2016
21d94c0
remove duplicate code in load function
keon Jul 2, 2016
a92afaa
delete load overload
keon Jul 3, 2016
bace8b2
modify MapToNumerical to work with MissingPolicy
keon Jul 4, 2016
896a018
MissingPolicy uses NaN instead of numbers
keon Jul 4, 2016
1a908c2
fix reference issue in DatasetMapper
keon Jul 4, 2016
2edbc40
Move MapToNumerical(MapTokens) to Policy class
keon Jul 5, 2016
d881cb7
make policy and imputation api more consistent
keon Jul 5, 2016
a881831
numerical values can be set as missing values
keon Jul 6, 2016
63268a3
add comments and use more proper names
keon Jul 7, 2016
2eb6754
modify custom impute interface and rename variables
keon Jul 10, 2016
6d43aa3
add input-only overloads to imputation methods
keon Jul 10, 2016
fedc5e0
update median imputation to exclude missing values
keon Jul 11, 2016
787fd82
optimize imputation methods with output overloads
keon Jul 18, 2016
a0b7d59
expressive comments in imputation_test
keon Jul 18, 2016
9a6dce7
shorten imputation tests
keon Jul 18, 2016
c3aeba1
optimize preprocess imputer executable
keon Jul 18, 2016
028c217
fix bugs in imputation test
keon Jul 18, 2016
03e19a4
add more comments and delete impute_test.csv
keon Jul 22, 2016
ef4536b
Merge pull request #8 from mlpack/master
keon Jul 22, 2016
6e2c1ff
Merge branch 'master' of github.com:keonkim/mlpack into imputer
keon Jul 22, 2016
5eb9abd
fix PARAM statements in imputer
keon Jul 22, 2016
d043235
delete Impute() overloads that produce output matrix
keon Jul 23, 2016
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Jump to
Jump to file
Failed to load files.
Diff view
Diff view
6 changes: 3 additions & 3 deletions src/mlpack/core/data/dataset_info.hpp
Expand Up @@ -59,7 +59,7 @@ class DatasetMapper
* @param string String to find/create mapping for.
* @param dimension Index of the dimension of the string.
*/
typename MapPolicy::map_type_t MapString(const std::string& string,
typename MapPolicy::mapped_type MapString(const std::string& string,
const size_t dimension);
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think we can make map_type_t as mapped_type, mapped_type is same as the typedef of std::map, mapped_type should be more familiar with c++ programmer(my example did not name it well, sorry about that)

Copy link
Member Author

@keon keon Jun 14, 2016

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

no problem, it is now updated :)


/**
Expand All @@ -81,7 +81,7 @@ class DatasetMapper
* @param string Mapped string for value.
* @param dimension Dimension to unmap string from.
*/
typename MapPolicy::map_type_t UnmapValue(const std::string& string,
typename MapPolicy::mapped_type UnmapValue(const std::string& string,
const size_t dimension);

//! Return the type of a given dimension (numeric or categorical).
Expand Down Expand Up @@ -118,7 +118,7 @@ class DatasetMapper
std::vector<Datatype> types;

// BiMapType definition
using BiMapType = boost::bimap<std::string, typename MapPolicy::map_type_t>;
using BiMapType = boost::bimap<std::string, typename MapPolicy::mapped_type>;

// Mappings from strings to integers.
// Map entries will only exist for dimensions that are categorical.
Expand Down
4 changes: 2 additions & 2 deletions src/mlpack/core/data/dataset_info_impl.hpp
Expand Up @@ -25,7 +25,7 @@ inline DatasetMapper<MapPolicy>::DatasetMapper(const size_t dimensionality) :
// When we want to insert value into the map,
// we could use the policy to map the string
template<typename MapPolicy>
inline typename MapPolicy::map_type_t DatasetMapper<MapPolicy>::MapString(
inline typename MapPolicy::mapped_type DatasetMapper<MapPolicy>::MapString(
const std::string& string,
const size_t dimension)
{
Expand All @@ -52,7 +52,7 @@ inline const std::string& DatasetMapper<MapPolicy>::UnmapString(

// Return the value corresponding to a string in a given dimension.
template<typename MapPolicy>
inline typename MapPolicy::map_type_t DatasetMapper<MapPolicy>::UnmapValue(
inline typename MapPolicy::mapped_type DatasetMapper<MapPolicy>::UnmapValue(
const std::string& string,
const size_t dimension)
{
Expand Down
4 changes: 2 additions & 2 deletions src/mlpack/core/data/map_policies/CMakeLists.txt
@@ -1,8 +1,8 @@
# Define the files we need to compile
# Anything not in this list will not be compiled into mlpack.
set(SOURCES
increment__policy.hpp
missing__policy.hpp
increment_policy.hpp
missing_policy.hpp
)

# Add directory name to sources.
Expand Down
30 changes: 30 additions & 0 deletions src/mlpack/core/data/map_policies/datatype.hpp
@@ -0,0 +1,30 @@
/**
* @file missing_policy.hpp
* @author Keon Kim
*
*/
#ifndef MLPACK_CORE_DATA_MAP_POLICIES_DATATYPE_HPP
#define MLPACK_CORE_DATA_MAP_POLICIES_DATATYPE_HPP

#include <mlpack/core.hpp>

namespace mlpack {
namespace data {

/**
* The Datatype enum specifies the types of data mlpack algorithms can use.
* The vast majority of mlpack algorithms can only use numeric data (i.e.
* float/double/etc.), but some algorithms can use categorical data, specified
* via this Datatype enum and the DatasetMapper class.
*/
enum Datatype : bool /* [> bool is all the precision we need for two types <] */
{
numeric = 0,
categorical = 1
};

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is very pedantic but there is an extra line here. :)

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Updated


} // namespace data
} // namespace mlpack

#endif
13 changes: 3 additions & 10 deletions src/mlpack/core/data/map_policies/increment_policy.hpp
Expand Up @@ -10,31 +10,24 @@
#include <mlpack/core.hpp>
#include <unordered_map>
#include <boost/bimap.hpp>

#include <mlpack/core/data/map_policies/datatype.hpp>

using namespace std;

namespace mlpack {
namespace data {

enum Datatype : bool /* bool is all the precision we need for two types */
{
numeric = 0,
categorical = 1
};


/**
* This class is used to map strings to incrementing unsigned integers (size_t).
* First string to be mapped will be mapped to 0, next to 1 and so on.
*/
class IncrementPolicy
{
public:
typedef size_t map_type_t;
typedef size_t mapped_type;

template <typename MapType>
map_type_t MapString(MapType& maps,
mapped_type MapString(MapType& maps,
std::vector<Datatype>& types,
const std::string& string,
const size_t dimension)
Expand Down
13 changes: 3 additions & 10 deletions src/mlpack/core/data/map_policies/missing_policy.hpp
Expand Up @@ -10,31 +10,24 @@
#include <mlpack/core.hpp>
#include <unordered_map>
#include <boost/bimap.hpp>
#include <mlpack/core/data/map_policies/datatype.hpp>


using namespace std;

namespace mlpack {
namespace data {

// TODO: move this to somewhere else so that it can be reused.
enum Datatype : bool /* bool is all the precision we need for two types */
{
numeric = 0,
categorical = 1
};


/**
* Same as increment map policy so far.
*/
class MissingPolicy
{
public:
typedef size_t map_type_t;
typedef size_t mapped_type;

template <typename MapType>
map_type_t MapString(MapType& maps,
mapped_type MapString(MapType& maps,
std::vector<Datatype>& types,
const std::string& string,
const size_t dimension)
Expand Down