This project is intended to implement Deep NN / RNN based solution in order to develop flexible methods that are able to adaptively fillin, backfill, and predict time-series using a large number of heterogeneous training datasets.
The perfect solution must at least exceed performance of plain vanila Random Forest Regressor which considered as scoring baseline.
The goal of this project is to develop flexible methods that are able to adaptively fillin, backfill, and predict time-series using a large number of heterogeneous training datasets. The data is a set of thousands of aggressively obfuscated, multi-variate timeseries measurements. There are multiple output variables and multiple input variables.
For each time-series, there are parts missing. Either individual measurements, or entire sections. Each time-series has a different number of known measurements and missing measurements, the goal is to fill in the missing output variables with the best accuracy possible. How the missing input variables are treated is an open question, and is one of the key challenges to solve.
This problem, unlike many data science contest problems, is not easy to fit into the standard machine learning framework. Some reasons that this is the case:
- There are multiple time-series outputs.
- There are multiple timeseries inputs.
- The time-series are sampled irregularly, and in different time points for each subject.
- There is a huge amount of missing data, and it is not missing at random.
- Many of the variables are nominal/categorical, and some of these are very high cardinality. The most important variable, subject id, is the primary example. A good solution should not ignore the subject id.
The score for each individual prediction p, compared against actual ground-truth value t, will be |p - t|. The score for each row, r, will then be the mean of the scores for the individual predictions on that row (possibly 1, 2, or 3 values). Over the full n rows, your final score will be calculated as 10 * (1 - Sum(r) / n). Thus a score of 10.00 represents perfect predictions with no error at all.
In order to fulfill requested task was implemented two solutions based on Recurent Neural Network and Deep Learning Neural Network architectures. It was compared performance of both against plain vanila implementation based Random Forest Regressor.
The Deep NN was found as superior to RNN for this task, but with not too big difference. But, unfortunatelly, both still lag behind Random Forest Regressor.
Scores per method:
- Deep NN: 9.861
- RNN: 9.830
- Random Forest Regressor: 9.880 (baseline)
0.026730638156987854 0.007701583490203154 0.03510046831242789
count 238897.000000 238897.000000 238897.000000
mean 0.072598 0.282652 0.195517
std 0.067135 0.157973 0.249721
min 0.000000 0.000000 0.000000
25% 0.030325 0.171160 0.001698
50% 0.053854 0.246471 0.005034
75% 0.086798 0.346608 0.400094
max 0.465094 0.891380 0.894128
60, 100, 0.05 -> 1e-8, Adagrad, RNN
epoch 59, train loss: [ 0.04069507], score: [ 9.84009604]
Validate score: 9.8366469
Test score: 9.82 (vp_30_07_19_52.csv)
________________________________________________________________________
0.0378213827224494 0.0 0.0.1
count 238897.000000 238897.000000 238897.000000
mean 0.071949 0.279996 0.193371
std 0.067764 0.156745 0.249946
min 0.000000 0.000000 0.000000
25% 0.030075 0.169914 0.000000
50% 0.053832 0.243377 0.002575
75% 0.086068 0.345924 0.397201
max 0.503966 0.925782 0.830723
180, 100, 0.05 -> 1e-8, Adagrad, RNN
epoch 179, train loss: [ 0.04046797], score: [ 9.84038537]
Validate score: 9.8366469
Test score: 9.82 (vp_31_07_00_21.csv)
------------------------
Predictions:
yvl1_est yvl2_est yvl3_est
count 238898.000000 238898.000000 238898.000000
mean 0.071048 0.278478 0.190451
std 0.068349 0.157351 0.247715
min 0.000000 0.000000 0.000000
25% 0.028766 0.166988 0.000000
50% 0.053941 0.245374 0.000350
75% 0.084177 0.340235 0.389137
max 0.501959 0.881149 0.793660
101, 100, 5e-4 -> adam, tanh, shuffle, RNN
epoch 100, train loss: [ 0.03829205], score: [ 9.88033067]
Validate score:
Test score: 9.83 (vp_02_08_23_13.csv)
------------------------
Predictions:
yvl1_est yvl2_est yvl3_est
count 238898.000000 238898.000000 238898.000000
mean 0.070911 0.278394 0.188772
std 0.066999 0.158090 0.247499
min 0.000000 0.000000 0.000000
25% 0.029590 0.166484 0.000000
50% 0.053094 0.246164 0.001593
75% 0.082912 0.340488 0.407425
max 0.507457 1.000000 0.828317
81, 100, 1e-4 -> adam 0.9/0.99, shuffle, reg1e-3, preprocessing, DeepNN[50, 20]
epoch 80, train loss: [ 0.03659411], score: [ 9.88233213]
Test score: 9.85 (vp_03_08_16_27.csv)
------------------------
Predictions:
yvl1_est yvl2_est yvl3_est
count 238898.000000 238898.000000 238898.000000
mean 0.071074 0.278243 0.188506
std 0.068041 0.157917 0.247308
min 0.000000 0.000000 0.000000
25% 0.029369 0.166125 0.000000
50% 0.053656 0.246783 0.001329
75% 0.082850 0.337197 0.406893
max 0.515826 0.976951 0.892692
80, 100, 5e-5 -> adam bias 0.9/0.99, shuffle, reg1e-3, preprocessing, DeepNN[60, 30]
epoch: 79, train loss: [ 0.03626305], score: [ 9.88334681], learning rate: 5e-07
Test score: 9.85 (vp_04_08_16_40.csv)
------------------------
Predictions:
yvl1_est yvl2_est yvl3_est
count 238898.000000 238898.000000 238898.000000
mean 0.070833 0.278369 0.188905
std 0.068485 0.157788 0.246909
min 0.000000 0.000000 0.000000
25% 0.028804 0.166068 0.000000
50% 0.053427 0.246008 0.003569
75% 0.083206 0.340325 0.406023
max 0.525686 0.954895 0.810934
60, 100, 5e-5, adam, reg1e-4, preprocessing DeepNN[256, 128]
epoch: 59, train loss: [ 0.03710156], score: [ 9.88219018], learning rate: 5e-07
Test score: 98.53 (vp_06_08_22_49.csv)
------------------------
Predictions:
yvl1_est yvl2_est yvl3_est
count 238898.000000 238898.000000 238898.000000
mean 0.070915 0.278305 0.189096
std 0.068387 0.157991 0.247285
min 0.000000 0.000000 0.000000
25% 0.029102 0.166380 0.000000
50% 0.053139 0.245979 0.003406
75% 0.083086 0.339003 0.406160
max 0.534707 0.998148 0.841650
180, 100, 5e-5, adam, reg1e-4, preprocessing DeepNN[256, 128]
epoch: 179, train loss: [ 0.03601578], score: [ 9.88568121], learning rate: 5e-07
Test score: 98.56 (vp_07_08_21_22.csv)
------------------------
Predictions:
count 238897.000000 238897.000000 238897.000000
mean 0.071045 0.277557 0.188097
std 0.068731 0.158530 0.247959
min 0.000000 0.000000 0.000000
25% 0.029443 0.165544 0.000000
50% 0.053455 0.246995 0.000000
75% 0.082807 0.338127 0.407639
max 0.658165 0.965393 0.885541
60, 100, 5e-2, Adagrad, reg1e-4, features selected, DeepNN[128, 32]
Test score: 98.61 (vp_10_08_11_45.csv)
------------------------
Predictions:
yvl1_est yvl2_est yvl3_est
count 238898.000000 238898.000000 238898.000000
mean 0.072625 0.284743 0.499098
std 0.071510 0.152186 0.184594
min 0.001185 0.014566 0.001994
25% 0.029601 0.170947 0.352773
50% 0.053907 0.249249 0.526542
75% 0.084633 0.343988 0.653956
max 0.830919 0.943561 0.883551
validation baseline - Random Forest Regressor
Test score: 98.80 (vp_tree_10_08_2016.csv)
- 'data' directory contains training / testing data samples
- 'src' directory has source files
The main runners are 'src/deep_learning_runner.py' and 'src/vanila_rnn.py' for starting 'Deep NN' and 'RNN' correspondingly. The 'src/score_validator.py' may be used to calculate score over test data saples run results.
The 'src/utils/train_validate_splitter.py' can be used in order to generate train/validate data samples for training from 'data/trainng.csv' file
The training and test data contains several columns:
----------+--------------------+------------+-------------------------------------------------------
Column#s | Column Name(s) | Data Type | Description
----------+--------------------+------------+-------------------------------------------------------
1-3 | y1, y2, y2 | Float | The three dependent variables to be predicted in test
----------+--------------------+------------+-------------------------------------------------------
4 | STUDYID | Integer |
----------+--------------------+------------+-------------------------------------------------------
5 | SITEID | Integer |
----------+--------------------+------------+-------------------------------------------------------
6 | COUNTRY | Integer |
----------+--------------------+------------+-------------------------------------------------------
7 | SUBJID | Integer |
----------+--------------------+------------+-------------------------------------------------------
8 | TIMEVAR1 | Float |
----------+--------------------+------------+-------------------------------------------------------
9 | TIMEVAR2 | Float |
----------+--------------------+------------+-------------------------------------------------------
10-39 | COVAR_CONTINUOUS_n | Float | (30 fields)
----------+--------------------+------------+-------------------------------------------------------
40-47 | COVAR_ORDINAL_n | Integer | (8 fields)
----------+--------------------+------------+-------------------------------------------------------
48-55 | COVAR_NOMINAL_n | Char | (8 fields)
----------+--------------------+------------+-------------------------------------------------------
56-58 | y1, y2, y3 missing | True/False | (3 fields) does the value exist in ground truth
----------+--------------------+------------+-------------------------------------------------------
The combination of STUDYID and SUBJID is sufficient to uniquely identify a specific individual. Adding TIMEVAR1 is sufficient to identify to uniquely identify each row.
The last three columns contain the values “True” or “False” indicate whether y1, y2, or y3 is missing from the ground truth data.
- Stanford CS class CS231n
- UFLDL Deep Learning Tutorial
- The Unreasonable Effectiveness of Recurrent Neural Networks
- Recurrent Neural Networks
- Generating Sequences With Recurrent Neural Networks arXiv:1308.0850
- Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modeling arXiv:1412.3555
- Adam: A Method for Stochastic Optimization arXiv:1412.6980
- Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification arXiv:1502.01852
- Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift arXiv:1502.03167
- RMSProp and equilibrated adaptive learning rates for non-convex optimization arXiv:1502.04390
- DRAW: A Recurrent Neural Network For Image Generation arXiv:1502.04623
- Directly Modeling Missing Data in Sequences with RNNs: Improved Classification of Clinical Time Series arXiv:1606.04130