Skip to content
master
Switch branches/tags
Code

ImputeBench: Benchmark of Imputation Techniques in Time Series

ImputeBench implements 15 SOTA recovery techniques for blocks of missing values in time series and evaluates their precision and runtime on various real-world time series datasets using different recovery scenarios. Technical details can be found in our PVLDB 2020 paper: Mind the Gap: An Experimental Evaluation of Imputation of Missing Values Techniques in Time Series . The benchmark can be easity extended with new algorithms (C/C++, Python or Matlab), new datasets and new scenarios.

  • The benchmark implements the following algorithms (in C++): CDRec, DynaMMo, GROUSE, ROSL, SoftImpute, SPIRIT, STMVL, SVDImpute, SVT, TeNMF, TRMF, and TKCM. We recently added these algorithms (in Python): SSA, MRNN and BRITS.
  • All the datasets used in this benchmark can be found here.
  • The full list of recovery scenarios can be found here.
  • Remark: Algorithms tkcm, spirit, ssa, and mr-nn cannot handle multiple incomplete time series. These allgorithms will not produce results for the following scenarios: miss_disj, miss_over, mcar and blackout.

Prerequisites | Build | Execution | Extension | Contributors | Award | Citation


Prerequisites

  • Ubuntu 16 or Ubuntu 18 (including Ubuntu derivatives, e.g., Xubuntu) or the same distribution under WSL.
  • Clone this repository.
  • Mono: Install mono from https://www.mono-project.com/download/stable/ and reboot your terminal.

Build

  • Build the Testing Framework using the installation script located in the root folder (takes several minutes)
    $ sh install_linux.sh
  • To evaluate the recently added algorithms (SSA, MRNN and BRITS), please install the following packages (takes several minutes):
    $ sh install_extra.sh

Execution

    $ cd TestingFramework/bin/Debug/
    $ mono TestingFramework.exe [arguments]

Arguments

-alg -d -scen
cdrec airq miss_perc
dynammo bafu ts_length
grouse chlorine ts_nbr
rosl climate miss_disj
softimp drift10 miss_over
svdimp electricity mcar
svt meteo blackout
stmvl temp all
spirit bafu_red
tenmf drift10_red
tkcm all
trmf
all
-------- -------- --------
New algs
-------- -------- --------
ssa
m-rnn
brits

Results

All results and plots will be added to Results folder. The accuracy results of all algorithms will be sequentially added for each scenario and dataset to: Results/.../.../error/. The runtime results of all algorithms will be added to: Results/.../.../runtime/. The plots of the recovered blocks will be added to the folder Results/.../.../recovery/plots/.

Execution examples

  1. Run a single algorithm (cdrec) on a single dataset (drift10) using one scenario (missing percentage)
    $ mono TestingFramework.exe -alg cdrec -d drift10 -scen miss_perc
  1. Run two algorithms (cdrec, spirit) on a single dataset (drift10) using one scenario (missing percentage)
    $ mono TestingFramework.exe -alg cdrec,spirit -d drift10 -scen miss_perc
  1. Run point 2 without runtime results
    $ mono TestingFramework.exe -alg cdrec,spirit -d drift10 -scen miss_perc -nort
  1. Run the whole VLDB'20 benchmark (all algorithms, all datasets, all scenarios, precision and runtime)
    $ mono TestingFramework.exe -alg all -d all -scen all

Warning: Running the whole benchmark will take a sizeable amount of time (up to 4 days depending on the hardware) and will produce up to 15GB of output files with all recovered data and plots unless stopped early.

  1. Create patterns of missing blocks on one complete dataset (airq) using one scenario (missing percentage)
    $ mono TestingFramework.exe -alg mvexport -d airq -scen miss_perc

Note: You need to run each scenario seperately on one or multiple datasets. Each time you execute one scenario, the Results folder will be overwritten with the new files.

  1. Additional command-line parameters
    $ mono TestingFramework.exe --help

Parametrized execution

  • You can parametrize each algorithm using the command -algx. For example, you can run the svdimp algorithm with a reduction value of 4 on the drift dataset and by varying the sequence length as follows:
    $ mono TestingFramework.exe -algx svdimp 4 -d drift10 -scen ts_nbr
  • If you want to run some algorithms with default parameters, and some with customized ones, you can use -alg and -algx together. For example, you can run stmvl algorithm with default parameter and cdrec algorithm with a reduction value of 4 on the airq dataset and by varying the sequence length as follows:
    $ mono TestingFramework.exe -alg stmvl -algx cdrec 4 -d airq -scen ts_nbr

Remark: The command -algx cannot be executed in group and thus must preceed the name of each algorithm.


Extension


Contributors

Mourad Khayati (mkhayati@exascale.info) and Zakhar Tymchenko (zakhar.tymchenko@unifr.ch).


Award

Imputebench has received the VLDB 2020 Most Reproducible Paper Award.


Citation

@inproceedings{imputebench2020vldb,
 author    = {Mourad Khayati and Alberto Lerner and Zakhar Tymchenko and Philippe Cudr{\'{e}}{-}Mauroux},
 title     = {Mind the Gap: An Experimental Evaluation of Imputation of Missing Values Techniques in Time Series},
 booktitle = {Proceedings of the VLDB Endowment},
 volume    = {13},
 number    = {5},
 year      = {2020}
}