DE-Forest - Differential Evolution Forest

This repository provides the source code, algorithms, experimental setup, and results for the article.

Repository content

Datasets

Imbalanced datasets with binary classes are from KEEL repository. The table shows all datasets with their name, Imabalance Ratio, number of examples and number of attributes in descending order by IR.

ID	Dataset	IR	Ex.	Attr.
1	glass1	1.82	214	9
2	vehicle3	1.92	846	18
3	vehicle1	1.97	846	18
4	haberman	2.78	306	3
5	yeast3	8.10	1484	8
6	page-blocks0	8.79	5472	10
7	yeast-2_vs_4	9.08	514	8
8	ecoli-0-6-7_vs_3-5	9.09	222	7
9	glass-0-1-5_vs_2	9.12	172	9
10	yeast-0-3-5-9_vs_7-8	9.12	506	8
11	yeast-0-2-5-6_vs_3-7-8-9	9.14	1004	8
12	ecoli-0-1_vs_2-3-5	9.17	244	7
13	ecoli-0-2-6-7_vs_3-5	9.18	224	7
14	ecoli-0-6-7_vs_5	10.00	220	6
15	glass-0-1-6_vs_2	10.29	192	9
16	ecoli-0-1-4-7_vs_2-3-5-6	10.59	336	7
17	glass-0-1-4-6_vs_2	11.06	205	9
18	cleveland-0_vs_4	12.31	173	13
19	page-blocks-1-3_vs_4	15.86	472	10
20	abalone9-18	16.40	731	8
21	glass-0-1-6_vs_5	19.44	184	9
22	yeast-2_vs_8	23.10	482	8
23	flare-F	23.79	1066	11
24	yeast4	28.10	1484	8
25	winequality-red-4	29.17	1599	11
26	poker-9_vs_7	29.50	244	10
27	winequality-white-9_vs_4	32.60	168	11
28	abalone-17_vs_7-8-9-10	39.31	2338	8
29	abalone-21_vs_8	40.50	581	8
30	yeast6	41.40	1484	8
31	abalone-19_vs_10-11-12-13	49.69	1622	8
32	kr-vs-k-zero_vs_eight	53.07	1460	6
33	ecoli1	63.75	336	7
34	winequality-red-3_vs_5	68.10	691	11
35	ecoli2	70.00	336	7
36	abalone-20_vs_8-9-10	72.69	1916	8
37	kddcup-buffer_overflow_vs_back	73.43	2233	41
38	poker-8_vs_6	85.88	1477	10
39	kddcup-rootkit-imap_vs_back	100.14	2225	41
40	abalone19	129.44	4174	8

Methods

• DE-Forest - the proposed method, ensemble classfier
• RandomFS - native implementation of Random Forest
• RandomFS_b - native implementation of Random Forest with bootstrapping
• optimization - Differential Evolution optimization used in DE-Forest
• bootstrap_optimization - Differential Evolution optimization with bootstrapping used in DE-Forest
• bootstrap_optimization_with_constraints - Differential Evolution optimization with bootstrapping and with constraints used in DE-Forest

Results

Results of experiments in form of raw results, tables of average metrics, Wilcoxon ranking.

SMAC

smac_outputs consists the output from the SMAC3 algorithm. It contains:

• the configuration space - hyperparameters of the method to be optimized
• the run history - successive optimization iterations along with hyperparameter values
• the scenario - parameters of the optimization algorithm and selected instances, i.e. datasets
• the statistic - final optimization statistics and selected hyperparameter values

Tabular results

All tabular results are sorted by Imbalance Ratio of datasets.

Balanced Accuracy

ID	DE-Forest	RandomFS	RandomFS_b	DT
1	0.740 ± 0.033	0.768 ± 0.052	0.748 ± 0.049	0.712 ± 0.032
2	0.657 ± 0.016	0.636 ± 0.019	0.634 ± 0.020	0.623 ± 0.037
3	0.656 ± 0.015	0.621 ± 0.019	0.624 ± 0.018	0.623 ± 0.037
4	0.537 ± 0.031	0.509 ± 0.023	0.509 ± 0.014	0.562 ± 0.041
5	0.759 ± 0.048	0.501 ± 0.003	0.502 ± 0.004	0.836 ± 0.028
6	0.902 ± 0.014	0.886 ± 0.018	0.878 ± 0.018	0.903 ± 0.016
7	0.801 ± 0.042	0.582 ± 0.053	0.627 ± 0.091	0.828 ± 0.031
8	0.749 ± 0.076	0.569 ± 0.066	0.554 ± 0.058	0.823 ± 0.066
9	0.517 ± 0.026	0.521 ± 0.041	0.497 ± 0.003	0.578 ± 0.064
10	0.560 ± 0.034	0.504 ± 0.008	0.504 ± 0.008	0.618 ± 0.045
11	0.680 ± 0.028	0.543 ± 0.025	0.525 ± 0.018	0.719 ± 0.030
12	0.748 ± 0.070	0.683 ± 0.054	0.670 ± 0.062	0.809 ± 0.061
13	0.686 ± 0.095	0.562 ± 0.076	0.576 ± 0.064	0.796 ± 0.067
14	0.784 ± 0.092	0.581 ± 0.068	0.572 ± 0.063	0.833 ± 0.067
15	0.504 ± 0.018	0.517 ± 0.030	0.517 ± 0.027	0.564 ± 0.051
16	0.751 ± 0.041	0.589 ± 0.046	0.595 ± 0.042	0.831 ± 0.053
17	0.503 ± 0.019	0.518 ± 0.039	0.511 ± 0.024	0.537 ± 0.065
18	0.620 ± 0.092	0.530 ± 0.038	0.531 ± 0.053	0.647 ± 0.064
19	0.938 ± 0.039	0.913 ± 0.038	0.902 ± 0.045	0.938 ± 0.069
20	0.560 ± 0.025	0.512 ± 0.017	0.520 ± 0.020	0.630 ± 0.052
21	0.676 ± 0.153	0.750 ± 0.062	0.591 ± 0.073	0.842 ± 0.118
22	0.673 ± 0.124	0.505 ± 0.015	0.500 ± 0.000	0.731 ± 0.088
23	0.547 ± 0.042	0.500 ± 0.000	0.500 ± 0.000	0.570 ± 0.032
24	0.546 ± 0.029	0.500 ± 0.000	0.500 ± 0.000	0.619 ± 0.046
25	0.505 ± 0.009	0.502 ± 0.006	0.502 ± 0.006	0.534 ± 0.039
26	0.500 ± 0.000	0.562 ± 0.062	0.500 ± 0.000	0.655 ± 0.127
27	0.515 ± 0.051	0.525 ± 0.075	0.498 ± 0.005	0.572 ± 0.107
28	0.558 ± 0.027	0.510 ± 0.017	0.503 ± 0.007	0.616 ± 0.041
29	0.591 ± 0.096	0.598 ± 0.066	0.528 ± 0.034	0.665 ± 0.084
30	0.602 ± 0.048	0.506 ± 0.018	0.500 ± 0.000	0.725 ± 0.053
31	0.500 ± 0.000	0.500 ± 0.000	0.500 ± 0.000	0.514 ± 0.031
32	0.887 ± 0.082	0.500 ± 0.000	0.500 ± 0.000	0.953 ± 0.057
33	0.530 ± 0.026	0.457 ± 0.035	0.463 ± 0.036	0.540 ± 0.026
34	0.500 ± 0.001	0.500 ± 0.001	0.500 ± 0.000	0.543 ± 0.064
35	0.532 ± 0.046	0.396 ± 0.027	0.430 ± 0.046	0.557 ± 0.022
36	0.504 ± 0.011	0.500 ± 0.000	0.500 ± 0.000	0.591 ± 0.070
37	1.000 ± 0.000	1.000 ± 0.000	0.973 ± 0.042	1.000 ± 0.000
38	0.500 ± 0.000	0.500 ± 0.000	0.500 ± 0.000	0.539 ± 0.113
39	0.959 ± 0.032	0.973 ± 0.030	0.977 ± 0.030	1.000 ± 0.000
40	0.500 ± 0.000	0.500 ± 0.000	0.500 ± 0.000	0.508 ± 0.016

G-mean (based on precision and recall)

ID	DE-Forest	RandomFS	RandomFS_b	DT
1	0.787 ± 0.034	0.804 ± 0.048	0.798 ± 0.040	0.733 ± 0.027
2	0.725 ± 0.014	0.705 ± 0.016	0.703 ± 0.017	0.687 ± 0.033
3	0.724 ± 0.014	0.691 ± 0.018	0.695 ± 0.015	0.687 ± 0.033
4	0.658 ± 0.027	0.669 ± 0.054	0.675 ± 0.047	0.656 ± 0.028
5	0.933 ± 0.008	0.851 ± 0.023	0.851 ± 0.023	0.933 ± 0.007
6	0.970 ± 0.003	0.968 ± 0.003	0.967 ± 0.004	0.965 ± 0.003
7	0.948 ± 0.011	0.912 ± 0.022	0.920 ± 0.027	0.941 ± 0.011
8	0.942 ± 0.013	0.891 ± 0.029	0.889 ± 0.030	0.938 ± 0.021
9	0.865 ± 0.021	0.870 ± 0.029	0.853 ± 0.008	0.853 ± 0.028
10	0.895 ± 0.022	0.864 ± 0.017	0.866 ± 0.022	0.858 ± 0.015
11	0.923 ± 0.006	0.904 ± 0.007	0.894 ± 0.021	0.898 ± 0.009
12	0.949 ± 0.013	0.938 ± 0.013	0.934 ± 0.018	0.937 ± 0.012
13	0.926 ± 0.027	0.887 ± 0.033	0.900 ± 0.032	0.933 ± 0.013
14	0.949 ± 0.016	0.904 ± 0.030	0.903 ± 0.027	0.951 ± 0.019
15	0.874 ± 0.017	0.880 ± 0.023	0.884 ± 0.024	0.856 ± 0.024
16	0.951 ± 0.011	0.918 ± 0.017	0.923 ± 0.016	0.939 ± 0.018
17	0.878 ± 0.012	0.888 ± 0.023	0.888 ± 0.022	0.853 ± 0.022
18	0.929 ± 0.028	0.906 ± 0.023	0.906 ± 0.024	0.907 ± 0.021
19	0.991 ± 0.007	0.988 ± 0.004	0.986 ± 0.005	0.984 ± 0.011
20	0.939 ± 0.009	0.927 ± 0.017	0.929 ± 0.015	0.918 ± 0.009
21	0.954 ± 0.023	0.967 ± 0.009	0.948 ± 0.018	0.967 ± 0.022
22	0.964 ± 0.016	0.941 ± 0.007	0.938 ± 0.000	0.954 ± 0.010
23	0.940 ± 0.004	0.940 ± 0.001	0.940 ± 0.002	0.937 ± 0.006
24	0.960 ± 0.008	0.949 ± 0.001	0.949 ± 0.001	0.948 ± 0.004
25	0.955 ± 0.007	0.951 ± 0.003	0.952 ± 0.006	0.939 ± 0.007
26	0.951 ± 0.000	0.963 ± 0.012	0.951 ± 0.000	0.960 ± 0.014
27	0.957 ± 0.010	0.958 ± 0.013	0.954 ± 0.009	0.952 ± 0.016
28	0.973 ± 0.003	0.966 ± 0.005	0.965 ± 0.004	0.962 ± 0.003
29	0.972 ± 0.009	0.974 ± 0.009	0.968 ± 0.006	0.970 ± 0.005
30	0.978 ± 0.004	0.966 ± 0.005	0.965 ± 0.001	0.971 ± 0.005
31	0.970 ± 0.000	0.970 ± 0.000	0.971 ± 0.000	0.961 ± 0.004
32	0.995 ± 0.003	0.972 ± 0.001	0.972 ± 0.001	0.996 ± 0.003
33	0.845 ± 0.022	0.801 ± 0.023	0.806 ± 0.024	0.847 ± 0.017
34	0.978 ± 0.001	0.978 ± 0.001	0.978 ± 0.000	0.973 ± 0.003
35	0.907 ± 0.019	0.844 ± 0.027	0.866 ± 0.014	0.890 ± 0.019
36	0.980 ± 0.001	0.980 ± 0.000	0.980 ± 0.000	0.978 ± 0.002
37	1.000 ± 0.000	1.000 ± 0.000	0.999 ± 0.001	1.000 ± 0.000
38	0.983 ± 0.001	0.983 ± 0.001	0.983 ± 0.001	0.979 ± 0.007
39	0.999 ± 0.001	0.999 ± 0.001	1.000 ± 0.001	1.000 ± 0.000
40	0.989 ± 0.000	0.988 ± 0.000	0.989 ± 0.000	0.984 ± 0.002

Recall

ID	DE-Forest	RandomFS	RandomFS_b	DT
1	0.787 ± 0.031	0.805 ± 0.046	0.797 ± 0.039	0.731 ± 0.026
2	0.733 ± 0.014	0.715 ± 0.014	0.713 ± 0.015	0.686 ± 0.034
3	0.733 ± 0.014	0.701 ± 0.018	0.706 ± 0.014	0.686 ± 0.034
4	0.673 ± 0.025	0.724 ± 0.031	0.724 ± 0.012	0.654 ± 0.027
5	0.936 ± 0.008	0.890 ± 0.001	0.891 ± 0.001	0.932 ± 0.007
6	0.971 ± 0.003	0.968 ± 0.003	0.967 ± 0.004	0.965 ± 0.003
7	0.950 ± 0.010	0.916 ± 0.011	0.924 ± 0.017	0.941 ± 0.011
8	0.941 ± 0.014	0.909 ± 0.012	0.904 ± 0.016	0.937 ± 0.021
9	0.895 ± 0.013	0.893 ± 0.016	0.897 ± 0.006	0.853 ± 0.033
10	0.906 ± 0.010	0.902 ± 0.001	0.902 ± 0.002	0.855 ± 0.019
11	0.927 ± 0.005	0.907 ± 0.003	0.905 ± 0.003	0.896 ± 0.010
12	0.948 ± 0.013	0.937 ± 0.012	0.934 ± 0.015	0.936 ± 0.012
13	0.930 ± 0.017	0.904 ± 0.018	0.914 ± 0.013	0.931 ± 0.016
14	0.950 ± 0.014	0.918 ± 0.014	0.917 ± 0.010	0.950 ± 0.021
15	0.909 ± 0.007	0.903 ± 0.015	0.914 ± 0.007	0.851 ± 0.035
16	0.952 ± 0.009	0.923 ± 0.007	0.927 ± 0.006	0.936 ± 0.020
17	0.912 ± 0.009	0.912 ± 0.013	0.918 ± 0.006	0.847 ± 0.029
18	0.939 ± 0.015	0.927 ± 0.009	0.930 ± 0.009	0.908 ± 0.026
19	0.991 ± 0.007	0.987 ± 0.004	0.986 ± 0.005	0.984 ± 0.011
20	0.946 ± 0.004	0.940 ± 0.006	0.943 ± 0.004	0.917 ± 0.009
21	0.961 ± 0.018	0.966 ± 0.011	0.957 ± 0.010	0.964 ± 0.026
22	0.969 ± 0.009	0.959 ± 0.001	0.959 ± 0.000	0.953 ± 0.011
23	0.947 ± 0.008	0.960 ± 0.001	0.959 ± 0.001	0.939 ± 0.010
24	0.966 ± 0.003	0.966 ± 0.001	0.966 ± 0.001	0.947 ± 0.006
25	0.967 ± 0.001	0.967 ± 0.001	0.967 ± 0.001	0.937 ± 0.011
26	0.967 ± 0.000	0.971 ± 0.004	0.967 ± 0.000	0.963 ± 0.012
27	0.968 ± 0.008	0.971 ± 0.008	0.967 ± 0.010	0.952 ± 0.018
28	0.976 ± 0.001	0.975 ± 0.001	0.975 ± 0.000	0.962 ± 0.005
29	0.977 ± 0.006	0.977 ± 0.007	0.976 ± 0.003	0.971 ± 0.005
30	0.979 ± 0.003	0.977 ± 0.001	0.976 ± 0.001	0.968 ± 0.007
31	0.980 ± 0.001	0.980 ± 0.001	0.980 ± 0.000	0.960 ± 0.007
32	0.995 ± 0.003	0.982 ± 0.001	0.982 ± 0.001	0.996 ± 0.003
33	0.851 ± 0.021	0.812 ± 0.021	0.815 ± 0.020	0.844 ± 0.017
34	0.985 ± 0.002	0.985 ± 0.002	0.986 ± 0.000	0.973 ± 0.006
35	0.912 ± 0.018	0.852 ± 0.019	0.870 ± 0.015	0.887 ± 0.021
36	0.986 ± 0.001	0.986 ± 0.000	0.986 ± 0.000	0.979 ± 0.003
37	1.000 ± 0.000	1.000 ± 0.000	0.999 ± 0.001	1.000 ± 0.000
38	0.988 ± 0.001	0.988 ± 0.001	0.988 ± 0.001	0.980 ± 0.008
39	0.999 ± 0.001	0.999 ± 0.001	1.000 ± 0.001	1.000 ± 0.000
40	0.992 ± 0.000	0.992 ± 0.000	0.992 ± 0.000	0.983 ± 0.003

Specificity

ID	DE-Forest	RandomFS	RandomFS_b	DT
1	0.692 ± 0.039	0.732 ± 0.060	0.698 ± 0.061	0.693 ± 0.043
2	0.867 ± 0.011	0.846 ± 0.016	0.843 ± 0.009	0.859 ± 0.017
3	0.869 ± 0.010	0.843 ± 0.012	0.838 ± 0.013	0.859 ± 0.017
4	0.401 ± 0.041	0.294 ± 0.031	0.295 ± 0.031	0.471 ± 0.063
5	0.583 ± 0.089	0.112 ± 0.004	0.113 ± 0.007	0.739 ± 0.051
6	0.832 ± 0.026	0.805 ± 0.034	0.790 ± 0.032	0.842 ± 0.031
7	0.652 ± 0.076	0.247 ± 0.096	0.329 ± 0.164	0.715 ± 0.055
8	0.557 ± 0.142	0.229 ± 0.122	0.205 ± 0.104	0.710 ± 0.117
9	0.139 ± 0.052	0.148 ± 0.071	0.098 ± 0.006	0.302 ± 0.117
10	0.213 ± 0.060	0.106 ± 0.014	0.106 ± 0.014	0.382 ± 0.087
11	0.433 ± 0.053	0.178 ± 0.047	0.146 ± 0.034	0.543 ± 0.055
12	0.549 ± 0.130	0.429 ± 0.096	0.406 ± 0.109	0.681 ± 0.123
13	0.442 ± 0.175	0.220 ± 0.138	0.237 ± 0.116	0.660 ± 0.137
14	0.617 ± 0.171	0.245 ± 0.122	0.227 ± 0.116	0.716 ± 0.125
15	0.098 ± 0.032	0.130 ± 0.053	0.121 ± 0.049	0.276 ± 0.086
16	0.551 ± 0.077	0.255 ± 0.089	0.262 ± 0.079	0.725 ± 0.096
17	0.094 ± 0.033	0.124 ± 0.070	0.104 ± 0.044	0.227 ± 0.116
18	0.301 ± 0.171	0.132 ± 0.070	0.132 ± 0.099	0.387 ± 0.124
19	0.886 ± 0.074	0.839 ± 0.075	0.818 ± 0.085	0.892 ± 0.128
20	0.174 ± 0.046	0.084 ± 0.030	0.098 ± 0.037	0.342 ± 0.099
21	0.391 ± 0.294	0.533 ± 0.126	0.225 ± 0.138	0.719 ± 0.217
22	0.377 ± 0.240	0.051 ± 0.029	0.041 ± 0.000	0.510 ± 0.174
23	0.147 ± 0.091	0.040 ± 0.001	0.040 ± 0.001	0.201 ± 0.065
24	0.125 ± 0.057	0.034 ± 0.001	0.034 ± 0.001	0.290 ± 0.092
25	0.044 ± 0.017	0.037 ± 0.011	0.037 ± 0.011	0.130 ± 0.076
26	0.033 ± 0.000	0.154 ± 0.121	0.033 ± 0.000	0.347 ± 0.243
27	0.062 ± 0.099	0.079 ± 0.145	0.030 ± 0.006	0.191 ± 0.204
28	0.139 ± 0.053	0.045 ± 0.034	0.032 ± 0.013	0.270 ± 0.081
29	0.205 ± 0.188	0.219 ± 0.128	0.080 ± 0.068	0.358 ± 0.167
30	0.225 ± 0.094	0.035 ± 0.034	0.024 ± 0.001	0.481 ± 0.106
31	0.020 ± 0.000	0.020 ± 0.000	0.020 ± 0.000	0.068 ± 0.060
32	0.779 ± 0.161	0.018 ± 0.001	0.018 ± 0.001	0.910 ± 0.111
33	0.709 ± 0.062	0.538 ± 0.081	0.552 ± 0.086	0.750 ± 0.065
34	0.014 ± 0.000	0.014 ± 0.000	0.014 ± 0.000	0.113 ± 0.132
35	0.640 ± 0.114	0.323 ± 0.060	0.400 ± 0.114	0.741 ± 0.050
36	0.021 ± 0.023	0.014 ± 0.000	0.014 ± 0.000	0.203 ± 0.141
37	1.000 ± 0.000	1.000 ± 0.000	0.947 ± 0.082	1.000 ± 0.000
38	0.012 ± 0.001	0.012 ± 0.001	0.012 ± 0.001	0.098 ± 0.221
39	0.919 ± 0.063	0.946 ± 0.060	0.955 ± 0.060	1.000 ± 0.000
40	0.008 ± 0.000	0.008 ± 0.000	0.008 ± 0.000	0.032 ± 0.030

Precision

ID	DE-Forest	RandomFS	RandomFS_b	DT
1	0.788 ± 0.036	0.803 ± 0.049	0.799 ± 0.041	0.736 ± 0.028
2	0.717 ± 0.015	0.696 ± 0.018	0.693 ± 0.018	0.688 ± 0.033
3	0.716 ± 0.014	0.681 ± 0.018	0.684 ± 0.016	0.688 ± 0.033
4	0.643 ± 0.031	0.621 ± 0.089	0.633 ± 0.083	0.659 ± 0.031
5	0.931 ± 0.009	0.815 ± 0.044	0.815 ± 0.045	0.934 ± 0.008
6	0.970 ± 0.003	0.967 ± 0.003	0.966 ± 0.004	0.965 ± 0.003
7	0.947 ± 0.012	0.909 ± 0.035	0.916 ± 0.039	0.941 ± 0.010
8	0.942 ± 0.013	0.873 ± 0.046	0.876 ± 0.046	0.939 ± 0.021
9	0.836 ± 0.035	0.848 ± 0.045	0.812 ± 0.010	0.852 ± 0.027
10	0.884 ± 0.035	0.828 ± 0.033	0.833 ± 0.041	0.862 ± 0.014
11	0.919 ± 0.007	0.902 ± 0.014	0.883 ± 0.038	0.899 ± 0.009
12	0.950 ± 0.012	0.939 ± 0.014	0.935 ± 0.022	0.938 ± 0.013
13	0.922 ± 0.038	0.869 ± 0.050	0.886 ± 0.052	0.936 ± 0.011
14	0.948 ± 0.018	0.891 ± 0.046	0.890 ± 0.044	0.951 ± 0.017
15	0.841 ± 0.029	0.857 ± 0.038	0.855 ± 0.041	0.861 ± 0.018
16	0.951 ± 0.013	0.912 ± 0.028	0.919 ± 0.028	0.942 ± 0.016
17	0.845 ± 0.019	0.864 ± 0.037	0.859 ± 0.037	0.859 ± 0.019
18	0.919 ± 0.042	0.886 ± 0.039	0.883 ± 0.040	0.906 ± 0.019
19	0.991 ± 0.006	0.988 ± 0.004	0.985 ± 0.005	0.985 ± 0.012
20	0.933 ± 0.015	0.914 ± 0.029	0.916 ± 0.026	0.919 ± 0.010
21	0.948 ± 0.031	0.969 ± 0.008	0.939 ± 0.028	0.970 ± 0.019
22	0.959 ± 0.023	0.923 ± 0.014	0.919 ± 0.000	0.956 ± 0.011
23	0.932 ± 0.007	0.921 ± 0.002	0.921 ± 0.002	0.935 ± 0.004
24	0.954 ± 0.013	0.932 ± 0.001	0.932 ± 0.001	0.949 ± 0.005
25	0.943 ± 0.014	0.937 ± 0.006	0.938 ± 0.011	0.940 ± 0.005
26	0.936 ± 0.000	0.956 ± 0.020	0.936 ± 0.000	0.957 ± 0.017
27	0.946 ± 0.015	0.945 ± 0.018	0.941 ± 0.011	0.951 ± 0.016
28	0.970 ± 0.005	0.956 ± 0.009	0.955 ± 0.008	0.963 ± 0.003
29	0.967 ± 0.014	0.971 ± 0.012	0.960 ± 0.011	0.970 ± 0.006
30	0.976 ± 0.005	0.956 ± 0.008	0.953 ± 0.001	0.973 ± 0.004
31	0.961 ± 0.000	0.961 ± 0.000	0.961 ± 0.000	0.962 ± 0.002
32	0.995 ± 0.003	0.963 ± 0.001	0.963 ± 0.001	0.996 ± 0.003
33	0.840 ± 0.024	0.790 ± 0.025	0.796 ± 0.027	0.850 ± 0.020
34	0.971 ± 0.000	0.971 ± 0.000	0.971 ± 0.000	0.973 ± 0.003
35	0.902 ± 0.021	0.836 ± 0.035	0.862 ± 0.016	0.894 ± 0.017
36	0.974 ± 0.002	0.973 ± 0.000	0.973 ± 0.000	0.978 ± 0.003
37	1.000 ± 0.000	1.000 ± 0.000	0.999 ± 0.001	1.000 ± 0.000
38	0.977 ± 0.001	0.977 ± 0.001	0.977 ± 0.001	0.979 ± 0.006
39	0.999 ± 0.001	0.999 ± 0.001	1.000 ± 0.001	1.000 ± 0.000
40	0.985 ± 0.000	0.985 ± 0.000	0.985 ± 0.000	0.985 ± 0.000

Time [s]

ID	DE-Forest	RandomFS	RandomFS_b	DT
1	2091.672	1.939	1.333	0.161
2	4972.587	2.911	3.241	0.237
3	4919.736	2.876	3.065	0.222
4	2692.734	1.177	1.173	0.119
5	6501.641	4.043	3.778	0.383
6	15442.146	5.420	5.170	0.322
7	2865.769	1.890	1.919	0.188
8	1959.435	1.868	1.680	0.163
9	1709.673	1.581	1.436	0.141
10	2930.864	1.907	2.015	0.188
11	5883.107	2.890	2.489	0.206
12	2027.744	1.710	1.541	0.154
13	1980.429	1.694	1.466	0.161
14	1609.911	1.346	1.408	0.228
15	2384.493	1.661	1.446	0.200
16	2368.928	1.973	2.018	0.236
17	2401.345	1.953	1.765	0.356
18	1964.383	2.166	1.886	0.196
19	2800.092	2.362	1.759	0.257
20	3597.666	2.904	2.323	0.242
21	1934.320	1.427	1.494	0.198
22	2574.093	2.168	1.741	0.146
23	9981.600	3.910	2.828	0.180
24	4142.914	2.168	2.100	0.143
25	7361.080	4.686	4.067	0.270
26	2174.480	1.853	1.996	0.171
27	1880.862	1.726	1.516	0.195
28	11467.678	5.069	4.869	0.209
29	2991.776	2.464	2.074	0.221
30	5625.531	4.088	3.771	0.190
31	6745.806	3.668	3.448	0.245
32	5474.070	3.489	3.432	0.191
33	2449.765	1.988	1.947	0.195
34	3502.271	2.866	2.467	0.242
35	2326.333	2.051	2.019	0.208
36	7332.644	5.120	4.497	0.311
37	7458.700	4.855	5.454	0.310
38	5432.953	3.875	3.986	0.214
39	5925.743	3.617	3.610	0.138
40	7864.928	4.438	4.548	0.202