PUBDEV-7138 Extended Isolation Forest

[valenad1]

The production version of algorithm will be finished here: #5246

Implementation of Extended Isolation Forest algorithm.

https://0xdata.atlassian.net/browse/PUBDEV-7138

All review feedback is implemented there or noted for future implementation.

[maurever]

Looks good! Minor changes are requested.

[valenad1]

Link: https://github.com/valenad1/h2o-3/blob/PUBDEV-7138-extended-isolation-forest-leak-fix/h2o-py/demos/extisofor/extended-isolation-forest-benchmark.ipynb

Extended Isolation Forest Benchmark - Training stage

In the tests, H2O cloud is initialized before each run and shutdown each every run. The algorithm is firstly tested on Training performance and compared to the Isolation Forest run time.

• IF = Isolation Forest
• EIF = Extended Isolation Forest
• N = number of rows
• P = number of columns
• ntrees = number of trees to train (always 100)
• sample_size = how many rows will be used to build a tree
• max_depth = only for IF, how big is the depth of the tree, in EIF is always set on math.ceil(math.log(sample_size, 2)) and max_depth is always depended on sample_size in the benchmark.

Computer parameters:

• Lenovo ThinkPad P53,
• MS Windows 10 Pro x64,
• Intel Core i7-9850H CPU @ 2.60GHz,
• 6 cores and 12 threads,
• 96.0 GB RAM.

Training process:

The algorithm runtime performance depends on ntrees and sample_size parameters. Parameter ntrees is fixed on 100 for all runs. Setting of sample_size and the amount of data is varying in runs. I focus to put values that switch ON/OFF the MRTask.

Results

1.1 Toy data

1.1.1 Small data and small dimension

1.1.2 Small data and high dimension

1.1.3 Big data - small dimension, small sample_size

1.1.4 Big data - high dimension, small sample_size

1.1.5 Big data - small dimension, big sample_size

1.1.6 Big data - high dimension, big sample_size

• N = 100_000
• P = 30
• sample_size = 10_000

Finish with error (More information in conclusion)

H2OConnectionError: Unexpected HTTP error: ('Connection aborted.', RemoteDisconnected('Remote end closed connection without response'))

1.2 Real Credit Card Fraud Detection Data

1.2.1 Test with samle_size = 256 (default value)

1.2.2 Real data - sample_size = 1% (2 848 rows) of data

Finish with error (More information in conclusion)

H2OConnectionError: Unexpected HTTP error: ('Connection aborted.', RemoteDisconnected('Remote end closed connection without response'))

Conclusion

• The biggest problem I see across various runs is the sensitivity on P parameter no matter what the sample_size setting is. EIF is significantly slower than IF even for small data with 500 rows (section 1.1.2). If N grows and sample_size is small, then EIF finishes slower (sections 1.1.4, 1.2.1). The real problem comes with large data with big sample_size where EIF is not able to finish most probably because of memory requirements(sections 1.1.6, 1.2.2, 1.2.3). When I test it in Java, all trees are started to be prepared at once and save their Nodes and temporary Frames which leads to following warning message. With the same settings, the IF finish without any problem with good performance.

WARN water.default: Unblock allocations; cache below desired, but also OOM: OOM, (K/V:139,7 MB + POJO:19,27 GB + FREE:1,87 GB == MEM_MAX:21,27 GB), desiredKV=2,66 GB OOM!

• If P parameter is small and sample_size is small, then EIF performs equally or better than IF.
• If P parameter is small and sample_size is big, then EIF performs equally to IF.

Solution attempt:

I have tried to solve it with better distribution IsolationTree building. I hardcode to build 144 trees (12 on each thread) but with no difference in run time. Even if it works, I don't think it would be usable for a cluster where the performance of each node can and most probably will vary. That is why I only submitting tasks to the H2O cluster (one task per one tree). Trees are operating with their own sub-sample of data and can be built totally independently. But build all trees at one time seems to be memory and time-consuming.

[maurever]

I am sorry @honzasterba and @michalkurka, I sent you a new request for review, but I would like to know just your opinion on the benchmark right now.

But thank you @honzasterba for your new review. We agreed with your previous comments about removing MOJO and extension from SharedTreeModel. @valenad1 just has not implemented it yet, but it is on his todo list.

[valenad1]

The production version of algorithm will be finished here: #5246