[REVIEW] Test FIL probabilities with absolute error thresholds in python

[levsnv]

Probabilities are limited between [0.0, 1.0]. Also, we generally care more about large probabilities which are O(1/n_classes).
The largest relative probability errors are usually caused by a small ground truth probability (e.g. 1e-3), as opposed to a large absolute error.
Hence, relative probability error is not the best metric. Absolute probability error is more relevant.
Moreover, absolute probability error is more stable, as relative errors have a long tail. When training or even inferring on many rows, the chance of getting a ground truth probability sized 1e-3 or 1e-4 grows. In some cases, there is no reasonable and reliable threshold. Last, if the number of predicted probabilities (clipped values) per input row grows, so does the long tail of relative probability errors, due to less undersampling. This unfairly compares binary classification with regression, and multiclass classification with binary classification.

The changes below are based on collecting absolute errors under --run_unit, --run_quality and --run_stress. These thresholds are violated at most a couple times per million samples, in most cases never.

[levsnv]

Below are the absolute error distributions collected in the process
https://gist.github.com/levsnv/9af496ca27778724191d6b7658eddfd7

[JohnZed]

I think this is a clear win for clarity of the testing guarantees. The old rel thresholds were hard to reason about. Here, we are clarifying that we will test proba to a 3e-7 threshold or better in all cases. So I like it ;) Thanks!

[JohnZed]

I will wait for @canonizer ’s comment before merging though in case he as additional thoughts

[levsnv]

Thanks John! Could you also review #2894 ? Just don't merge it yet, so that we have a proper documentation of where changes came from.

[levsnv]

rerun tests
./test/ml: symbol lookup error: ./test/ml: undefined symbol: _ZN5faiss3gpu20StandardGpuResources20setCudaMallocWarningEb

[codecov-io]

Codecov Report

Merging #3582 (4e61c31) into branch-0.19 (9fa6e17) will increase coverage by 45.58%.
The diff coverage is n/a.

@@               Coverage Diff                @@
##           branch-0.19    #3582       +/-   ##
================================================
+ Coverage        35.24%   80.83%   +45.58%     
================================================
  Files              378      227      -151     
  Lines            26910    17737     -9173     
================================================
+ Hits              9485    14337     +4852     
+ Misses           17425     3400    -14025     

Flag	Coverage Δ
dask	45.30% <ø> (+0.30%)	⬆️
non-dask	73.09% <ø> (?)

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files	Coverage Δ
python/cuml/dask/solvers/cd.py	100.00% <0.00%> (ø)
python/cuml/common/numba_utils.py	0.00% <0.00%> (ø)
python/cuml/neighbors/__init__.py	100.00% <0.00%> (ø)
python/cuml/internals/global_settings.py	100.00% <0.00%> (ø)
python/cuml/dask/preprocessing/encoders.py	100.00% <0.00%> (ø)
cuml/dask/solvers/__init__.py
cuml/dask/naive_bayes/naive_bayes.py
cuml/datasets/__init__.py
cuml/experimental/explainer/__init__.py
cuml/dask/linear_model/linear_regression.py
... and 285 more

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 9fa6e17...4e61c31. Read the comment docs.

[dantegd]

@gpucibot merge