Numenta Anomaly Benchmark

[ctrl-z-9000-times]

Investigate using NAB with htm-community/nupic\

---

EDIT: breznak

During the weekend, I want to try running NAB with our community HTM.

• NAB uses anomaly Likelihood, as stated below
  • possible to run NAB with "raw" scores only (but state of the art seems +likelihood)

Numenta HTM detector with no likelihood uses the raw anomaly scores directly. To run without likelihood, set the variable self.useLikelihood in numenta_detector.py to False.

Scores:

Detector 	Standard Profile 	Reward Low FP 	Reward Low FN
Numenta HTMusing NuPIC v0.5.6* 	70.1 	63.1 	74.3
nab-comportex† 	64.6 	58.8 	69.6
NumentaTM HTM* 	64.6 	56.7 	69.2
HTM Java 	56.8 	50.7 	61.4
Numenta HTM*, no likelihood 	53.62 	34.15 	61.89

• fix our Likelihood and provide bindings
• or use python version of Likelihood (copy from numenta/nupic (py) repo) Provide missing AnomalyLikelihood from python #455
  • this might be a better option, as we want to see bugs/improvements in SP/TM/Anomaly, and not worry about bugs in AnomalyLikelihood implementation.

[ctrl-z-9000-times]

Blocked on python wrappers for anomaly likelihood.

[breznak]

Blocked on python wrappers for anomaly likelihood.

We could still use python Likelihood, but cpp new SP/TM, ...

[breznak]

During the weekend, I want to try running NAB with our community HTM.

• NAB uses anomaly Likelihood, as stated below
  • possible to run NAB with "raw" scores only (but state of the art seems +likelihood)

Numenta HTM detector with no likelihood uses the raw anomaly scores directly. To run without likelihood, set the variable self.useLikelihood in numenta_detector.py to False.

Scores:

Detector 	Standard Profile 	Reward Low FP 	Reward Low FN
Numenta HTMusing NuPIC v0.5.6* 	70.1 	63.1 	74.3
nab-comportex† 	64.6 	58.8 	69.6
NumentaTM HTM* 	64.6 	56.7 	69.2
HTM Java 	56.8 	50.7 	61.4
Numenta HTM*, no likelihood 	53.62 	34.15 	61.89

• fix our Likelihood and provide bindings
• or use python version of Likelihood (copy from numenta/nupic (py) repo) Provide missing AnomalyLikelihood from python #455
  • this might be a better option, as we want to see bugs/improvements in SP/TM/Anomaly, and not worry about bugs in AnomalyLikelihood implementation.

[ctrl-z-9000-times]

Breznak,

There are several other people working on NAB right now too, which you should be aware of.

I forked NAB and converted it to python3 (and python3 only, no py2). I tested it on most of the detectors, except for the HTM based ones which will take a significant effort to make work. It is at:
https://github.com/htm-community/NAB

There is another person working to refactor NAB to be easier to use:
https://github.com/iandanforth?tab=repositories
Who has two repos: NAB & NAB-detectors. The idea is that the code to run & score the detectors and the detectors themselves are split apart so that it's easier to add new detectors. He's also converted some of the code to python3.

And on the Numenta Forum, there is a thread for this topic:
https://discourse.numenta.org/t/does-the-current-community-version-fully-support-nab-in-python-3-6/5915/8

[breznak]

There are several other people working on NAB right now too, which you should be aware of.

thanks for the headsup! I might need your help when dealing with NAB 👍

I'll have to estimate how alive, difficult to work with, and open to new PRs the numenta/NAB is.
If feasible, I'd still try to target the upstream (numenta's) for its recognition and visibility. Even though it's behind with python3, and I'll see how much work is getting to run our API with that of nupic.core's NAB.

[breznak]

• OPF: OPF is a large chunk of code, on the other hand, with a quick review, it does not rely any any removed code from c++, so we might just port enough of its python dependencies (other nupic.xxx code).
  • if we get to support OPF, I can much easily run our HTM in NAB benchmark, might even run out of the box.
  • it would be easier for people to use our repo with python, all those that use OPF models, swarming, etc..

[breznak]

I have started working with NAB, which is curerntly in transition to py3, and I am trying to add htm.core as a runable detector to it.

If you have time, please help me review my code in https://github.com/numenta/NAB/pulls and referenced PRs to Ian's open PR.

I'm looking for help with numenta/NAB#343

[ctrl-z-9000-times]

I have a fork of NAB which works with python3:
https://github.com/htm-community/NAB

The only things which do not work are the HTM based detectors.

[breznak]

I have a fork of NAB which works with python3:
https://github.com/htm-community/NAB
The only things which do not work are the HTM based detectors.

then I'll redirect my PRs also to this repo, I made the HTM detectors runnable with fake python3 (needs py2 installed).

I got message from Matt that the official NAB should be ported by end of the year, which is not really fast moving, unfortunately.

[ctrl-z-9000-times]

not really fast moving, unfortunately.

That's the advantage of using the htm-community version, you don't need to wait for numenta to do a code review.

[breznak]

I'll reopen for now, let's see if we can run the bench as a part of the CI