BacktrackingTM Removal

[ctrl-z-9000-times]

Pros/Cons comparison versus the regular Temporal Memory.

Pros:

• Outperforms the regular Temporal Memory on the Numenta Anomaly Benchmark (NAB) by ~5%. There are 3 scores for NAB, all of which have a maximum of 100.

BacktrackingTM: 70.1 	63.1 	74.3
RegularTM:	64.6 	58.8 	69.6

Cons:

• It's not biologically plausible.
• It's specialized for anomaly detection.

rhyolight
tm & tm_cpp are the newer Temporal Memory algorithm (python and cpp), which is more biologically pure. The other implementations are the BacktrackingTM (both python and cpp), which is performs better for anomaly detection tasks because of extra bells and whistles added specifically for anomaly detection tasks.
https://discourse.numenta.org/t/is-there-a-good-write-up-re-differences-between-cpp-and-tm-cpp/4249/2

• Numenta has stopped working on backtracking in favor of the regular TM

rhyolight
FYI: The backtracking version of the TM is not being further developed. Current research uses TemporalMemory 5 or extensions of it.
https://discourse.numenta.org/t/is-there-a-good-write-up-re-differences-between-cpp-and-tm-cpp/4249/4

• It has never been published in a peer reviewed journal. Details about how it works inside are hard to find. We, the implementers and maintainers of the code can't even explain what all of the options do. Specifically what are the following options for?

  • doPooling
  • segUpdateValidDuration: TODO: document. Yes this todo note is in the docs & source code.
  • anomalyMode & inferenceMode
  • printColConfidence what? how?

• pamLength. This is one of those bells or whistles. From the docs:

(int) Number of time steps to remain in “Pay Attention Mode” after we detect we’ve reached the end of a learned sequence. Setting this to 0 disables PAM mode. When we are in PAM mode, we do not burst unpredicted columns during learning, which in turn prevents us from falling into a previously learned sequence for a while (until we run through another ‘pamLength’ steps).

The advantage of PAM mode is that it requires fewer presentations to learn a set of sequences which share elements. The disadvantage of PAM mode is that if a learned sequence is immediately followed by set set of elements that should be learned as a 2nd sequence, the first pamLength elements of that sequence will not be learned as part of that 2nd sequence.

Also PAM is required, from the source:
NTA_ASSERT(pamLength > 0) << "This implementation must have pamLength > 0";

• maxSeqLength. Another feature for anomaly detection. From the docs:

(int) If not 0, we will never learn more than maxSeqLength inputs in a row without starting over at start cells. This sets an upper bound on the length of learned sequences and thus is another means (besides maxAge and globalDecay) by which to limit how much the TM tries to learn.

• Confusing, has 26 arguments.

• Unit tests for the current C++ implementation

  • 3 test cases. Implementation is ~1800 lines, unit tests are ~400 lines.
  • No test actually looks at the results...
  • No test for anomaly detection...
  • All tests use default parameters...

In favor of Removal

1. It's in version control, so it's not gone forever. The leader board NAB results can still be reproduced.
2. It would allow us to spend more time making the regular TM better.

[breznak]

Similar/related #95 with removal of TP (BackTM depends on TP)

[breznak]

It's not biologically plausible.

this and the fact that the code is not being actively developed/used.

The idea of having only one, universal implementation (for TM) is tempting,
on the other hand, the code is there, working, and currently does not hurt(require action) from anybody.

The timing is little unfortunate, as we've just merged the BackTM region PR.

I'd be either for removing both BackTM and TP(Cells4), or for letting that code be until it breaks severely enough. Slightly in favor for the removal, but no strong opinion.

[ctrl-z-9000-times]

the [C++] code is there, working, and currently does not require action from anybody.

I'm not so sure it's working because:

• It's newly written code
• The unit tests don't check any of the critical properties, instead checking if it crashes when run
• There are no examples of it in action, it's never been trained on real world data.

[ctrl-z-9000-times]

Cells4 appears to implement the temporal memory algorithm. It contains references to payAttentionMode and it is only used by the backtrackingTM.

• Cells4 is also used by the example HelloSpTp but this tests the performance of it and does not check the results in any meaningful way. HelloSpTp could easily be changed to HelloSpTm since it already includes a regular TM.

---

I think we've reached a consensus. We all agree that BacktrackingTM & Cells4 should probably go away. Unless there are any objections?

[dkeeney]

I think we've reached a consensus. We all agree that BacktrackingTM & Cells4 should probably go away. Unless there are any objections?

I do not agree. I don't think we should spend any more time on it ...i.e. don't put SDR's in it. But it does not hurt to remain. I propose that we leave it until it "gets in the way". I think it might be useful for comparisons.

[dkeeney]

It's newly written code

No, the BacktrackingTMRegion is new but Cells4 has been there for a long time as far as I can tell.

[breznak]

Closing here and merging to #95