Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

SpatialPooler with log boosting #545

Open
wants to merge 15 commits into
base: master
from

Conversation

Projects
None yet
2 participants
@breznak
Copy link
Member

commented Jul 6, 2019

instead of current exponential.

Log boosting has been discussed by @ctrl-z-9000-times on the forums and I believe is actually used in #293

I have tried experimenting with it in the SP.

  • Actually I don't plan merging this, as it currently has slightly worse performance and worse computation time.
    • on MNIST:

[master, exponential boost * strength 7.0]
MNIST train time: 61.1118
Testing for 10000 cycles ...
..........
===========RESULTs=================
Score: 95.31% (469 / 10000 wrong).

vs

[this PR, log boosting, parameterless]
MNIST train time: 72.5795
Testing for 10000 cycles ...
..........
===========RESULTs=================
Score: 95.18% (482 / 10000 wrong)

breznak added some commits Jul 4, 2019

if(boost < htm::Epsilon) return; //skip for disabled boosting
output[i] = exp((targetDensity - actualDensity[i]) * boost); //TODO doc this code
//output[i] = exp((targetDensity - actualDensity[i]) * boost); //exponential boosting, default for Numenta
output[i] = log(actualDensity[i]) / log(targetDensity);

This comment has been minimized.

Copy link
@ctrl-z-9000-times

ctrl-z-9000-times Jul 6, 2019

Performance optimizations, which helped when I did this in the columnPooler2 branch.

  • Move the log(targetDensity) to the outermost loop, since it is constant across all of the mini-columns. In this case calling code contains the loop which you should avoid, not this function.
  • Use log2 instead of log

This comment has been minimized.

Copy link
@breznak

breznak Jul 6, 2019

Author Member
log2 instead of log

done

log(targetDensity) to the outermost loop, since it is constant

the problem here is I still need to keep local inh working, which changes the density for each column

@ctrl-z-9000-times

This comment has been minimized.

Copy link

commented Jul 6, 2019

Thanks for putting in the effort to evaluate this function.

What I would really like to see from your analysis are the activation frequencies of the cells (AKA variable actualDensity in the source code). The MNIST experiment should print the min/mean/std/max of them. If boosting is working well then all of the cells will have an activation frequency which is close to the target sparsity.

@breznak breznak referenced this pull request Jul 6, 2019

Open

Boosting cleanup #544

@breznak

This comment has been minimized.

Copy link
Member Author

commented Jul 6, 2019

Log boost

(optimized for global inh for this case)

columnStats SDR( 28 28 8 )
Sparsity Min/Mean/Std/Max 0 / 0.0829329 / 0.0260002 / 0.0999681
Activation Frequency Min/Mean/Std/Max 4.74358e-05 / 0.0829303 / 0.0771232 / 0.420391
Entropy 0.870742
Overlap Min/Mean/Std/Max 0 / 0.0665764 / 0.100776 / 0.711324

Spatial Pooler Connections:
Inputs (784) ~> Outputs (6272) via Segments (6272)
Segments on Cell Min/Mean/Max 1 / 1 / 1
Potential Synapses on Segment Min/Mean/Max 23 / 23 / 23
Connected Synapses on Segment Min/Mean/Max 6 / 13.2816 / 23
Synapses Dead (0.3717%) Saturated (0.333546%)

MNIST train time: 63.9549
Testing for 10000 cycles ...
..........
===========RESULTs=================
Score: 95.21% (479 / 10000 wrong).

Exp boost (str = 7.0)

columnStats SDR( 28 28 8 )
Sparsity Min/Mean/Std/Max 0.000159439 / 0.072963 / 0.0301806 / 0.0999681
Activation Frequency Min/Mean/Std/Max 0 / 0.0729609 / 0.0556396 / 0.23982
Entropy 0.900655
Overlap Min/Mean/Std/Max 0 / 0.0436669 / 0.0828263 / 1

Spatial Pooler Connections:
Inputs (784) ~> Outputs (6272) via Segments (6272)
Segments on Cell Min/Mean/Max 1 / 1 / 1
Potential Synapses on Segment Min/Mean/Max 23 / 23 / 23
Connected Synapses on Segment Min/Mean/Max 6 / 13.6775 / 23
Synapses Dead (0.336388%) Saturated (0.323619%)

MNIST train time: 61.6571
Testing for 10000 cycles ...
..........
===========RESULTs=================
Score: 95.1% (490 / 10000 wrong)

Exp, str=2.0

columnStats SDR( 28 28 8 )
Sparsity Min/Mean/Std/Max 0 / 0.0868832 / 0.0236562 / 0.0999681
Activation Frequency Min/Mean/Std/Max 0 / 0.0868804 / 0.116585 / 0.501022
Entropy 0.72438
Overlap Min/Mean/Std/Max 0 / 0.14655 / 0.17099 / 0.880383
Spatial Pooler Connections:
Inputs (784) ~> Outputs (6272) via Segments (6272)
Segments on Cell Min/Mean/Max 1 / 1 / 1
Potential Synapses on Segment Min/Mean/Max 23 / 23 / 23
Connected Synapses on Segment Min/Mean/Max 6 / 14.5979 / 23
Synapses Dead (0.201399%) Saturated (0.278332%)
MNIST train time: 62.3814
Testing for 10000 cycles ...
..........
===========RESULTs=================
Score: 94.3% (570 / 10000 wrong).

Boosting none

columnStats SDR( 28 28 8 )
Sparsity Min/Mean/Std/Max 0.00334821 / 0.0981863 / 0.00904482 / 0.0999681
Activation Frequency Min/Mean/Std/Max 0 / 0.0981854 / 0.22962 / 0.953333
Entropy 0.357319
Overlap Min/Mean/Std/Max 0.0191388 / 0.628503 / 0.23122 / 1

Spatial Pooler Connections:
Inputs (784) ~> Outputs (6272) via Segments (6272)
Segments on Cell Min/Mean/Max 1 / 1 / 1
Potential Synapses on Segment Min/Mean/Max 23 / 23 / 23
Connected Synapses on Segment Min/Mean/Max 6 / 14.9954 / 23
Synapses Dead (0.0604204%) Saturated (0.21073%)

MNIST train time: 60.7151
Testing for 10000 cycles ...
..........
===========RESULTs================
Score: 93.04% (696 / 10000 wrong).

Target sparsity is 0.1

@breznak breznak marked this pull request as ready for review Jul 6, 2019

@breznak breznak added the ready label Jul 6, 2019

@@ -744,6 +744,7 @@ void SpatialPooler::updateDutyCyclesHelper_(vector<Real> &dutyCycles,


void SpatialPooler::updateBoostFactors_() {
if(boostStrength_ < htm::Epsilon) return; //skip for disabled boosting

This comment has been minimized.

Copy link
@ctrl-z-9000-times

ctrl-z-9000-times Jul 6, 2019

This won't work because SpatialPooler::BOOSTING_LOG < htm::Epsilon and so it will skip log-boosting.

This comment has been minimized.

Copy link
@breznak

breznak Jul 6, 2019

Author Member

I could use fabs(), or use exact match str_ == DISABLED

@breznak breznak removed the ready label Jul 12, 2019

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.