Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

MNIST params #533

Merged
merged 13 commits into from Jul 6, 2019

Conversation

Projects
None yet
2 participants
@breznak
Copy link
Member

commented Jul 3, 2019

  • MNIST experiments with manual configuration
    • found better performance
    • and much smaller, runnable model
  • SP cleanup
    • (OT: I'd like to head to removing boosting)

TODO:

  • copy the best params over to python

In OTHER PR:

  • make py mnist runnable out of the box (use c++'s MNIST data)
  • mnist now runs fast enough -> make it a test same as hotgym is

@breznak breznak self-assigned this Jul 3, 2019

* -------------------------------------------------------------------------------------------------------------------------------------------------------------------------
* 1/ Score: 97.11% (289 / 10000 wrong) : 28x28x16 : 4 : 557 : 1f0187fc6 : epochs help, at cost of time
*
* 2/ Score: 96.56% (344 / 10000 wrong) : 28x28x16 : 1 : 142 : 3ccadc6d6

This comment has been minimized.

Copy link
@breznak

breznak Jul 3, 2019

Author Member

competitive results, considering it's single pass, and many "ML" features missing (dropout, data-modification, cross-validation, param opt, many training passes...)

This comment has been minimized.

Copy link
@ctrl-z-9000-times

ctrl-z-9000-times Jul 3, 2019

I literally can not read this table because the lines are too long.

This comment has been minimized.

Copy link
@breznak

breznak Jul 3, 2019

Author Member

better now?

/* synPermActiveInc */ 0.014f,
/* synPermConnected */ 0.1f,
/* minPctOverlapDutyCycles */ 0.001f,
/* synPermInactiveDec */ 0.002f, //FIXME inactive decay permanence plays NO role, investigate! (slightly better w/o it)

This comment has been minimized.

Copy link
@breznak

breznak Jul 3, 2019

Author Member

I never get to decay play a role!?

const Real boost,
vector<Real>& output) {
if(boost < htm::Epsilon) return; //skip for disabled boosting
output[i] = exp((targetDensity - actualDensity[i]) * boost); //TODO doc this code

This comment has been minimized.

Copy link
@breznak

breznak Jul 3, 2019

Author Member

where is a doc for this exp based boosting, please?

@breznak breznak requested review from ctrl-z-9000-times and dkeeney Jul 3, 2019

if(boostStrength_ < htm::Epsilon) {
const auto begin = static_cast<const SynapseIdx*>(overlaps.data());
boosted.assign(begin, begin + overlaps.size());
return; //boost ~ 0.0, we can skip these computations.

This comment has been minimized.

Copy link
@breznak

breznak Jul 3, 2019

Author Member

this probably changed the results for the test

This comment has been minimized.

Copy link
@ctrl-z-9000-times

ctrl-z-9000-times Jul 3, 2019

Can you be a little more certain of this? Why exactly did the unit tests change?

This comment has been minimized.

Copy link
@ctrl-z-9000-times

ctrl-z-9000-times Jul 3, 2019

Nevermind, I see how your unit test works.

This comment has been minimized.

Copy link
@ctrl-z-9000-times

ctrl-z-9000-times Jul 6, 2019

nitpick: boosted.assign(overlaps.begin(), overlaps.end())

@breznak breznak referenced this pull request Jul 3, 2019

Draft

Smarter SP parameters #536

0 of 3 tasks complete
@breznak

This comment has been minimized.

Copy link
Member Author

commented Jul 3, 2019

Can we merge this? I'll use it for testing the other PRs

@@ -172,7 +172,7 @@ void SpatialPooler::setDutyCyclePeriod(UInt dutyCyclePeriod) {
Real SpatialPooler::getBoostStrength() const { return boostStrength_; }

void SpatialPooler::setBoostStrength(Real boostStrength) {
NTA_CHECK(boostStrength >= 0.0f);
NTA_CHECK(boostStrength == 0.0f or boostStrength >= 1.0) << "Boost strength must be >= 1.0, or exactly 0.0 (=disabled).";

This comment has been minimized.

Copy link
@ctrl-z-9000-times

ctrl-z-9000-times Jul 3, 2019

Why is this a restriction? Does it break with 0 < boostStrength < 1 ?

This comment has been minimized.

Copy link
@breznak

breznak Jul 4, 2019

Author Member

I can remove this, my idea was that if we boost, we want to do so in a positive way, not "silencing" the column.
Actually, I'd like to move to "your" log-boost function that is parameterless.

This comment has been minimized.

Copy link
@breznak

breznak Jul 4, 2019

Author Member

reverted these limits. It could make sense to use 0 < strength < 1, to "soften" the boosting.

@breznak breznak requested a review from ctrl-z-9000-times Jul 4, 2019

breznak added some commits Jul 6, 2019

@breznak breznak requested a review from ctrl-z-9000-times Jul 6, 2019

@ctrl-z-9000-times

This comment has been minimized.

Copy link

commented Jul 6, 2019

copy the best params over to python

Please do not do this. The python version already has good parameters. IIRC it scores above 95% after a single iteration through the training data.

@ctrl-z-9000-times
Copy link

left a comment

These changes look fine to me.

@breznak

This comment has been minimized.

Copy link
Member Author

commented Jul 6, 2019

Please do not do this. The python version already has good parameters. IIRC it scores above 95% after a single iteration through the training data.

ok, I won't now. Uups, should have looked there for bootstrap params!
But, one thing I notice there, it uses quite large (~35k) # of columns (and so probably runs slow, on my ntb MNIST before this PR is cca 10mins, now 140s!).
I noticed this and it's one thing we need to consider and should add into the param optimization: with large enough number of columns we can solve any problem (Kolmogorov), but we'll end up with a large network of dummy neurons.
I noticed that with some params, where such network would run faster -> allowing me to increase the number of columns for the same training time. I try to avoid this to some reasonable extent.

@breznak breznak merged commit b59f646 into master Jul 6, 2019

3 of 4 checks passed

ci/circleci: arm64-build-test CircleCI is running your tests
Details
ci/circleci: build-and-test Your tests passed on CircleCI!
Details
continuous-integration/appveyor/pr AppVeyor build succeeded
Details
continuous-integration/travis-ci/pr The Travis CI build passed
Details

@breznak breznak deleted the mnist_params branch Jul 6, 2019

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.