Implementing the t-SNE algorithm

[AtharvaKhare]

WIP

The tasks are:

• Apply PCA on input data to bring dimensions down to 50
• Hbeta method; which, for a given pairwise distance, calculates Gaussian kernel values
• Complete the x2p method
• Apply gradient descent
• Write examples and remaining tests

[AtharvaKhare]

Can someone with write access update these tasks in the "Projects" tab?

@ SergeStinckwich I modified the structure of the class to include params and gradientDescentParams, please review it.
In future, if we create a seperate gradientDescent class/method, it will become easier to refactor. Also, if we want to include additional parameters, such as earlyExaggeration and earlyExaggerationIteration as suggested in FItSNE paper, having a dictionary will improve readability.
Should I use the new structure or continue with the old one?

[AtharvaKhare]

Please review 21cea90, especially the tests.

entropyOf: andPRow: withBeta: will automatically get tested when x2p method is completed and tests are written for x2p.

Also check bfb017a.

[SergeStinckwich]

Why do we check that aClass is a subclass of PMPrincipalComponentAnalyser ?
I would remove this test.

[AtharvaKhare]

Yes, I'll remove it.

[AtharvaKhare]

Resolved in 7c69a7f
(This commit will be squashed)

[SergeStinckwich]

Maybe we need to change the name X and be more explicit about its meaning in the t-SNE algorihm.

[AtharvaKhare]

Are you talking about instance variable x or the method name runPcaOnX?
We can:

1. Change x to input and change runPcaOnX to reduceInputToInputDims
2. Or change change runPcaOnX to reduceXToInputDims

[AtharvaKhare]

Changed in 49f3b2a
Is it better now?

[AtharvaKhare]

@SergeStinckwich regarding your two comments on the tests, I have:

1. Removed the obvious comment on tests
2. Removed self assert: pVector == pVectorRef. - I thought it was necessary, but it's not something that's needed.

[AtharvaKhare]

@SergeStinckwich
Please review, the algorithm is complete and has been tested on small examples, I will test on MNIST tomorrow and show the results.

[AtharvaKhare]

Here is one of the examples:
Input:

Output:

I don't know why a single point is out of place, it is replicable over different runs (the point is different over runs)

Edit: Not replicable anymore - reopened my image

[AtharvaKhare]

The speed is quite slow. Upon profiling, it looks like most of the time is spent on doing PMVector sum. Here is the profiler output:

 - 336067 tallies, 336072 msec.

**Tree**
--------------------------------
Process: (40s) Morphic UI Process: nil
--------------------------------
99.8% {335432ms} UndefinedObject>>DoIt
  99.8% {335432ms} PMTSNE>>start
    99.8% {335432ms} Job>>run
      99.8% {335432ms} BlockClosure>>ensure:
        99.8% {335428ms} Job>>run
          99.8% {335425ms} CurrentJob class(DynamicVariable class)>>value:during:
            99.8% {335425ms} CurrentJob(DynamicVariable)>>value:during:
              99.8% {335425ms} BlockClosure>>ensure:
                99.8% {335425ms} CurrentJob(DynamicVariable)>>value:during:
                  99.8% {335425ms} Job>>run
                    99.8% {335425ms} BlockClosure>>cull:
                      99.8% {335425ms} PMTSNE>>initializeJob
                        99.8% {335381ms} PMTSNE>>gradientDescent
                          99.0% {332802ms} PMTSNE>>computeGradient:withProgressUpdate:
                            93.5% {314071ms} PMMatrix>>sum
                              |93.4% {314036ms} PMVector>>sum
                              |  91.8% {308598ms} SmallFloat64>>+
                              |    |90.9% {305367ms} PMVector(Object)>>adaptToFloat:andSend:
                              |    |  90.8% {305162ms} PMVector(Collection)>>adaptToNumber:andSend:
                              |    |    90.7% {304947ms} PMVector(SequenceableCollection)>>collect:
                              |    |      51.2% {172224ms} PMVector(Collection)>>adaptToNumber:andSend:
                              |    |      39.5% {132723ms} primitives
                              |  1.2% {4178ms} SmallInteger>>+
                              |    1.2% {4161ms} SmallInteger(Integer)>>+
                              |      1.2% {4150ms} PMVector(Object)>>adaptToInteger:andSend:
                              |        1.2% {4114ms} PMVector(Collection)>>adaptToNumber:andSend:
                              |          1.2% {4105ms} PMVector(SequenceableCollection)>>collect:
                            2.3% {7676ms} PMTSNE>>computeLowDimensionalStudentT
                              |1.3% {4518ms} PMMatrix>>*
                              |  1.2% {3949ms} PMMatrix>>productWithMatrix:
                              |    1.2% {3949ms} PMMatrix>>productWithMatrixFinal:
                              |      1.2% {3918ms} PMMatrix>>rowsCollect:
                              |        1.2% {3918ms} PMVector(SequenceableCollection)>>collect:
                              |          1.2% {3916ms} PMMatrix>>productWithMatrixFinal:
                              |            1.2% {3914ms} PMMatrix>>rowsCollect:
                              |              1.2% {3905ms} PMVector(SequenceableCollection)>>collect:
                              |                1.1% {3676ms} PMMatrix>>productWithMatrixFinal:
                            1.0% {3398ms} PMMatrix>>rowsCollect:
                              1.0% {3390ms} PMVector(SequenceableCollection)>>collect:

**Leaves**
52.1% {175196ms} PMVector(Collection)>>adaptToNumber:andSend:
41.0% {137626ms} PMVector(SequenceableCollection)>>collect:

**Memory**
	old			-16,777,216 bytes
	young		+20,077,240 bytes
	used		+3,300,024 bytes
	free		-20,077,240 bytes

**GCs**
	full			1 totalling 141ms (0.0% uptime), avg 141.0ms
	incr		24140 totalling 5,099ms (2.0% uptime), avg 0.0ms
	tenures		40,165 (avg 0 GCs/tenure)
	root table	0 overflows

Any ideas for speedup of PMVector sum?