iterative gradient descent using theno

[sadaf2605]

Implemented Iterative Gradient Descent [2D] using theno, tested using sklearn, plots look good.

[coveralls]

Coverage decreased (-4.69%) when pulling 8b1ef17 on sadaf2605:batch-gradiant-descent-iterative into d5291df on MartinThoma:master.

[MartinThoma]

PEP8, please use Python 3 print function, and it seems not to work:

/usr/local/lib/python2.7/dist-packages/scipy/lib/_util.py:67: DeprecationWarning: Module scipy.linalg.blas.fblas is deprecated, use scipy.linalg.blas instead
  DeprecationWarning)
False
Traceback (most recent call last):
  File "test.py", line 41, in <module>
    theta = gradient_descent_2(alpha, x, y, 2000)
  File "test.py", line 21, in gradient_descent_2
    hypothesis = find_hypothesis(x, theta).T  # TODO: could not emplement it without reshape
  File "/usr/local/lib/python2.7/dist-packages/theano/compile/function_module.py", line 588, in __call__
    self.fn.thunks[self.fn.position_of_error])
  File "/usr/local/lib/python2.7/dist-packages/theano/compile/function_module.py", line 579, in __call__
    outputs = self.fn()
ValueError: Shape mismatch: x has 2 cols (and 10 rows) but y has 1 rows (and 2 cols)
Apply node that caused the error: Dot22(x, theta)
Inputs shapes: [(10, 2), (1, 2)]
Inputs strides: [(16, 8), (16, 8)]
Inputs types: [TensorType(float64, matrix), TensorType(float64, matrix)]
Use the Theano flag 'exception_verbosity=high' for a debugprint of this apply node.

[sadaf2605]

Opz, accidently I commented line 21.

[sadaf2605]

Did I violated PEP8 again? dammit! This time I was using sublime-text-2 and I will read the whole thing again!

[MartinThoma]

Copy your code to http://pep8online.com/ - then you will see that there are plenty of PEP8 violations.
I can recommend the "Python Checker" plugin for sublime text to check PEP8 automatically.

[sadaf2605]

I have uncommented the code! Can you give it a look? It works fine in my end...

[MartinThoma]

Could you please explain your code? It seems not to implement gradient descent. For example, where do you calculate the gradient?

The code seems to make only a few matrix operations (additions and dot products). What are your neurons? What represents hidden layers?

By the way, have you seen https://github.com/MartinThoma/nntoolkit/blob/master/nntoolkit/train.py ?

[MartinThoma]

The image and the code looks rather as if you implemented linear regression.

[coveralls]

Coverage decreased (-4.88%) when pulling ac901ea on sadaf2605:batch-gradiant-descent-iterative into d5291df on MartinThoma:master.

[sadaf2605]

OOpz, in issue 4 (#4) you were talking about "batch gradient descent" so I thaught you were talking about http://en.wikipedia.org/wiki/Stochastic_gradient_descent. Probably I have mistaken something then :/

[MartinThoma]

There are three variants of gradient descent:

• Batch gradient descent: Takes all training examples into consideration, then improves the weights
• Mini-batch gradient descent: Takes a couple (typically 256) training examples into consideration, then improves the weights
• Stochastic gradient descent: Improves the weights after every single training example.

I want to implement mini-batch gradient descent, as you can change a parameter and get both other variants.

You implementations seems not to be able to be generizable to fit higher-order function (higher than linear) to data. Am I wrong?
Neural networks are typically very high dimensional functions (e.g. R^160 -> R^369).

[sadaf2605]

Alright leave it here. I think I messed up :P

[sadaf2605]

I mean it is gradient descent regression, you were looking for classification [for nn]. Alright, I also need to read about nn, I will get back after that.