I have developed the gradient descent from scratch in Python
Check out the theory part in below articles : https://www.linkedin.com/pulse/gradient-decent-algorithm-machine-learning-mayur-ingole
Gradient Decent algorithm is based on calculus theory and is one of the most commonly used optimization algorithm to train the machine and deep learning model by minimizing error between actual output and predicted output by model. In mathematical term, Optimization algorithm aim is to minimizing/maximizing an objective function f(x) parameterized by x(using derivative). Similarly, in machine learning , optimization is the task of minimizing the cost function parameterized by the parameter. The main objective of gradient descent algorithm is to minimize the error function using iteration of parameter updates and minimize the loss.