I'm going to solve some convex optimization problems in this repository as I'm going through my learning path of this topic.

Optimization

Optimization problems are ubiquitous in statistics and machine learning. A huge number of problems that we consider in these disciplines can indeed be posed as optimization tasks.

Studying the details is important for two major reasons:

Different algorithms can perform (sometimes drastically) better or worse in different scenarios, and an understanding of why this happens requires an understanding of optimization

Often times, understanding a problem from the optimization perspective can contribute to our statistical understanding of the problem as well.

Half of the Machine Learning field is the construction of the learning method (optimization), there're many techniqques used for optimization if the optimization problem is solvable:

• Closed form solution ==> Least Mean squares
• Numerically and guaranteed ==> Conex optimization & Linear Programming
• Numerically but not guaranteed ==> Non-Convex, and can be solved using many techniques:
  • Numerical algorithms like Gradient descent.
  • Local optimization.
  • Heuristic swarm-based methods.
  • Brute force exhaustive search.