Format of Machine Learning API

[Kelvin-Ng]

The content is put to https://github.com/husky-team/husky/wiki/TODO:-Format-of-Machine-Learning-API

I have thought of several choices, but I don't have a clue of which one is absolutely better. Currently my preference is in descending order. Discussion may help for deciding the final choice. The current machine learning library is in quite a mess, so this is quite important.

[Kelvin-Ng]

Performance issue is added.

[ddmbr]

Somehow I feel this design is limited to gradient-based method only? I think all first-order gradient-based methods fit into this class. Is this your intention?

If so, we can actually create different type traits for different classes of problems.

[Kelvin-Ng]

I said that this is for example only.

For genetic algorithm, for example again, the Problem class should contain public methods of mutation(), crossover(), etc. I was only going to explain what kind of methods should be included in the classes, not exactly what methods should be included.

If we implement genetic algorithm later, we need to add mutation(), crossover(), etc, to the Problem classes existing in the husky lib so that users can choose to use gradient-based methods or genetic algorithm.

[TatianaJin]

Your idea is really inspiring. I am very interested in how to generalize the design to non-gradient-based methods. For example, to me the optimization problem for k-means like algorithms has a quite specific logic to find the optimum. My current work (yes they are messy especially the svm one) represents the optimization problem by the gradient function in models and let optimizers handle which samples to use and the update sequence. And a model assigns its problem to an optimizer. I do not know how to extend this to other optimization methods... Could you explain more on mutation() and crossover() (forgive my ignorance)?

And I think the optimization problems are only part of a model? For some machine learning models the inference parts are the same while the optimization part can be alternated. And I considered them as the same model with different formulations.

Also, if considering neural network (I am not sure whether you think this belongs to the same topic), a model may contain many 'submodels' if each neuron is considered a model according to its activation function, and each activation function corresponds to a type of optimization problem. It appears to me the whole machine learning algorithm takes the form: a model assigns the optimization problems to different types of optimizers and uses the solution to do inference.

[lijinf2]

@Kelvin-Ng I agree with you. Yuzhen and me once developed a general framework for genetic algorithms for AI course, and it works. For each class of machine learning algorithms, I believe there should be a general, efficient and user-friendly framework, which can be built upon Husky. But there is no such framework that is general, efficient and user-friendly to all machine learning algorithms. I think any of the three methods you mentioned is reasonable and no one is always better than the other two. Why not we first determine what class of ml algorithms we want to support, and then consider which of the three method is the best.

[Kelvin-Ng]

@TatianaJin

Your last sentence precisely and concisely summarize what machine learning is basically doing.

What you say is very correct. Sometimes one machine learning model can be optimized in different ways by modelling it as different optimization problems (e.g. L1 vs L2 regularized linear regression), or by using different optimization algorithms on the same model and same optimization problem (e.g. L2 regularized linear regression using FGD, SGD, SVRG or genetic algorithm). The best design should be able to separate all of these so that models, optimization problems and optimization algorithms can be combined in different ways flexibly.

(An optimization problem is tightly tied to a model, but a model can be formulated as different optimization problems.)

[Kelvin-Ng]

The wiki page is updated with a new section. Thanks @TatianaJin for inspiring me.