Hyperparameter optimization framework and sample

[fwaris]

Framework and sample to perform hyperparameter optimization with IAsyncSweeper wrapped KdoSweeper.

https://github.com/fwaris/HpOpt/tree/master/HpOptModelRunner

Allows for parallel evaluation of models via multiple client processes and named pipes communication.

However I could not use the NelderMeadSweeper as it runs into some issue with not having enough results returned (I think).

In general we need better documentation of the Microsoft.ML.Sweeper namespace.

[justinormont]

Quite nice!

Though as a general warning, the code base is in flux and we're looking to internalize as many components as we can before releasing as v1.0, the sweepers may or may not be accessible.

[fwaris]

thanks for the heads up. hopefully equivalent functionality will be available in 1.0.

Model architecture and hyper parameter optimization is the next level up. MS research has an interesting project - AutoML (https://www.microsoft.com/en-us/research/project/automl/) along these lines. Wonder if there are any plans to integrate AutoML with ML.Net

[TomFinley]

Hi @fwaris , @justinormont actually works on somethign called AutoML. The goal, as I understand it, is that the technology will be capable of operating with many frameworks, including ML.NET. I think they have a Github repo where they're working, but @justinormont might know better about that.

[fwaris]

@TomFinley thanks - good to know

[wschin]

As AutoML has entered a preview stage, I personally think we have enough information to close this issue. Don't hesitate to reopen it if I miss something. Thank you.

[justinormont]

@fwaris: The ML.NET AutoML code has since become public. Please contribute to it. For instance, it could use an external run context (out of process) similar to your project's sweeper.

[fwaris]

Thanks. I will switch to AutoML and explore that more.