Join GitHub today
GitHub is home to over 28 million developers working together to host and review code, manage projects, and build software together.Sign up
Implementation of Stochastic Coordinate Descent #1075
Here is an implementation of another optimizer, SCD. I have started with a serial implementation first, and made some changes to
Currently, I am working on getting two things done :
I'll update soon with more changes.
Looks good to me. I do think we need a formalized write-up of all the different
FunctionTypes that we have now and what the differences between them are. Would you be willing to sketch up some documentation for that? I figured it could be look the documentation in
Another question I have is, are you planning on making any of the other functions (like softmax regression) into
And last question for now, have you done any timing simulations to compare SCD with, e.g., LBFGS or SGD for logistic regression? I am excited to see how it performs. :)
Only minor comments from me.
This looks good. I am glad you went into the testing rabbit hole with HOGWILD!, now testing this feels more straightforward.
I finished up the changes in
I guess it would be nice to make this uniform across all the functions (SCD could then do the update on the relevant column instead of working on the entire decision variable), also easing up the parallelisation?
I am working on the changes in Logistic regression (using a
Again only minor comments.
Thanks for normalizing the logistic regression interface as well, that was a good catch :)
Looks great to me. Thank you so much for adding the tutorial on FunctionType parameters, I think it is a great improvement. These are the last comments I have; I think they are all pretty minor. From my end the PR is good to go if you can address them. Great work!
@shikharbhardwaj: thanks for the responses, everything looks good to me. I think there are still two comments worth addressing---
Sure, I had added tests for the descent policies (