New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Added incremental building of ensembles and fixed adaboost #138
Conversation
I merged this last night with master in my local repo so should be a straightforward diff, although there are lots of commits listed here. |
Thank you. I will get this merged in soon! |
I am holding on merging this change for now. There are quite a few potentially breaking changes to core Encog classes. Mostly adding the dropout rate the the update weight. This function is called quite a bit in multi-threaded code. Generating a random number for every weight update, even when we have no dropout rate really drags performance. Additionally, the code is creating a new random number generator for each weight update. I like the changes overall, but allocating a new random number generator for every weight update is good for performance. |
Those are valid points. I think some of those are covered in some of my
|
I just pushed a few changes that address the problems you mentioned. You may also want to ignore the changes to DetermineWorkload.java - I only did them in my repo because I got a small speed gain when using a large cluster, but this may very well not be the case everywhere. If it makes things easy I can do an upstream pull as well, since it's been a while. |
Thank you very much for adding this code, and sorry the merge has taken so long. As soon as I finish my current book (August) getting this merged into Encog is my next priority. I am sure I will have some questions. |
No worries Jeff. I've been finding bugs here and there so it's been giving
|
Okay that sounds good. |
One more question. I am submitting an article to the open source track of the Journal of Machine Learning Research (JRML) for Encog. Your ensemble code is one of the larger blocks of contributed code, so I plan to list you in the acknowledgements (assuming that is okay). Do you have a paper, or similar, that you would like me to cite? I think your code will be very useful to the Encog community once we get it fully integrated, documented, exampled, and hooked into the Workbench. This is what I plan to help with soon. |
It is absolutely okay to list me in the acknowledgements. I have a paper @INPROCEEDINGS{Mosc1409:Empirical, @INPROCEEDINGS{Mosca:Empirical, @mastersthesis{Mosca:mscdissertation, On 2 July 2014 15:21, Jeff Heaton notifications@github.com wrote:
Alan |
Thank you, the bibtex is useful. I added a cite to your master's thesis. I also list you (with affiliation) in the acknowledgments section. I've not worked with dropout or ensemble for neural networks yet, and mostly understand them in concept. I am looking forward to adding this to Encog, and learning more myself. Thank you for you willingness to add this to Encog. Best of luck with your paper, and I will let you know how the Encog paper with JMLR turns out. |
Thank you, that's very kind. I have been using Encog for the experiments On 5 July 2014 18:35, Jeff Heaton notifications@github.com wrote:
Alan |
Hi Jeff, I noticed that some of the new Ensemble classes have appeared in the main repo. I don't know if you are midway through the merge or have managed to get everything to work decently, but please do let me know if there's anything I can do to help. |
I did add a few classes from an earlier attempt to merge it. I hope to release Encog 3.4 before the end of the year. It would be great to get your code added to 3.4. What would be the best way? Have you added/fixed anything beyond what is currently in this pull request? If it would not be too much trouble for you to update to the current 3.3 code and then send another pull request, I would be glad to review any compatibility issues where your code intersects Encog and adjust as needed. |
That sounds like a good idea. I ended up making the pull request from HEAD
|
.DS_Store | ||
.gradle | ||
>>>>>>> refs/remotes/choose_remote_name/master |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Looks like unresolved conflict
- use the correct layer index for dropoutRates in MLP labels
I finally merged this in, very sorry about the long delay on this one. I've actually gotten into some of the technology you've added, so this is very interesting for me to go through this. I am going to experiment with my existing Encog code and make sure everything works as I expect, and will make any changes as needed. |
Thanks, do let me know if anything is unclear or looks wrong! On 7 July 2015 at 18:29, Jeff Heaton notifications@github.com wrote:
Alan |
Just one small thing I noticed, the FlatNetwork does have a default constructor that is used sometimes with serialization. It does not call init, so the dropout percentage variable was not getting set and causing NullPointerException's later on. The following very simple fix resolves that: Otherwise all of the previous code seems to be functioning as I would expect! Thanks again! I am sure I will have more questions as I go through it. |
FYI, the paper with my citation to your thesis for this has been published. http://jmlr.org/papers/volume16/heaton15a/heaton15a.pdf |
Adaboost implementation had a bug which resulted in the D vector of dataset weights being incorrect and in some extreme cases leading to non-termination and infinity errors.
Ensembles now can have "steps" where they have a member added on to the existing ensemble. This only works for some ensemble techniques that can be expanded this way.