New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

support DART - new regularization = dropout trees during learning #809

Closed
gugatr0n1c opened this Issue Feb 7, 2016 · 24 comments

Comments

Projects
None yet
5 participants
@gugatr0n1c

gugatr0n1c commented Feb 7, 2016

There is nice article about dropout from neural nets, applied to gradient boosting:

http://arxiv.org/pdf/1505.01866.pdf

It is about drop out some trees during learning process and rescale weight of trees because of that...
It can help to accuracy...

Thanx for consideration...

@tqchen

This comment has been minimized.

Show comment
Hide comment
@tqchen

tqchen Feb 10, 2016

Member

This should not be too hard to add by adding a new gradient booster, or extending current gbtree.

Member

tqchen commented Feb 10, 2016

This should not be too hard to add by adding a new gradient booster, or extending current gbtree.

@Far0n

This comment has been minimized.

Show comment
Hide comment
@Far0n

Far0n Mar 8, 2016

Contributor

I would suggest:

  • extending gbtree
  • new parameter 'dart_dropout', 'dart_p' or something in that direction (float, range [0,1], default: 0)

thoughts?

Contributor

Far0n commented Mar 8, 2016

I would suggest:

  • extending gbtree
  • new parameter 'dart_dropout', 'dart_p' or something in that direction (float, range [0,1], default: 0)

thoughts?

@marugari

This comment has been minimized.

Show comment
Hide comment
@marugari

marugari Apr 17, 2016

Contributor

I've created a prototype.(not tested yet)
https://github.com/marugari/xgboost/tree/prototype_dart

Contributor

marugari commented Apr 17, 2016

I've created a prototype.(not tested yet)
https://github.com/marugari/xgboost/tree/prototype_dart

@tqchen

This comment has been minimized.

Show comment
Hide comment
@tqchen

tqchen Apr 17, 2016

Member

@marugari It would be great if you can run some benchmarks and see the performance, we can look into bring it back to main repo

Member

tqchen commented Apr 17, 2016

@marugari It would be great if you can run some benchmarks and see the performance, we can look into bring it back to main repo

@marugari

This comment has been minimized.

Show comment
Hide comment
@marugari
Contributor

marugari commented Apr 20, 2016

@tqchen

This comment has been minimized.

Show comment
Hide comment
@tqchen

tqchen Apr 20, 2016

Member

This sounds interesting. @marugari Can we also refactored the code a bit?

Since I guess most part of dart could reuse the code of GBTree, and only prediction function needs to be replace. Let us consider use inheritance or a common base code, so the code is cleaner and easier to maintain.

Then I am happy to take a review and bring it to xgboost main repo

Member

tqchen commented Apr 20, 2016

This sounds interesting. @marugari Can we also refactored the code a bit?

Since I guess most part of dart could reuse the code of GBTree, and only prediction function needs to be replace. Let us consider use inheritance or a common base code, so the code is cleaner and easier to maintain.

Then I am happy to take a review and bring it to xgboost main repo

@marugari

This comment has been minimized.

Show comment
Hide comment
@marugari

marugari Apr 20, 2016

Contributor

@tqchen Sure.
I also think Dart shold inherit from GBTree but could not fix duplicate symbols. 😢

Contributor

marugari commented Apr 20, 2016

@tqchen Sure.
I also think Dart shold inherit from GBTree but could not fix duplicate symbols. 😢

@tqchen

This comment has been minimized.

Show comment
Hide comment
@tqchen

tqchen Apr 20, 2016

Member

If there is not too much code, we can put two classes both in gbm.cc which should solve the problem

Member

tqchen commented Apr 20, 2016

If there is not too much code, we can put two classes both in gbm.cc which should solve the problem

@marugari

This comment has been minimized.

Show comment
Hide comment
@marugari

marugari Apr 20, 2016

Contributor

@tqchen If it is agreeable, I can refact the code.
The following methods are defined in Dart.

  • Configure
  • Load
  • Save
  • DoBoost
  • Predict
  • CommitModel
  • Pred
  • DropTrees
  • NormalizeTrees
Contributor

marugari commented Apr 20, 2016

@tqchen If it is agreeable, I can refact the code.
The following methods are defined in Dart.

  • Configure
  • Load
  • Save
  • DoBoost
  • Predict
  • CommitModel
  • Pred
  • DropTrees
  • NormalizeTrees
@tqchen

This comment has been minimized.

Show comment
Hide comment
@tqchen

tqchen Apr 20, 2016

Member

Sounds good.

Member

tqchen commented Apr 20, 2016

Sounds good.

@marugari

This comment has been minimized.

Show comment
Hide comment
@marugari

marugari Apr 23, 2016

Contributor

How's this?
marugari@d4e3a6b

Contributor

marugari commented Apr 23, 2016

How's this?
marugari@d4e3a6b

@Far0n

This comment has been minimized.

Show comment
Hide comment
@Far0n

Far0n Apr 23, 2016

Contributor

great work @marugari. thank you for bringing DART in.

Contributor

Far0n commented Apr 23, 2016

great work @marugari. thank you for bringing DART in.

@tqchen

This comment has been minimized.

Show comment
Hide comment
@tqchen

tqchen Apr 23, 2016

Member

@marugari This looks good, can you

  • rebase your commits so the history of the old dart.cc is removed, another simpler way might be start from a clean fork
  • Add a few comments on what dart is, and the algorithms
  • Open a pull request and I will do more detailed code review in the PR
  • It is likely that you can reuse a few functions, like SaveModel, or change the parent function to isolate the common parts and further reduce the code.
  • To prevent the codebase from getting too large, let us avoid ipynb in the main repo. Use markdown and python scripts instead. You can always put ipynb to https://github.com/dmlc/web-data
  • Please write an markdown introduction in documents on how dart can be used.

Thanks for the great effort

Member

tqchen commented Apr 23, 2016

@marugari This looks good, can you

  • rebase your commits so the history of the old dart.cc is removed, another simpler way might be start from a clean fork
  • Add a few comments on what dart is, and the algorithms
  • Open a pull request and I will do more detailed code review in the PR
  • It is likely that you can reuse a few functions, like SaveModel, or change the parent function to isolate the common parts and further reduce the code.
  • To prevent the codebase from getting too large, let us avoid ipynb in the main repo. Use markdown and python scripts instead. You can always put ipynb to https://github.com/dmlc/web-data
  • Please write an markdown introduction in documents on how dart can be used.

Thanks for the great effort

@marugari

This comment has been minimized.

Show comment
Hide comment
@marugari

marugari Jun 12, 2016

Contributor

It has been merged. 949d1e3

The tutorial is under construction.
https://github.com/marugari/xgboost/blob/prototype_dart/doc/tutorials/dart.md

Contributor

marugari commented Jun 12, 2016

It has been merged. 949d1e3

The tutorial is under construction.
https://github.com/marugari/xgboost/blob/prototype_dart/doc/tutorials/dart.md

@tqchen

This comment has been minimized.

Show comment
Hide comment
@tqchen

tqchen Jun 12, 2016

Member

great:)

Member

tqchen commented Jun 12, 2016

great:)

@tqchen

This comment has been minimized.

Show comment
Hide comment
@tqchen

tqchen Jun 24, 2016

Member

@marugari any updates on english version of guest blogpost?

Member

tqchen commented Jun 24, 2016

@marugari any updates on english version of guest blogpost?

@marugari

This comment has been minimized.

Show comment
Hide comment
@tqchen

This comment has been minimized.

Show comment
Hide comment
@tqchen

tqchen Jun 25, 2016

Member

looks nice,

Member

tqchen commented Jun 25, 2016

looks nice,

@marugari

This comment has been minimized.

Show comment
Hide comment
@marugari

marugari Jun 27, 2016

Contributor

Can I use MathJax in dmlc.github.io?

Contributor

marugari commented Jun 27, 2016

Can I use MathJax in dmlc.github.io?

@tqchen

This comment has been minimized.

Show comment
Hide comment
@tqchen

tqchen Jun 27, 2016

Member

seems not, maybe use images for formula for safety

Member

tqchen commented Jun 27, 2016

seems not, maybe use images for formula for safety

@tqchen

This comment has been minimized.

Show comment
Hide comment
@tqchen

tqchen Jul 4, 2016

Member

Link to the post http://dmlc.ml/xgboost/2016/07/02/support-dropout-on-xgboost.html
Thanks for the great effort!

One last thing, let us PR a copy of tutorial to https://github.com/dmlc/xgboost/tree/master/doc/tutorials so it can benefit users who look into the document for recipes

Member

tqchen commented Jul 4, 2016

Link to the post http://dmlc.ml/xgboost/2016/07/02/support-dropout-on-xgboost.html
Thanks for the great effort!

One last thing, let us PR a copy of tutorial to https://github.com/dmlc/xgboost/tree/master/doc/tutorials so it can benefit users who look into the document for recipes

@marugari

This comment has been minimized.

Show comment
Hide comment
@marugari

marugari Jul 5, 2016

Contributor

I'm fixing formulas.

Contributor

marugari commented Jul 5, 2016

I'm fixing formulas.

@edmondja

This comment has been minimized.

Show comment
Hide comment
@edmondja

edmondja Jul 6, 2016

param = {'booster': 'dart',
'max_depth': 5, 'learning_rate': 0.1,
'objective': 'multi:softmax', 'silent': True,
'sample_type': 'uniform',
'normalize_type': 'tree',
'rate_drop': 0.1,
'skip_drop': 0.5}
num_round = 50
bst = xgb.train(param, xg_train, num_round)
Traceback (most recent call last):

File "", line 15, in
bst = xgb.train(param, xg_train, num_round)

File "//anaconda/lib/python2.7/site-packages/xgboost/training.py", line 121, in train
bst.update(dtrain, i, obj)

File "//anaconda/lib/python2.7/site-packages/xgboost/core.py", line 694, in update
_check_call(_LIB.XGBoosterUpdateOneIter(self.handle, iteration, dtrain.handle))

File "//anaconda/lib/python2.7/site-packages/xgboost/core.py", line 97, in _check_call
raise XGBoostError(_LIB.XGBGetLastError())

XGBoostError: unknown booster type: dart

Isnt it implemented in XGBoost ? I don't get it

edmondja commented Jul 6, 2016

param = {'booster': 'dart',
'max_depth': 5, 'learning_rate': 0.1,
'objective': 'multi:softmax', 'silent': True,
'sample_type': 'uniform',
'normalize_type': 'tree',
'rate_drop': 0.1,
'skip_drop': 0.5}
num_round = 50
bst = xgb.train(param, xg_train, num_round)
Traceback (most recent call last):

File "", line 15, in
bst = xgb.train(param, xg_train, num_round)

File "//anaconda/lib/python2.7/site-packages/xgboost/training.py", line 121, in train
bst.update(dtrain, i, obj)

File "//anaconda/lib/python2.7/site-packages/xgboost/core.py", line 694, in update
_check_call(_LIB.XGBoosterUpdateOneIter(self.handle, iteration, dtrain.handle))

File "//anaconda/lib/python2.7/site-packages/xgboost/core.py", line 97, in _check_call
raise XGBoostError(_LIB.XGBGetLastError())

XGBoostError: unknown booster type: dart

Isnt it implemented in XGBoost ? I don't get it

@marugari

This comment has been minimized.

Show comment
Hide comment
@marugari

marugari Jul 7, 2016

Contributor

Last version release does not support DART.

Please refer to gbtree.cc(or gbtree-inl.hpp).

Contributor

marugari commented Jul 7, 2016

Last version release does not support DART.

Please refer to gbtree.cc(or gbtree-inl.hpp).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment