XGBoost is the state of the art machine learning algorithm. So to get a good understanding of what is going under the hood of the XGBoost I implemented it from scratch using only Numpy.
While implementing the gradient boosting I realize that it's similar to gradient descent in neural networks, so rather than just doing simple gradient descent, I tried using ADAM and RMSProp Optimizers and to my surprise, the results improved compared to Simple XGBoost.
The comparison between algorithms is given inside Comparision.ipynb and all the code for CustomXGBoost is given in CustomXGBoost.py
CustomXGBoost has 4 main Classes
- XGBRegressor - for regression problem
- XGBRegressorAdam - XGBoost regressor with ADAM Optimizer
- XGBRegressorRMS - XGBoost regressor with RMSProp Optimizer
- XGBClassifier - for multiclass classification problem
implementational example
custom = XGBRegressor() # same for XGBRegressorAdam and XGBRegressorRMS
custom.fit(X_train, y_train, eval_set = (X_test, y_test))
y_pred = custom.predict(X_test.values)
custom = XGBClassifier(n_classes=5) # n_classes is total diiferent labels of target
custom.fit(X_train, y_train)
y_pred = custom.predict(X_test)
All the other default values for parameters are as given in XGBoost official Documentation.