[R] Refactor callback structure and attributes

[david-cortes]

ref #9810

To be merged after the following previous PRs which are dependencies here:

• [R] On-demand serialization + standardization of attributes #9924
• [R] Remove parameters and attributes related to ntree and rebase iterationrange #9935
• [R] Enable booster slicing #9948

This PR refactors the structure and logic of the callbacks in order to match those of the python interface, which allows more flexibility and feels a lot easier to manipulate.

After this PR, callbacks have their own environment which they can manipulate without affecting the environment of the training function call, and can execute functions at different stages of the process (before/after training and before/after round), with a signature for those functions which is now standardized and documented, and an example added there for creating custom user callbacks.

After this, the logic for the attributes of the booster object also changes, as now callbacks can leave arbitrary attributes in the booster through their finalizers which will be kept with the same name as the callback, and the names of some existing attributes are now moved to a sub-list within the name of the callback that produces them - e.g. before, the CV prediction left an entry pred, which now will be under cv_predict$pred instead.

Most of the code here is straight copy-paste from the previous callbacks, with some minor modifications:

• Possibility to slice the booster up to the best round when using early stopping.
• Changing the default format for saving model snapshots to JSON.
• Minor efficiency improvements like pre-allocating arrays for the expected size of the artifacts.

[trivialfis]

Thank you for working on the callback function interface! It's one of the most important parts of language bindings.

After briefly looking into the code, I'm more interested in the interface than the implementation details. Looking at the documents, I think some of the options and parameters are not necessary, it would be great if we could try to make the interface surface as small as possible. For instance, if some of the parameters are passed by the user in the first place, then maybe they can use closure to capture those parameters in callbacks.
In addition, some more design choices need to be made, for instance, less difference between CV and normal training would be great. XGBoost has a historically complex codebase and interface, it would be great if we could reduce the amount of conditional behaviors like "if x is used in y, then it's different than the behavior when using x in z"

[trivialfis]

Is this useful somehow other than pre-allocating a vector to store the evaluation history? These types of vectors/lists are quite cheap that don't need pre-allocation, and users who write callback usually know exactly the number of boosting rounds (they also write the training code). I think it's not necessary to have an additional parameter with a condition that we (and the user) need to check whether it matches the number of boosting rounds.

[david-cortes]

Yes, it is useful if e.g. one wants to:

• Create a custom step size calculator or otherwise modify some other parameter(s) accordingly.
• Target a certain number of snapshots during training instead of every N rounds or so.
• Create re-usable callbacks for xgboost in other packages.
• Create a different type of booster with an external optimizer like this guy did here.

Among others.

It's also useful when it comes to creating custom code to combine results of previous a callback with new results after an update (e.g. it would help to determine whether to append or replace).

[trivialfis]

Maybe we need an internal callback that does this and split it up with the early stopping? Along with the comment for evaluation history instead of actual data, see CallbackContainer in Python.

[david-cortes]

But if it's implemented as a callback, it'd still need to be executed before other callbacks.

[trivialfis]

yes, it will be a callback only for evaluation and storing the result. Users don't need to learn about its existence and we don't need to document it in public.

[david-cortes]

But isn't that how it's currently done? Only difference is that it's part of the public interface so the user can choose to pass it manually.

[david-cortes]

less difference between CV and normal training would be great

That's part of what this PR modifies: now in both cases, callbacks add attributes under their own (the callback's) name, and these attributes match between the two, with exceptions like the prediction-saving which is only for CV, or the coefficients history which saves one result per booster.

Or do you have some other type of changes in mind that could be adopted here?

it would be great if we could try to make the interface surface as small as possible

Yes, it feels like some things aren't really helpful, but for the most part, the functionalities here are also present in the python interface (e.g. python also has a LearningRateScheduler), so I didn't remove anything when switching to this new structure but perhaps removals could be done in subsequent PRs.

if some of the parameters are passed by the user in the first place

Most of these callbacks are added automatically based on arguments to xgb.train or xgb.cv. The only ones that are required to be passed by the user are the coefficients history and the 'reset parameters' callbacks.

[david-cortes]

For instance, if some of the parameters are passed by the user in the first place, then maybe they can use closure to capture those parameters in callbacks

One remark here: the user might know these parameters from passing them to xgb.train, but if someone wants to develop a reusable custom callback that could be simply added to the list the same way as xgboost's (e.g. to offer it as a package), then it'd be helpful to have things that might seem redundant, such as the evaluation data or the labels in the DMatrix.

[trivialfis]

then it'd be helpful to have things that might seem redundant, such as the evaluation data or the labels in the DMatrix.

That's a valid use case, but I think callbacks that need access to these data, especially data that don't change throughout the training session, can make them parameters of the constructor instead of the hook methods.

[david-cortes]

then it'd be helpful to have things that might seem redundant, such as the evaluation data or the labels in the DMatrix.

That's a valid use case, but I think callbacks that need access to these data, especially data that don't change throughout the training session, can make them parameters of the constructor instead of the hook methods.

Then there's cases like external-memory DMatrices, higher-level interfaces that might apply some processing on their own (e.g. encoding of categorical labels), etc.