Reproducibility of distributed model building

[kiran-vj]

Hi team,

I currently am trying to use XGBoost for training certain models in a distributed manner (Dask based) using GPUs.

As per my understanding achieving identical models is possible only by restricting usage to an identical GPU config, setting seeds etc.

I wanted to understand from a HPT perspective- would the optimal model achieved via Hyperparameter Tuning on 2 different GPU configurations atleast be comparable to a certain extent - (ignoring edge cases where HPT gets stuck in a particular local maxima etc.) ?

Or does cluster configuration need to be factored in at some level which complicates things

[trivialfis]

Yes, they are comparable. The differences come from floating point errors and data splits (histogram bin construction/quantile sketching). The underlying algorithm is the same regardless of the environment.