What does this training and validation loss mean?

[lesshaste]

I am using HistGradientBoostingClassifier with verbose=2. I see:

Binning 0.001 GB of training data: 0.037 s
Binning 0.000 GB of validation data: 0.025 s
Fitting gradient boosted rounds:
[1/100] 26 trees, 358 leaves (13 on avg), max depth = 4, train loss: 2.77475, val loss: 2.90723, in 1.306s
[2/100] 26 trees, 401 leaves (15 on avg), max depth = 4, train loss: 2.60734, val loss: 2.59786, in 1.364s
[3/100] 26 trees, 402 leaves (15 on avg), max depth = 4, train loss: 2.61666, val loss: 2.75812, in 2.097s
[4/100] 26 trees, 400 leaves (15 on avg), max depth = 4, train loss: 2.50390, val loss: 2.50473, in 1.920s
[5/100] 26 trees, 396 leaves (15 on avg), max depth = 4, train loss: 2.56797, val loss: 2.57659, in 1.402s
[6/100] 26 trees, 397 leaves (15 on avg), max depth = 4, train loss: 2.43553, val loss: 2.35564, in 1.289s
[7/100] 26 trees, 393 leaves (15 on avg), max depth = 4, train loss: 2.55682, val loss: 2.45582, in 1.672s
[8/100] 26 trees, 400 leaves (15 on avg), max depth = 4, train loss: 2.54803, val loss: 2.70659, in 1.083s
[9/100] 26 trees, 401 leaves (15 on avg), max depth = 4, train loss: 2.47577, val loss: 2.25496, in 1.412s
[10/100] 26 trees, 402 leaves (15 on avg), max depth = 4, train loss: 2.66319, val loss: 2.41011, in 1.808s
[11/100] 26 trees, 390 leaves (15 on avg), max depth = 4, train loss: 2.68041, val loss: 2.49166, in 1.321s
[12/100] 26 trees, 400 leaves (15 on avg), max depth = 4, train loss: 2.66118, val loss: 2.18302, in 0.881s
[13/100] 26 trees, 395 leaves (15 on avg), max depth = 4, train loss: 2.94769, val loss: 2.47838, in 1.998s
[14/100] 26 trees, 397 leaves (15 on avg), max depth = 4, train loss: 2.85530, val loss: 2.14537, in 1.243s
[15/100] 26 trees, 389 leaves (14 on avg), max depth = 4, train loss: 3.27165, val loss: 2.73873, in 1.602s
[16/100] 26 trees, 399 leaves (15 on avg), max depth = 4, train loss: 3.29203, val loss: 2.11170, in 2.547s
[17/100] 26 trees, 402 leaves (15 on avg), max depth = 4, train loss: 3.85362, val loss: 2.72681, in 1.478s
[18/100] 26 trees, 388 leaves (14 on avg), max depth = 4, train loss: 3.94590, val loss: 2.08426, in 1.676s
[19/100] 26 trees, 400 leaves (15 on avg), max depth = 4, train loss: 4.57334, val loss: 2.56780, in 1.430s
[20/100] 26 trees, 398 leaves (15 on avg), max depth = 4, train loss: 4.41104, val loss: 2.05993, in 1.236s
[21/100] 26 trees, 395 leaves (15 on avg), max depth = 4, train loss: 5.13700, val loss: 2.77980, in 1.491s
[22/100] 26 trees, 401 leaves (15 on avg), max depth = 4, train loss: 4.96310, val loss: 2.03746, in 1.592s
[23/100] 26 trees, 388 leaves (14 on avg), max depth = 4, train loss: 5.95392, val loss: 3.24908, in 1.443s
[24/100] 26 trees, 394 leaves (15 on avg), max depth = 4, train loss: 6.36990, val loss: 2.01593, in 2.376s
[25/100] 26 trees, 387 leaves (14 on avg), max depth = 4, train loss: 7.43258, val loss: 2.93428, in 2.024s
[26/100] 26 trees, 399 leaves (15 on avg), max depth = 4, train loss: 7.83435, val loss: 1.99510, in 1.124s
[27/100] 26 trees, 366 leaves (14 on avg), max depth = 4, train loss: 9.13146, val loss: 3.65188, in 2.450s
[28/100] 26 trees, 380 leaves (14 on avg), max depth = 4, train loss: 8.70324, val loss: 1.97605, in 1.753s
[29/100] 26 trees, 383 leaves (14 on avg), max depth = 4, train loss: 10.47592, val loss: 4.65343, in 1.265s
[30/100] 26 trees, 375 leaves (14 on avg), max depth = 4, train loss: 9.58694, val loss: 2.86254, in 1.281s
[31/100] 26 trees, 388 leaves (14 on avg), max depth = 4, train loss: 16.61555, val loss: 9.45004, in 1.417s
[32/100] 26 trees, 343 leaves (13 on avg), max depth = 4, train loss: 18.67819, val loss: 9.75153, in 1.037s
[33/100] 26 trees, 335 leaves (12 on avg), max depth = 4, train loss: 20.03424, val loss: 11.71814, in 1.871s
[34/100] 26 trees, 341 leaves (13 on avg), max depth = 4, train loss: 22.84089, val loss: 13.55724, in 1.503s
[35/100] 26 trees, 341 leaves (13 on avg), max depth = 4, train loss: 198.27027, val loss: 195.79159, in 0.768s
[36/100] 26 trees, 325 leaves (12 on avg), max depth = 4, train loss: 354.63563, val loss: 244.03380, in 1.444s
[37/100] 26 trees, 305 leaves (11 on avg), max depth = 4, train loss: 359.33633, val loss: 250.24355, in 1.280s
[38/100] 26 trees, 335 leaves (12 on avg), max depth = 4, train loss: 353.98696, val loss: 243.83921, in 0.883s

What does the huge increase in in train and val loss mean?

[ogrisel]

This is probably a bug that was already discussed here: #11452 (comment) but as far as I know, nobody found the time to investigate and identify the root cause.

Can you try to see if alternative implementations such as xgboost, lightgbm or catboost show a similar behavior on the same dataset with similar hyperparameters?

A workaround would be to enable the early stopping option.

[NicolasHug]

Could it just be that the learning rate is too high?

[lesshaste]

Setting learning_rate=0.04 does get rid of the problem it seems. Is the default just too high? My features are categorical if that makes a difference.

[lesshaste]

I spoke too soon. learning_rate=0.04 failed for the next data set. I am trying learning_rate=0.02 which seems to be working better. Could there be a way to set this learning_rate dynamically?

[NicolasHug]

The learning rate is probably the most important parameter to tune and it's dataset-dependent so the behaviour in your original post doesn't seem too surprising to me. @ogrisel are you sure that this and #11452 are really that unexpected?