Skip to content

Loss Functions

Jump to bottom
Ivan Svetunkov edited this page May 27, 2026 · 7 revisions

Loss Functions

This page documents the loss parameter used across smooth functions for model estimation.

Read more about them in Chapter 11 of Svetunkov (2023).

The loss Parameter

The loss parameter specifies the loss (or cost) function minimized during model estimation.

Basic Usage

# R: Maximum likelihood estimation (default)
model <- adam(y, model="AAA", lags=12, loss="likelihood")

# R: Mean Squared Error
model <- adam(y, model="AAA", lags=12, loss="MSE")
# Python: Maximum likelihood (default)
model = ADAM(model="AAA", lags=12, loss="likelihood")

# Python: Mean Squared Error
model = ADAM(model="AAA", lags=12, loss="MSE")

Available Loss Functions

Standard Loss Functions

Loss Full Name Description Formula
"likelihood" Maximum Likelihood Default. Uses the distribution specified in distribution parameter Depends on distribution
"MSE" Mean Squared Error Minimizes average squared errors mean((y - fitted)^2)
"MAE" Mean Absolute Error Minimizes average absolute errors, robust to outliers mean(abs(y - fitted))
"HAM" Half Absolute Moment Focuses on small errors, super robust to outliers mean(sqrt(abs(y - fitted)))

Regularization Loss Functions

Loss Description Use Case
"LASSO" L1 regularization Shrink parameters toward zero, variable selection
"RIDGE" L2 regularization Shrink parameters, prevent overfitting

Read paper of Pritularga et al. (2023) to understand what LASSO/RIDGE mean in case of dynamic models.

When using LASSO/RIDGE, the lambda parameter controls regularization strength and can be passed via ...:

model <- adam(y, model="AAA", lags=12, loss="LASSO", lambda=0.1)
model = ADAM(model="AAA", lags=12, loss="LASSO", lambda_param=0.1)

lambda parameter is restricted with (0, 1) region, defining whether to give more weight to MSE or to the parameter shrinkage (closer to 1 => higher shrinkage).

Multi-step Loss Functions

Multi-step loss functions optimize the model based on forecast errors at horizon h, not just one-step-ahead errors. The h parameter must be specified.

Loss Full Name Description
"MSEh" h-step MSE Only uses h-steps ahead forecast error
"TMSE" Trace MSE Sum of MSE for horizons 1 to h
"GTMSE" Geometric Trace MSE Geometric mean of MSE across horizons
"MSCE" Mean Squared Cumulative Error MSE of cumulative forecasts

Read more about multistep losses and their effect on models in Svetunkov et al. (2023a).

# R: Optimize for 12-step ahead forecasts
model <- adam(y, model="AAA", lags=12, loss="GTMSE", h=12)
# Python: Multi-step optimization
model = ADAM(model="AAA", lags=12, loss="GTMSE", h=12)

Absolute and Half Analogues

For completeness, absolute and half-moment versions of multi-step losses exist:

Squared Absolute Half
MSEh MAEh HAMh
TMSE TMAE THAM
GTMSE GTMAE GTHAM
MSCE MACE CHAM

Custom Loss Functions

You can provide your own loss function. It must accept three parameters:

  • actual: Vector of actual values
  • fitted: Vector of fitted values
  • B: Vector of all estimated parameters

R Example

# Custom loss: Mean Absolute Percentage Error
customLoss <- function(actual, fitted, B) {
    return(mean(abs((actual - fitted) / actual)))
}

model <- adam(y, model="AAA", lags=12, loss=customLoss)

Python Example

import numpy as np

# Custom loss: Mean Absolute Percentage Error
def custom_loss(actual, fitted, B):
    return np.mean(np.abs((actual - fitted) / actual))

model = ADAM(model="AAA", lags=12, loss=custom_loss)

The same (actual, fitted, B) → scalar callable interface is accepted by OM / OMG / om() / omg(). In an occurrence model context, actual is the binary 0/1 indicator and fitted is the predicted probability p_t ∈ (0, 1).

Loss functions for OM and OMG

R om() and omg() plus Python OM and OMG accept the same single- step menu as ADAM minus the multi-step losses (which are meaningless for a binary Bernoulli target):

Loss OM / OMG Description
"likelihood" ✓ Bernoulli log-likelihood on the predicted probability (default)
"MSE" ✓ Mean squared (ot - p_t)
"MAE" ✓ Mean absolute (ot - p_t)
"HAM" ✓ Mean √absolute (ot - p_t)
"LASSO" ✓ (1 - λ) · sqrt(mean(errors²)) + λ · sum(|B|) — L1 on the parameter vector
"RIDGE" ✓ (1 - λ) · sqrt(mean(errors²)) + λ · sqrt(sum(B²)) — L2 on the parameter vector
custom callable ✓ User function (actual, fitted, B) → scalar
multistep (MSEh/TMSE/GTMSE/MSCE/GPL) ✗ Not supported — single-step only

For LASSO/RIDGE pass lambda (R) / reg_lambda (Python) to control the penalty weight (0 → unregularised, 1 → pure penalty). The penalty mirrors adam() exactly: it acts on the full parameter vector B. For OMG, B is the joint vector concat(B_A, B_B) so the penalty is shared across both sub-models.

# R
om(y, model="MNN", occurrence="odds-ratio", loss="LASSO", lambda=0.3)
omg(y, modelA="ANN", modelB="ANN", loss="RIDGE", lambda=0.3)
omg(y, modelA="ANN", modelB="ANN",
    loss=function(actual, fitted, B) sum(abs(actual - fitted)^3))
# Python
OM(model="MNN", occurrence="odds-ratio", loss="LASSO", reg_lambda=0.3).fit(y)
OMG(model_a="ANN", model_b="ANN", loss="RIDGE", reg_lambda=0.3).fit(y)
OMG(model_a="ANN", model_b="ANN",
    loss=lambda actual, fitted, B: np.sum(np.abs(actual - fitted) ** 3)).fit(y)

For the OMG joint-likelihood path, the C++ omfitGeneral state-space step always runs first (producing the combined probability p = p_A / (p_A + p_B)); the chosen loss then decides what scalar to hand to the optimiser. "likelihood" is the joint Bernoulli; the others use the probability-scale residual ot - p_combined.

Important Notes

  1. Model selection: Model selection and combination work properly only for loss="likelihood". Other loss functions may produce suboptimal model selection.

  2. Information criteria: When loss!="likelihood", the log-likelihood and information criteria (AIC, BIC) are calculated based on the connection between losses and distributions (if one exists). e.g. in case of loss="MAE", distribution is assumed to be Laplace. Only works for ADAM.

  3. Multi-step loss: Always specify h when using multi-step loss functions.

  4. LASSO/RIDGE normalization: Variables are not normalized prior to estimation, but parameters are divided by the mean values of explanatory variables.

  5. Parameter uncertainty: Fisher Information can only be calculated in case of loss="likelihood". In all the other case, when using summary(), vcov(), confint(), reapply() or reforecast(), use bootstrap=TRUE. (Bootstrap inference is R-only; the Python vcov()/confint()/summary() are Fisher-Information based and do not accept a bootstrap argument.)

When to Use Each Loss

Loss Best For
"likelihood" General use, model selection, prediction intervals
"MSE" Minimizing squared errors, produces mean forecasts
"MAE" Robust estimation, minimised by median
"HAM" Super robust estimation
"GTMSE" Optimizing multi-horizon forecasts
"LASSO" Alternative to multistep losses, better controlled
Custom Domain-specific loss functions

References

See Also

Clone this wiki locally