measures

Measures — Forecast Accuracy Metrics

Overview

The forecast accuracy metrics are organized across three modules:

• greybox.point_measures — Point forecast measures (ME, MAE, MSE, RMSE, MPE, MAPE, MASE, etc.) and convenience function (measures())
• greybox.quantile_measures — Quantile scoring (pinball()) and interval scoring (mis(), smis(), rmis())
• greybox.hm — Half-moment measures (hm(), ham(), asymmetry(), extremity(), cextremity()) and Mean Root Error (mre())

Point Forecast Measures

Scale-dependent measures

Measure	Formula	R Function	Python Function	Description
ME	mean(y - f)	ME(actual, forecast)	me(actual, forecast)	Mean Error — measures bias
MAE	mean(|y - f|)	MAE(actual, forecast)	mae(actual, forecast)	Mean Absolute Error
MSE	mean((y - f)^2)	MSE(actual, forecast)	mse(actual, forecast)	Mean Squared Error — penalizes large errors
RMSE	sqrt(MSE)	RMSE(actual, forecast)	rmse(actual, forecast)	Root MSE — same units as data

Percentage measures

Measure	Formula	R Function	Python Function	Description
MPE	mean((y - f) / y) * 100	MPE(actual, forecast)	mpe(actual, forecast)	Mean Percentage Error — percentage bias
MAPE	mean(|y - f| / |y|) * 100	MAPE(actual, forecast)	mape(actual, forecast)	Mean Absolute Percentage Error — undefined when y=0

Scaled measures

Measure	Formula	R Function	Python Function	Description
MASE	MAE / mean(|diff(y)|)	MASE(actual, forecast)	mase(actual, forecast, scale)	Mean Absolute Scaled Error (Hyndman & Koehler, 2006)
RMSSE	sqrt(MSE / mean(diff(y)^2))	RMSSE(actual, forecast)	rmsse(actual, forecast, scale)	Root Mean Squared Scaled Error (M5 Competition)
SAME	|ME| / mean(|diff(y)|)	—	same(actual, forecast, scale)	Scaled Absolute Mean Error — scaled bias
sMSE	MSE / scale^2	sMSE(actual, forecast, scale)	smse(actual, forecast, scale)	Scaled MSE (Petropoulos & Kourentzes, 2015)
sCE	sum(y - f) / scale	sCE(actual, forecast, scale)	sce(actual, forecast, scale)	Scaled Cumulative Error
sPIS	sum(cumsum(f - y)) / scale	sPIS(actual, forecast, scale)	spis(actual, forecast, scale)	Scaled Periods-In-Stock (Wallstrom & Segerstedt, 2010)

Relative measures (require benchmark forecast)

Measure	Formula	R Function	Python Function	Description
rMAE	MAE / MAE_bench	rMAE(actual, forecast, bench)	rmae(actual, forecast, benchmark)	Relative MAE (Davydenko & Fildes, 2013)
rRMSE	RMSE / RMSE_bench	rRMSE(actual, forecast, bench)	rrmse(actual, forecast, benchmark)	Relative RMSE
rAME	|ME| / |ME_bench|	rAME(actual, forecast, bench)	rame(actual, forecast, benchmark)	Relative Absolute Mean Error
GMRAE	exp(mean(log(|e| / |e_bench|)))	GMRAE(actual, forecast, bench)	gmrae(actual, forecast, benchmark)	Geometric Mean Relative Absolute Error

Quantile Measures

pinball() — Pinball cost function

The pinball function measures the quality of quantile or expectile forecasts.

from greybox.quantile_measures import pinball

holdout = np.array([1, 2, 3, 4, 5])
forecast = np.array([1.1, 2.0, 3.2, 3.9, 5.1])

pinball(holdout, forecast, level=0.5)    # Median pinball
pinball(holdout, forecast, level=0.975)  # Upper quantile
pinball(holdout, forecast, level=0.025)  # Lower quantile

Parameters:

Parameter	Type	Default	Description
holdout	np.ndarray	—	Actual values
forecast	np.ndarray	—	Forecasted quantile/expectile values
level	float	—	Quantile level (e.g., 0.5 for median, 0.975 for upper)
loss	int	1	1 = L1 (quantile loss), 2 = L2 (expectile loss)
na_rm	bool	True	Remove NA values

Formulas:

For quantiles (loss=1):

pinball = (1 - level) * sum(|e| * I(e <= 0)) + level * sum(|e| * I(e > 0))
where e = holdout - forecast

For expectiles (loss=2):

pinball = (1 - level) * sum(e^2 * I(e <= 0)) + level * sum(e^2 * I(e > 0))

Interval Forecast Measures

from greybox.quantile_measures import mis, smis, rmis

Measure	R Function	Python Function	Description
MIS	MIS(actual, lower, upper, level)	mis(actual, lower, upper, level)	Mean Interval Score (Gneiting & Raftery, 2007)
sMIS	sMIS(actual, lower, upper, scale, level)	smis(actual, lower, upper, scale, level)	Scaled MIS
rMIS	—	rmis(actual, lower, upper, bench_lower, bench_upper, level)	Relative MIS

The MIS rewards narrow intervals and penalizes when actuals fall outside:

MIS = mean(upper - lower + (2/alpha) * (lower - y) * I(y < lower) + (2/alpha) * (y - upper) * I(y > upper))
where alpha = 1 - level

Half-Moment Measures

The half-moment measures (from greybox.hm) characterize distribution asymmetry and extremity using square root transformations. They are based on the concept of the Half Central Moment (Svetunkov, Kourentzes & Svetunkov, 2023).

from greybox.hm import hm, ham, asymmetry, extremity, cextremity, mre

Function	Signature	Returns	Description
hm	hm(x, center=None)	complex	Half Moment — mean(sqrt(x - C)) where C defaults to mean(x)
ham	ham(x, center=None)	float	Half Absolute Moment — mean(sqrt(|x - C|))
asymmetry	asymmetry(x, center=None)	float	Asymmetry coefficient — range [-1, 1], 0 = symmetric
extremity	extremity(x, center=None)	float	Extremity coefficient — measures tail heaviness
cextremity	cextremity(x, center=None)	complex	Complex Extremity — captures both magnitude and phase
mre	mre(actual, forecast)	float	Mean Root Error (Kourentzes, 2014) — Re(mean(sqrt(y - f)))

Notes:

• hm() returns a complex number because sqrt(x - C) is complex when x < C.
• asymmetry() is computed as 1 - Arg(hm(x)) / (pi/4). Values: 1 = all below center, 0 = symmetric, -1 = all above center.
• For all functions, center defaults to mean(x) when not provided.
• mre() is the real part of hm() applied to forecast errors.

Convenience Functions

measures() — Comprehensive evaluation with training data

# Python
from greybox.point_measures import measures

# holdout = test actuals, forecast = predictions, actual = training data
result = measures(holdout, forecast, actual_train, digits=4, benchmark="naive")
# Returns dict with: ME, MAE, MSE, MPE, MAPE, sCE, sMAE, sMSE, MASE,
#                     RMSSE, SAME, rMAE, rRMSE, rAME, asymmetry, sPIS

# R
measures(holdout, forecast, actual_train, digits=4)

Examples

Individual Measures

# R
actual <- c(1, 2, 3, 4, 5, 6, 7, 8, 9, 10)
forecast <- c(1.1, 2.0, 3.2, 3.9, 5.1, 6.0, 7.1, 8.0, 9.2, 10.1)

MAE(actual, forecast)     # 0.12
MSE(actual, forecast)     # 0.018
MAPE(actual, forecast)    # percentage error

# Python
import numpy as np
from greybox.point_measures import mae, mse, rmse, mape, mase

actual = np.array([1, 2, 3, 4, 5, 6, 7, 8, 9, 10])
forecast = np.array([1.1, 2.0, 3.2, 3.9, 5.1, 6.0, 7.1, 8.0, 9.2, 10.1])

print(f"MAE:  {mae(actual, forecast):.4f}")
print(f"MSE:  {mse(actual, forecast):.4f}")
print(f"RMSE: {rmse(actual, forecast):.4f}")
print(f"MAPE: {mape(actual, forecast):.2f}%")
print(f"MASE: {mase(actual, forecast):.4f}")

Relative Measures with Benchmark

# Python
from greybox.point_measures import rmae, rrmse

# Compare against a naive forecast (last value repeated)
benchmark = np.full_like(forecast, actual[-1])

print(f"rMAE:  {rmae(actual, forecast, benchmark):.4f}")
print(f"rRMSE: {rrmse(actual, forecast, benchmark):.4f}")
# Values < 1 mean the forecast is better than the benchmark

Interval Score

# Python
from greybox.quantile_measures import mis

actual = np.array([1, 2, 3, 4, 5])
lower = np.array([0.5, 1.5, 2.5, 3.5, 4.5])
upper = np.array([1.5, 2.5, 3.5, 4.5, 5.5])

score = mis(actual, lower, upper, level=0.95)
print(f"MIS: {score:.4f}")

Pinball Loss

# Python
from greybox.quantile_measures import pinball

holdout = np.array([1, 2, 3, 4, 5])
forecast_median = np.array([1.1, 2.0, 3.2, 3.9, 5.1])
forecast_upper = np.array([1.5, 2.5, 3.5, 4.5, 5.5])

# Pinball at median
print(f"Pinball (median): {pinball(holdout, forecast_median, level=0.5):.4f}")
# Pinball at 97.5th percentile
print(f"Pinball (upper):  {pinball(holdout, forecast_upper, level=0.975):.4f}")

Half-Moment Analysis

# Python
import numpy as np
from greybox.hm import hm, ham, asymmetry, extremity, mre

x = np.array([1, 2, 3, 4, 5, 6, 7, 8, 9, 10])

print(f"Half Moment:  {hm(x)}")
print(f"HAM:          {ham(x):.4f}")
print(f"Asymmetry:    {asymmetry(x):.4f}")
print(f"Extremity:    {extremity(x):.4f}")

# MRE for forecast evaluation
actual = np.array([1, 2, 3, 4, 5])
forecast = np.array([1.1, 2.0, 3.2, 3.9, 5.1])
print(f"MRE:          {mre(actual, forecast):.4f}")

Implementation Status

Measure	R	Python
ME, MAE, MSE, RMSE	Yes	Yes
MPE, MAPE	Yes	Yes
MASE, RMSSE, SAME	Yes	Yes
rMAE, rRMSE, rAME	Yes	Yes
GMRAE	Yes	Yes
sMSE, sPIS, sCE	Yes	Yes
MIS, sMIS, rMIS	Yes	Yes (via greybox.quantile_measures)
pinball	Yes	Yes (via greybox.quantile_measures)
asymmetry	Yes	Yes (via greybox.hm)
hm, ham	Yes	Yes (via greybox.hm)
extremity, cextremity	Yes	Yes (via greybox.hm)
MRE	Yes	Yes (via greybox.hm)
measures()	Yes	Yes

References

• Hyndman, R.J. and Koehler, A.B. (2006). Another look at measures of forecast accuracy. International Journal of Forecasting, 22, pp.679-688.
• Davydenko, A. and Fildes, R. (2013). Measuring Forecasting Accuracy: The Case Of Judgmental Adjustments To Sku-Level Demand Forecasts. International Journal of Forecasting, 29(3), pp.510-522.
• Petropoulos, F. and Kourentzes, N. (2015). Forecast combinations for intermittent demand. Journal of the Operational Research Society, 66, pp.914-924.
• Wallstrom, P. and Segerstedt, A. (2010). Evaluation of forecasting error measurements and techniques for intermittent demand. International Journal of Production Economics, 128, pp.625-636.
• Gneiting, T. and Raftery, A.E. (2007). Strictly proper scoring rules, prediction, and estimation. Journal of the American Statistical Association, 102(477), pp.359-378.
• Kourentzes, N. (2014). The Bias Coefficient: a new metric for forecast bias.
• Svetunkov, I., Kourentzes, N. and Svetunkov, S. (2023). Half Central Moment for Data Analysis. Working Paper of Department of Management Science, Lancaster University, 2023:3, pp.1-21.
• Svetunkov, I. (2017). Naughty APEs and the quest for the holy grail. https://openforecast.org/2017/07/29/naughty-apes-and-the-quest-for-the-holy-grail/