Skip to content

RMCB

Jump to bottom
Ivan Svetunkov edited this page Jun 29, 2026 · 2 revisions

RMCB — Regression for Multiple Comparison with the Best

Function signatures

R

rmcb(data, level=0.95, outplot=c("mcb","lines","none"), select=NULL, ...)

# Methods on the returned "rmcb" object
print(x, ...)
plot(x, outplot=c("mcb","lines"), select=NULL, ...)

The distribution, na.last and ties.method arguments are passed through the ....

Python

def rmcb(
    data,
    level: float = 0.95,
    outplot: str = "mcb",
    select=None,
    distribution: str = "tukey",
    na_last: bool = True,
    ties_method: str = "average",
) -> RMCBResult: ...

# Methods on the returned RMCBResult object
RMCBResult.__str__(self) -> str                       # print(result)
RMCBResult.plot(self, outplot="mcb", select=None, ax=None, **kwargs)

Overview

When several forecasting methods are evaluated on the same set of time series, a natural question is which method is best, and which others are statistically indistinguishable from it? RMCB — Regression for Multiple Comparison with the Best — answers this (Demsar, 2006).

The input data has the methods in columns and the series (or observations) in rows — typically a matrix of an error measure (e.g. one of the measures) computed for every method on every series. In all cases each method ends up with a mean value and a confidence interval; methods whose intervals overlap are statistically indistinguishable and are collected into groups (used by the "lines" plot and the groups/methods/vlines fields).

The distribution argument selects one of two families of comparison:

  • Nemenyi / MCB — the default (distribution="tukey"): the classical rank-based test. The inference comes from the Studentised-range statistic and the Friedman test, not from a regression.
  • Regression for MCB — (distribution="dnorm", "dlnorm", or any other): a regression-based generalisation that fits a dummy-variable model and takes the standard errors and the significance test from that fitted model.

Import

# R — function is in the greybox namespace
library(greybox)
# Python
from greybox import rmcb

Nemenyi / MCB (default, distribution="tukey")

With the default distribution="tukey", rmcb() reproduces the classical Nemenyi test exactly. It is not a regression-based comparison:

  1. The data is ranked within each row (ties.method/ties_method controls ties).
  2. Each method's statistic is its mean rank across the series.
  3. The critical distance — the half-width shared by every interval — is the Studentised-range statistic: qtukey(level, k, Inf) · sqrt(k(k+1) / (12·n)) / 2, where k is the number of methods and n the number of series.
  4. The overall significance (is any method different from the others?) is the Friedman test p-value.

So the confidence intervals and the p-value come purely from the classical Nemenyi / Friedman machinery; no regression standard errors are used. (Internally the mean ranks happen to be computed as the coefficients of a dummy regression on the ranks — they equal the column means of the ranks — but that regression's standard errors and F-test play no part in the default result.)

Regression for MCB (other distributions)

For any non-default distribution, rmcb() lives up to its name: it fits a dummy-variable regression and derives the whole comparison from that fitted model:

y = b' X + e

X is treatment-coded with one dummy per method (the intercept is the first method), so the coefficients b give each method's location. The interval half-width is taken from the model's standard errors, qt((1+level)/2, df) · SE, and the significance is a regression-based test:

  • "dnorm" — ranks the data, then fits an OLS regression on the ranks; p-value from the regression F-test. On large datasets this is close to the default Nemenyi test.
  • "dlnorm" — fits OLS on log(data) (raw, positive data); the coefficients and intervals are exponentiated; chi-squared (log-likelihood-ratio) p-value.
  • any other (e.g. "dlaplace", "ds", …) — fits an ALM with that distribution on the raw data, comparing the methods' locations; chi-squared p-value.

Summary of the variants

distribution Approach Data used Critical distance Significance p-value
"tukey" (default) Nemenyi / MCB ranks Studentised range: qtukey(level, k, Inf) · sqrt(k(k+1) / (12·n)) / 2 Friedman test → classical Nemenyi test
"dnorm" Regression for MCB ranks qt((1+level)/2, df) · SE from an OLS regression regression F-test
"dlnorm" Regression for MCB raw data qt · SE on log(data); coefficients/intervals exponentiated chi-squared test (log-likelihood ratio)
any other (e.g. "dlaplace", "ds", …) Regression for MCB raw data qt · SE from alm()/ALM() fitted with that distribution chi-squared test

rmcb — parameters

Parameter R Python Type Default Description
data data data matrix/data.frame / array-like or DataFrame — Methods in columns, series in rows. Column names become method names.
level level level numeric/float 0.95 Width of the confidence intervals.
outplot outplot outplot character/str "mcb" Plot style: "mcb", "lines" or "none". See the interface note below.
select select select index / NULL / None NULL/None Method to highlight. NULL/None selects the best (lowest-mean) method.
distribution ... distribution character/str "tukey" Test variant (see table above).
na.last / na_last ... na_last logical/bool TRUE/True Whether NA/NaN are ranked last.
ties.method / ties_method ... ties_method character/str "average" Tie-handling for the ranks.

Interface difference. In R, rmcb() draws the plot as a side effect unless outplot="none". The Python rmcb() never auto-plots — it only computes and returns the result; call result.plot(...) to draw. The outplot value is stored and used as the default style for that method. See R-Python-differences.

Return value

R returns an S3 object of class "rmcb"; Python returns an RMCBResult. Both expose the same fields:

Field R Python Type Description
mean $mean .mean named vector / pd.Series Mean values (mean ranks for "tukey"/"dnorm") per method, sorted ascending.
interval $interval .interval matrix / pd.DataFrame Lower/upper confidence bounds per method.
vlines $vlines .vlines matrix / pd.DataFrame Group start/end positions used by the "lines" plot.
groups $groups .groups logical matrix / pd.DataFrame TRUE/True if a method belongs to a group.
methods $methods .methods logical matrix / pd.DataFrame Pairwise interval-overlap table.
p.value $p.value .p_value numeric / float Significance test p-value.
level $level .level numeric / float Confidence level.
model $model .model model object Fitted model used for the intervals.
select $select .select integer 1-based position (in sorted order) of the highlighted method.
distribution — .distribution str The distribution used (Python convenience field).

Example: four methods

The example mirrors ?rmcb: four methods with deliberately shifted locations, so method A should rank best.

R

library(greybox)

N <- 50
M <- 4
ourData <- matrix(rnorm(N*M, mean=0, sd=1), N, M)
ourData[,2] <- ourData[,2]+4
ourData[,3] <- ourData[,3]+3
ourData[,4] <- ourData[,4]+2
colnames(ourData) <- c("Method A","Method B","Method C - long name","Method D")

ourTest <- rmcb(ourData, level=0.95)

# Mean ranks and intervals
ourTest$mean
ourTest$interval

# Plots
plot(ourTest)                    # "mcb" style (default)
plot(ourTest, outplot="lines")   # grouped "lines" style

Python

import numpy as np
import pandas as pd
from greybox import rmcb

rng = np.random.default_rng(42)
data = rng.normal(size=(50, 4))
data[:, 1] += 4
data[:, 2] += 3
data[:, 3] += 2
df = pd.DataFrame(
    data,
    columns=["Method A", "Method B", "Method C - long name", "Method D"],
)

result = rmcb(df, level=0.95)
print(result)
# Regression for Multiple Comparison with the Best
# The significance level is 5%
# The number of observations is 50, the number of methods is 4
# Significance test p-value: 0.0

print(result.mean.round(4))
# Method A                1.04
# Method D                2.18
# Method C - long name    3.12
# Method B                3.66

print(result.interval.round(4))
#                         2.5%   97.5%
# Method A              0.7083  1.3717
# Method D              1.8483  2.5117
# Method C - long name  2.7883  3.4517
# Method B              3.3283  3.9917

# Plots (Python never auto-plots — call plot() explicitly)
result.plot()                    # "mcb" style (default)
result.plot(outplot="lines")     # grouped "lines" style

Other distributions

# R — Student-distribution variant (≈ tukey on large data)
ourTest <- rmcb(ourData, distribution="dnorm")
# Python
result = rmcb(df, distribution="dnorm")

Plot interpretation

  • "mcb" (default): one vertical line per method spanning its confidence interval, with a point at the mean rank. The selected method (by default the best, i.e. lowest-mean) is highlighted and its interval bounds are drawn as horizontal dashed reference lines. Any method whose interval crosses those lines overlaps the best method and shares its colour — it is not significantly worse. The title reports the test p-value and the confidence level.
  • "lines": each group of statistically indistinguishable methods is drawn as a vertical segment spanning the methods it covers, so overlapping groups are easy to read off. The method names label the vertical axis.

Use select= (a method name or index) to highlight a different reference method on either plot.

R vs Python Function Names

R Python
rmcb() rmcb()
print(x) print(result) / str(result)
plot(x, outplot=...) result.plot(outplot=...)
x$p.value result.p_value

Notes on the Python implementation

  • Top-level imports: from greybox import rmcb, RMCBResult.
  • matplotlib>=3.5 is a hard dependency, so result.plot() always works.
  • Parity: mean, interval and p_value match R to about 1e-8 for the rank/OLS branches ("tukey", "dnorm", "dlnorm"). For the general alm()-based branch the means and p-values match, but interval widths can differ: R falls back to a bootstrapped covariance matrix when the Hessian is inaccurate, whereas Python uses the Hessian-based vcov. See R-Python-differences.
  • select is reported as a 1-based position in the sorted order (matching R); the Python plot(select=...) additionally accepts a method name or a 0-based column index.
  • The tabular fields (interval, vlines, groups, methods) are returned as pandas DataFrames and mean as a Series, indexed by method name.

References

Clone this wiki locally