Skip to content

RMCB

Jump to bottom
Ivan Svetunkov edited this page Jun 29, 2026 · 2 revisions

RMCB — Regression for Multiple Comparison with the Best

Function signatures

R

rmcb(data, level=0.95, outplot=c("mcb","lines","none"), select=NULL, ...)

# Methods on the returned "rmcb" object
print(x, ...)
plot(x, outplot=c("mcb","lines"), select=NULL, ...)

The distribution, na.last and ties.method arguments are passed through the ....

Python

def rmcb(
    data,
    level: float = 0.95,
    outplot: str = "mcb",
    select=None,
    distribution: str = "tukey",
    na_last: bool = True,
    ties_method: str = "average",
) -> RMCBResult: ...

# Methods on the returned RMCBResult object
RMCBResult.__str__(self) -> str                       # print(result)
RMCBResult.plot(self, outplot="mcb", select=None, ax=None, **kwargs)

Overview

When several forecasting methods are evaluated on the same set of time series, a natural question is which method is best, and which others are statistically indistinguishable from it? RMCB — Regression for Multiple Comparison with the Best — answers this. It is a regression-based version of the Nemenyi / MCB (Multiple Comparison with the Best) test (Demsar, 2006).

The input data has the methods in columns and the series (or observations) in rows — typically a matrix of an error measure (e.g. one of the measures) computed for every method on every series. RMCB ranks the data within each row, then fits a dummy-variable regression on the ranks:

y = b' X + e

where y is the vector of ranks, X is the matrix of dummy variables (one per method) and b holds the mean rank of each method. Because the data is ranked, the test compares the medians of the methods. With the default distribution="tukey" the result is exactly the classical Nemenyi test.

Import

# R — function is in the greybox namespace
library(greybox)
# Python
from greybox import rmcb

How it works

  1. Each row of data is ranked (distribution "tukey"/"dnorm"); ties are handled by ties.method/ties_method.
  2. A treatment-coded dummy design is built: an intercept plus one dummy per method beyond the first. The regression coefficients give the mean rank of each method.
  3. A critical distance (half the interval width) is derived from the chosen distribution (see below). Each method gets a confidence interval mean ± critical distance.
  4. Methods whose intervals overlap are statistically indistinguishable and are collected into groups (used by the "lines" plot).

The distribution parameter

The default reproduces the Nemenyi test; the alternatives fit a regression and take the standard errors from it.

distribution Data used Critical distance Significance p-value
"tukey" (default) ranks Studentised range: qtukey(level, k, Inf) · sqrt(k(k+1) / (12·n)) / 2 Friedman test → identical to the Nemenyi test
"dnorm" ranks qt((1+level)/2, df) · SE from an OLS regression regression F-test
"dlnorm" raw data qt · SE on log(data); coefficients/intervals exponentiated chi-squared test (log-likelihood ratio)
any other (e.g. "dlaplace", "ds", …) raw data qt · SE from alm()/ALM() fitted with that distribution chi-squared test

Here k is the number of methods and n the number of series. On large datasets "dnorm" behaves much like the default "tukey". Any distribution other than the three above fits an ALM on the original data (comparing the locations of the methods rather than their ranks).

rmcb — parameters

Parameter R Python Type Default Description
data data data matrix/data.frame / array-like or DataFrame — Methods in columns, series in rows. Column names become method names.
level level level numeric/float 0.95 Width of the confidence intervals.
outplot outplot outplot character/str "mcb" Plot style: "mcb", "lines" or "none". See the interface note below.
select select select index / NULL / None NULL/None Method to highlight. NULL/None selects the best (lowest-mean) method.
distribution ... distribution character/str "tukey" Test variant (see table above).
na.last / na_last ... na_last logical/bool TRUE/True Whether NA/NaN are ranked last.
ties.method / ties_method ... ties_method character/str "average" Tie-handling for the ranks.

Interface difference. In R, rmcb() draws the plot as a side effect unless outplot="none". The Python rmcb() never auto-plots — it only computes and returns the result; call result.plot(...) to draw. The outplot value is stored and used as the default style for that method. See R-Python-differences.

Return value

R returns an S3 object of class "rmcb"; Python returns an RMCBResult. Both expose the same fields:

Field R Python Type Description
mean $mean .mean named vector / pd.Series Mean values (mean ranks for "tukey"/"dnorm") per method, sorted ascending.
interval $interval .interval matrix / pd.DataFrame Lower/upper confidence bounds per method.
vlines $vlines .vlines matrix / pd.DataFrame Group start/end positions used by the "lines" plot.
groups $groups .groups logical matrix / pd.DataFrame TRUE/True if a method belongs to a group.
methods $methods .methods logical matrix / pd.DataFrame Pairwise interval-overlap table.
p.value $p.value .p_value numeric / float Significance test p-value.
level $level .level numeric / float Confidence level.
model $model .model model object Fitted model used for the intervals.
select $select .select integer 1-based position (in sorted order) of the highlighted method.
distribution — .distribution str The distribution used (Python convenience field).

Example: four methods

The example mirrors ?rmcb: four methods with deliberately shifted locations, so method A should rank best.

R

library(greybox)

N <- 50
M <- 4
ourData <- matrix(rnorm(N*M, mean=0, sd=1), N, M)
ourData[,2] <- ourData[,2]+4
ourData[,3] <- ourData[,3]+3
ourData[,4] <- ourData[,4]+2
colnames(ourData) <- c("Method A","Method B","Method C - long name","Method D")

ourTest <- rmcb(ourData, level=0.95)

# Mean ranks and intervals
ourTest$mean
ourTest$interval

# Plots
plot(ourTest)                    # "mcb" style (default)
plot(ourTest, outplot="lines")   # grouped "lines" style

Python

import numpy as np
import pandas as pd
from greybox import rmcb

rng = np.random.default_rng(42)
data = rng.normal(size=(50, 4))
data[:, 1] += 4
data[:, 2] += 3
data[:, 3] += 2
df = pd.DataFrame(
    data,
    columns=["Method A", "Method B", "Method C - long name", "Method D"],
)

result = rmcb(df, level=0.95)
print(result)
# Regression for Multiple Comparison with the Best
# The significance level is 5%
# The number of observations is 50, the number of methods is 4
# Significance test p-value: 0.0

print(result.mean.round(4))
# Method A                1.04
# Method D                2.18
# Method C - long name    3.12
# Method B                3.66

print(result.interval.round(4))
#                         2.5%   97.5%
# Method A              0.7083  1.3717
# Method D              1.8483  2.5117
# Method C - long name  2.7883  3.4517
# Method B              3.3283  3.9917

# Plots (Python never auto-plots — call plot() explicitly)
result.plot()                    # "mcb" style (default)
result.plot(outplot="lines")     # grouped "lines" style

Other distributions

# R — Student-distribution variant (≈ tukey on large data)
ourTest <- rmcb(ourData, distribution="dnorm")
# Python
result = rmcb(df, distribution="dnorm")

Plot interpretation

  • "mcb" (default): one vertical line per method spanning its confidence interval, with a point at the mean rank. The selected method (by default the best, i.e. lowest-mean) is highlighted and its interval bounds are drawn as horizontal dashed reference lines. Any method whose interval crosses those lines overlaps the best method and shares its colour — it is not significantly worse. The title reports the test p-value and the confidence level.
  • "lines": each group of statistically indistinguishable methods is drawn as a vertical segment spanning the methods it covers, so overlapping groups are easy to read off. The method names label the vertical axis.

Use select= (a method name or index) to highlight a different reference method on either plot.

R vs Python Function Names

R Python
rmcb() rmcb()
print(x) print(result) / str(result)
plot(x, outplot=...) result.plot(outplot=...)
x$p.value result.p_value

Notes on the Python implementation

  • Top-level imports: from greybox import rmcb, RMCBResult.
  • matplotlib>=3.5 is a hard dependency, so result.plot() always works.
  • Parity: mean, interval and p_value match R to about 1e-8 for the rank/OLS branches ("tukey", "dnorm", "dlnorm"). For the general alm()-based branch the means and p-values match, but interval widths can differ: R falls back to a bootstrapped covariance matrix when the Hessian is inaccurate, whereas Python uses the Hessian-based vcov. See R-Python-differences.
  • select is reported as a 1-based position in the sorted order (matching R); the Python plot(select=...) additionally accepts a method name or a 0-based column index.
  • The tabular fields (interval, vlines, groups, methods) are returned as pandas DataFrames and mean as a Series, indexed by method name.

References

Clone this wiki locally