🎉 Introducing BinBench - A full DIBCO metrics replacement

[brandonmpetty]

Background

From 2009 to 2019, the International Conference on Document Analysis and Recognition (ICDAR) and the International Conference on Frontiers in Handwriting Recognition (ICFHR) held alternating contests. These contests brought together universities from around the world to advance the state of document image binarization. Known as DIBCO (short for Document Image Binarization Contest) and H-DIBCO (the handwritten-focused version), they required each contestant to submit a paper and an executable capable of generating a binary image using their proposed algorithm. Roughly 10 to 20 images would be run through each team's algorithm. The results were then evaluated with a suite of performance metrics to determine how well they performed before being tallied up and ranked. The submission with the highest rank was declared the winner.

To accomplish this, the (H)DIBCO officials built two tools to calculate these performance metrics: DIBCO_metrics.exe and BinEvalWeights.exe. DIBCO_metrics was a MATLAB-based executable that evolved over time and eventually open sourced. It generated metrics like F-Measure, PSNR, and DRDM, along with a special-purpose metric named Pseudo F-Measure. To generate Pseudo F-Measure, the caller had to pass two weight-based .dat files to DIBCO_metrics and it was BinEvalWeights' responsibility to generate those files. Unlike its fellow executable, BinEvalWeights has remained a closed-source, C++-based, Windows binary. The algorithm it used to generate these weight files was based on a published research paper authored by the same (H)DIBCO coordinators. For over a decade, anyone wanting to test their algorithms had to use this BinEvalWeights binary... until today.

BinBench Released

Today I am announcing the latest version of the Δoxa Binarization Framework and an accompanying CLI utility, BinBench. With version 0.9.8, ΔBF can now generate not only all DIBCO metrics but also bit-exact weight files. Many attempts were made to reach out to the original authors of BinEvalWeights, with no success. For over 10 years, an entire community was built partially upon a closed-source, Windows-only binary that no one could reproduce. Today, BinBench is my answer to that problem. It is open source, openly licensed, cross-platform, and extremely fast. This single utility not only takes the place of the two DIBCO binaries, but also ships with 18 binarization algorithms and 10 grayscale conversions, each with full parameter documentation. While developing BinBench, I found a slight floating-point rounding issue in DIBCO_metrics' DRDM implementation, giving BinBench one more reason to be considered a viable alternative.

Takeaway

While the (H)DIBCO contests are over, other contests have taken their place using the same tooling. Today's release is a statement — a statement that we should expect more. We should no longer be relying on closed-source Windows binaries whose actual implementation could never be reproduced from a research paper alone. We should not be handed binarization metrics from algorithms like Otsu and Sauvola without the full source code, the algorithm parameters, and the grayscale conversion used. BinBench's Otsu implementation with BT601 grayscale produces paper-exact results (which, incidentally, allowed me to identify a label mismatch in Table III of "ICDAR 2019 Competition on Document Image Binarization (DIBCO 2019)"). The Sauvola implementation used throughout the contest, however, remains undetermined. The binarized images produced by the contestants should also be released; without them, the community loses a critical evaluation resource and the only means of ever correcting erroneous results. This is not an indictment of the (H)DIBCO organizers. They were true pioneers who helped get us to where we are today. That said, in 2026 and beyond, we should expect more and BinBench gets us one step closer.

The goal of ΔBF has always been to expose the brilliance of this niche binarization community to a broader audience. With today's release, I feel like I am finally giving back to the binarization community itself. As this is the first release of BinBench, I am sure it is far from perfect and I would love to hear any feedback you have to make it better.

Get BinBench v0.9.8 today!