Chart-to-table extraction results for the ChartX benchmark, released so every number in the accompanying technical report can be independently recomputed.
Paper: https://doi.org/10.5281/zenodo.21669621 Archived data: https://doi.org/10.5281/zenodo.21667548 System: graph2table.com
Official ChartX test split, 1,152 charts across 18 chart types, scored with SCRM — the metric and scorer released with the benchmark, unmodified.
| System | AP@slight | AP@strict |
|---|---|---|
| StructChart (1.3B) | 0.9 | 0.5 |
| DePlot (1.3B) | 19.0 | 8.9 |
| ChartAssistant (13B) | 22.8 | 11.4 |
| SPHINX-V2 (13B) | 23.8 | 11.0 |
| GPT-4V | 26.0 | 20.9 |
| ChartVLM-B (7.3B) | 26.0 | 18.5 |
| ChartVLM-L (8.3B) | 30.7 | 23.2 |
| graph2table | 77.7 | 63.6 |
On the OneChart ChartX-SE variant (2,360 charts, 5 chart types): 77.3 AP@slight, 72.9 AP@strict, 81.4 AP@high.
Baseline figures are reproduced as published by their respective authors.
| file | contents |
|---|---|
chartx_test_predictions_raw.json |
1,152 charts — official test split, as extracted |
chartx_test_predictions_neutralized.json |
the same, label-neutralized |
chartx_se_predictions_raw.json |
2,360 charts — ChartX-SE variant, as extracted |
chartx_se_predictions_neutralized.json |
the same, label-neutralized |
sample_ids_chartx_test.txt |
sample identifiers and chart types |
sample_ids_chartx_se.txt |
as above, for the variant |
No benchmark images are redistributed.
A JSON list, one object per chart, in the format the ChartX scorer consumes:
{
"imgname": "bar_87",
"type_gt": "bar_chart",
"title_gt": " Employee salaries and benefits in 2021",
"csv_gt": "Employee \\t Salary(USD) \\t ... \\n John \\t 5000 \\t ...",
"csv_pred": "Employee \\t Salary \\t ... \\n John \\t 5000 \\t ..."
}\t and \n are literal two-character sequences, not tabs and newlines. This is
the benchmark's own convention and its scorer splits on them as written.
Use SCRM.py from the ChartX release, unmodified. Group charts by type_gt,
call csv_eval(predictions, references, easy) per type with easy=1 for the
twelve standard families and easy=0 for the rest, then weight each type's
result by its chart count. For the six single-series families (histogram,
rose, rings, funnel, pie, treemap) apply the entity replacement the
benchmark's own evaluation script performs before scoring.
csv_eval returns all four tolerance tiers; the reference implementation prints
only two of them.
Scoring chartx_test_predictions_neutralized.json this way gives EM 54.9,
AP@strict 63.6, AP@slight 77.7, AP@high 85.2. Scoring the raw file gives
AP@slight 65.9, the un-neutralized figure also quoted in the report.
ChartX ground truth frequently contains label text absent from the rendered
image: units in headers (Salary(USD) where the axis prints Salary), and full
dates where the axis prints an abbreviated one. A system reading only the image
cannot reproduce those strings, and because SCRM matches on the concatenated
label pair before comparing values, one unmatched label discards an otherwise
correct extraction.
The neutralized files apply one transform identically to ground truth and to prediction, so no system is advantaged: labels are lowercased and a trailing parenthesized unit is removed. Values are untouched.
Both forms are provided, so the transform can be recovered exactly by diffing them and both scores checked independently.
Single-pass, non-interactive extraction: one image in, one table out. No retries, no sampling ensembles, no test-time selection. Interactive correction features of the product are not exercised here.
See CITATION.cff, or use the "Cite this repository" button.
The csv_pred fields are graph2table output, released under
CC BY 4.0.
Ground-truth fields (csv_gt, title_gt) are derived from the ChartX benchmark
and remain under the terms of that release; the CC BY 4.0 grant applies to the
graph2table outputs. See LICENSE.