ArxivFormula is the first dataset framing mathematical formula detection as a joint task of formula entity detection and formula relation extraction, rather than a simple task of object detection or instance segmentation. It's constructed using a weak supervision approach and comprises 500K document images for training, 50K for validation and 50K for testing.
- We released annotations and origin document images of the ArxivFormula datasets (OneDrive), please refer to
Get Datasection. - We released some examples of the ArxivFormula datasets, please refer to
ArxivFormula_examples.zip.
Most of existing mathematical formula detectors focus on detecting formula entities through object detection or instance segmentation techniques. However, these methods often fail to convey complete messages due to the absence of the contextual and layout information of mathematical formulas. For a more comprehensive understanding of mathematical formulas in document images, it is preferable to detect logical formula blocks that include one or multiple formula entities arranged in their natural reading order. These logical formula blocks enable the transmission of complete contextual messages of mathematical formulas and aid in the reconstruction of layout information of the document images, resulting in a more accurate mathematical formula detection. To this end, ArxivFormula first presents a novel perspective on the problem of mathematical formula detection by framing it as a joint task of formula entity detection and formula relation extraction, rather than a simple task of object detection or instance segmentation. This new perspective enables us to detect logical formula blocks that convey complete contextual and layout information of mathematical formulas, while also eliminating labeling issues in existing benchmark datasets.
| Dataset | #Documents | #Images | Formula Entity | Formula Relation | ||||
|---|---|---|---|---|---|---|---|---|
| IF | DFL | DFB | FN | NFL | FRN | |||
| InftyCDB-1 | 30 | 467 | 21k | - | - | - | - | |
| Marmot | 194 | 400 | 7.9k | 1.6k | - | - | - | - |
| ICDAR-2017 POD | 1.5k | 2.4k | - | 5.4k | - | - | - | - |
| TFD-ICDAR 2019 | 46 | 805 | 38k | - | - | - | - | |
| ICDAR-2021 IBEM | 600 | 8.3k | 137k | - | 29k | - | - | - |
| ArxivFormula | 100k | 600k | 15M | 1.9M | 1.4M | 795k | 540k | 795k |
Note: (IF: Inline Formula, DFL: Displayed Formula Line, DFB: Displayed Formula Block, FN: Formula Number, NFL: Next Formula Line, FRN: Formula Reference Number.)
**Please DO NOT re-distribute the dataset.**
The annotations and original document images of the ArxivFormula dataset can be download from the OneDrive. In order to reduce the loss caused by download interruption, we divided "ArxivFormula_Training_set_images.zip" into 10 parts, and after downloading all of them, use the decompression software to decompress them together. Additionally, we would like to note that we have only released the document images and annotations of arXiv papers that comply with the arXiv.org perpetual, non-exclusive license. This constitutes approximately 93% of the entire dataset.
| File | Size | md5sum |
|---|---|---|
| ArxivFormula_Training_set_images.zip [[1], [2], [3], [4], [5], [6], [7], [8], [9], [10]] |
5.9 GB /each part |
- |
| ArxivFormula_Validation_set_images.zip | 5.9 GB | a0f944b4150940c1d3cf060ff4307d7f |
| ArxivFormula_Testing_set_images.zip | 5.9 GB | fc3b956bbbe5a20ae82e74b4a182951e |
| ArxivFormula_Annotations.zip | 1.1 GB | a5735ae789850a854b0326079e3c925f |
The annotation format used is the standard COCO-style annotation format. More details about the dataset please refer to ArxivFormula_Details.
We propose a new approach, called FormulaDet, to address these two sub-tasks simultaneously. FormulaDet first employs a dynamic convolution-based formula entity detector, named DynFormula, to detect formula entities. It then uses a multi-modal transformer-based relation extraction method, named RelFormer, to group these detected formula entities into logical formula blocks.
| Methods | Validation Set (%) | Testing Set (%) | |||||||
|---|---|---|---|---|---|---|---|---|---|
| IF | DFL | FN | Avg. | IF | DFL | FN | Avg. | ||
| Detection | Faster R-CNN | 84.2 | 92.9 | 98.6 | 91.9 | 83.9 | 92.9 | 98.7 | 91.8 |
| Oriented R-CNN | 85.3 | 93.4 | 98.8 | 92.5 | 85.0 | 93.5 | 98.9 | 92.5 | |
| Segmentation | Mask R-CNN | 87.0 | 93.5 | 98.8 | 93.1 | 86.8 | 93.9 | 99.0 | 93.2 |
| Vanilla CondInst | 56.8 | 95.4 | 75.9 | 76.0 | 56.6 | 95.4 | 75.9 | 76.0 | |
| Mask2Former | 60.7 | 95.9 | 97.1 | 84.6 | 60.4 | 95.9 | 97.1 | 84.5 | |
| DynFormula | 89.5 | 96.0 | 99.2 | 94.9 | 89.3 | 95.9 | 99.4 | 94.9 | |
| Dataset | Methods | NFL (%) | FRN (%) | Avg. F1 (%) | ||||
|---|---|---|---|---|---|---|---|---|
| P | R | F1 | P | R | F1 | |||
| Val | X-Y Cut | 80.21 | 71.24 | 75.46 | 77.65 | 77.65 | 77.65 | 76.6 |
| Relation Network | 93.85 | 89.28 | 91.51 | 92.45 | 96.41 | 94.39 | 93.0 | |
| GCN | 94.44 | 90.34 | 92.34 | 95.02 | 95.31 | 95.16 | 93.8 | |
| RelFormer | 94.84 | 92.52 | 93.67 | 95.81 | 96.78 | 96.29 | 95.0 | |
| Test | X-Y Cut | 79.84 | 72.65 | 76.08 | 77.25 | 77.25 | 77.25 | 76.7 |
| Relation Network | 94.03 | 89.61 | 91.77 | 92.54 | 96.15 | 94.31 | 93.0 | |
| GCN | 94.59 | 91.60 | 93.07 | 95.21 | 95.19 | 95.20 | 94.1 | |
| RelFormer | 94.45 | 92.31 | 93.37 | 96.37 | 96.19 | 96.28 | 94.8 | |
| Dataset | Methods | IoU=0.7 (%) | IoU=0.8 (%) | IoU=0.9 (%) | Avg. F1 (%) | ||||||
|---|---|---|---|---|---|---|---|---|---|---|---|
| P | R | F1 | P | R | F1 | P | R | F1 | |||
| Val | Mask R-CNN | 94.1 | 94.1 | 94.1 | 87.5 | 87.5 | 87.5 | 62.2 | 62.4 | 62.3 | 81.3 |
| DynFormula | 94.3 | 94.7 | 94.5 | 88.1 | 90.0 | 89.0 | 68.3 | 70.7 | 69.5 | 84.3 | |
| FormulaDet | 94.9 | 96.1 | 95.5 | 93.4 | 94.6 | 94.0 | 80.4 | 80.8 | 80.6 | 90.0 | |
| Test | Mask R-CNN | 94.2 | 94.1 | 94.2 | 87.8 | 87.8 | 87.8 | 62.4 | 62.4 | 62.4 | 81.5 |
| DynFormula | 94.4 | 94.5 | 94.4 | 87.8 | 90.2 | 89.0 | 68.1 | 71.0 | 69.5 | 84.3 | |
| FormulaDet | 95.0 | 96.1 | 95.5 | 93.8 | 94.5 | 94.1 | 80.3 | 80.7 | 80.5 | 90.0 | |
This project welcomes contributions and suggestions. Most contributions require you to agree to a Contributor License Agreement (CLA) declaring that you have the right to, and actually do, grant us the rights to use your contribution. For details, visit https://cla.opensource.microsoft.com.
When you submit a pull request, a CLA bot will automatically determine whether you need to provide a CLA and decorate the PR appropriately (e.g., status check, comment). Simply follow the instructions provided by the bot. You will only need to do this once across all repos using our CLA.
This project has adopted the Microsoft Open Source Code of Conduct. For more information see the Code of Conduct FAQ or contact opencode@microsoft.com with any additional questions or comments.
This project may contain trademarks or logos for projects, products, or services. Authorized use of Microsoft trademarks or logos is subject to and must follow Microsoft's Trademark & Brand Guidelines. Use of Microsoft trademarks or logos in modified versions of this project must not cause confusion or imply Microsoft sponsorship. Any use of third-party trademarks or logos are subject to those third-party's policies.

