Skip to content

This repo is used to release the ArxivFormula dataset.

License

Notifications You must be signed in to change notification settings

microsoft/ArxivFormula

Repository files navigation

ArxivFormula

ArxivFormula is the first dataset framing mathematical formula detection as a joint task of formula entity detection and formula relation extraction, rather than a simple task of object detection or instance segmentation. It's constructed using a weak supervision approach and comprises 500K document images for training, 50K for validation and 50K for testing.

News

  • We released annotations and origin document images of the ArxivFormula datasets (OneDrive), please refer to Get Data section.
  • We released some examples of the ArxivFormula datasets, please refer to ArxivFormula_examples.zip.

Introduction

Most of existing mathematical formula detectors focus on detecting formula entities through object detection or instance segmentation techniques. However, these methods often fail to convey complete messages due to the absence of the contextual and layout information of mathematical formulas. For a more comprehensive understanding of mathematical formulas in document images, it is preferable to detect logical formula blocks that include one or multiple formula entities arranged in their natural reading order. These logical formula blocks enable the transmission of complete contextual messages of mathematical formulas and aid in the reconstruction of layout information of the document images, resulting in a more accurate mathematical formula detection. To this end, ArxivFormula first presents a novel perspective on the problem of mathematical formula detection by framing it as a joint task of formula entity detection and formula relation extraction, rather than a simple task of object detection or instance segmentation. This new perspective enables us to detect logical formula blocks that convey complete contextual and layout information of mathematical formulas, while also eliminating labeling issues in existing benchmark datasets.

Comparisons among ArxivFormula and existing mathematical formula detection datasets.

Dataset #Documents #Images Formula Entity Formula Relation
IF DFL DFB FN NFL FRN
InftyCDB-1 30 467 21k - - - -
Marmot 194 400 7.9k 1.6k - - - -
ICDAR-2017 POD 1.5k 2.4k - 5.4k - - - -
TFD-ICDAR 2019 46 805 38k - - - -
ICDAR-2021 IBEM 600 8.3k 137k - 29k - - -
ArxivFormula 100k 600k 15M 1.9M 1.4M 795k 540k 795k

Note: (IF: Inline Formula, DFL: Displayed Formula Line, DFB: Displayed Formula Block, FN: Formula Number, NFL: Next Formula Line, FRN: Formula Reference Number.)

Get Data

**Please DO NOT re-distribute the dataset.**

The annotations and original document images of the ArxivFormula dataset can be download from the OneDrive. In order to reduce the loss caused by download interruption, we divided "ArxivFormula_Training_set_images.zip" into 10 parts, and after downloading all of them, use the decompression software to decompress them together. Additionally, we would like to note that we have only released the document images and annotations of arXiv papers that comply with the arXiv.org perpetual, non-exclusive license. This constitutes approximately 93% of the entire dataset.

File Size md5sum
ArxivFormula_Training_set_images.zip
[[1], [2], [3], [4], [5], [6], [7], [8], [9], [10]]
5.9 GB
/each part
-
ArxivFormula_Validation_set_images.zip 5.9 GB a0f944b4150940c1d3cf060ff4307d7f
ArxivFormula_Testing_set_images.zip 5.9 GB fc3b956bbbe5a20ae82e74b4a182951e
ArxivFormula_Annotations.zip 1.1 GB a5735ae789850a854b0326079e3c925f

The annotation format used is the standard COCO-style annotation format. More details about the dataset please refer to ArxivFormula_Details.

FormulaDet

We propose a new approach, called FormulaDet, to address these two sub-tasks simultaneously. FormulaDet first employs a dynamic convolution-based formula entity detector, named DynFormula, to detect formula entities. It then uses a multi-modal transformer-based relation extraction method, named RelFormer, to group these detected formula entities into logical formula blocks.

Results

Formula Entity Segmentation

Methods Validation Set (%) Testing Set (%)
IF DFL FN Avg. IF DFL FN Avg.
Detection Faster R-CNN 84.2 92.9 98.6 91.9 83.9 92.9 98.7 91.8
Oriented R-CNN 85.3 93.4 98.8 92.5 85.0 93.5 98.9 92.5
Segmentation Mask R-CNN 87.0 93.5 98.8 93.1 86.8 93.9 99.0 93.2
Vanilla CondInst 56.8 95.4 75.9 76.0 56.6 95.4 75.9 76.0
Mask2Former 60.7 95.9 97.1 84.6 60.4 95.9 97.1 84.5
DynFormula 89.5 96.0 99.2 94.9 89.3 95.9 99.4 94.9

Formula Relation Extraction

Dataset Methods NFL (%) FRN (%) Avg. F1 (%)
P R F1 P R F1
Val X-Y Cut 80.21 71.24 75.46 77.65 77.65 77.65 76.6
Relation Network 93.85 89.28 91.51 92.45 96.41 94.39 93.0
GCN 94.44 90.34 92.34 95.02 95.31 95.16 93.8
RelFormer 94.84 92.52 93.67 95.81 96.78 96.29 95.0
Test X-Y Cut 79.84 72.65 76.08 77.25 77.25 77.25 76.7
Relation Network 94.03 89.61 91.77 92.54 96.15 94.31 93.0
GCN 94.59 91.60 93.07 95.21 95.19 95.20 94.1
RelFormer 94.45 92.31 93.37 96.37 96.19 96.28 94.8

End-to-End Evaltion

Dataset Methods IoU=0.7 (%) IoU=0.8 (%) IoU=0.9 (%) Avg. F1 (%)
P R F1 P R F1 P R F1
Val Mask R-CNN 94.1 94.1 94.1 87.5 87.5 87.5 62.2 62.4 62.3 81.3
DynFormula 94.3 94.7 94.5 88.1 90.0 89.0 68.3 70.7 69.5 84.3
FormulaDet 94.9 96.1 95.5 93.4 94.6 94.0 80.4 80.8 80.6 90.0
Test Mask R-CNN 94.2 94.1 94.2 87.8 87.8 87.8 62.4 62.4 62.4 81.5
DynFormula 94.4 94.5 94.4 87.8 90.2 89.0 68.1 71.0 69.5 84.3
FormulaDet 95.0 96.1 95.5 93.8 94.5 94.1 80.3 80.7 80.5 90.0

Contributing

This project welcomes contributions and suggestions. Most contributions require you to agree to a Contributor License Agreement (CLA) declaring that you have the right to, and actually do, grant us the rights to use your contribution. For details, visit https://cla.opensource.microsoft.com.

When you submit a pull request, a CLA bot will automatically determine whether you need to provide a CLA and decorate the PR appropriately (e.g., status check, comment). Simply follow the instructions provided by the bot. You will only need to do this once across all repos using our CLA.

This project has adopted the Microsoft Open Source Code of Conduct. For more information see the Code of Conduct FAQ or contact opencode@microsoft.com with any additional questions or comments.

Trademarks

This project may contain trademarks or logos for projects, products, or services. Authorized use of Microsoft trademarks or logos is subject to and must follow Microsoft's Trademark & Brand Guidelines. Use of Microsoft trademarks or logos in modified versions of this project must not cause confusion or imply Microsoft sponsorship. Any use of third-party trademarks or logos are subject to those third-party's policies.

About

This repo is used to release the ArxivFormula dataset.

Resources

License

Code of conduct

Security policy

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published

Languages