Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: add rotated boxes #281

Merged
merged 52 commits into from
Jun 2, 2021
Merged

feat: add rotated boxes #281

merged 52 commits into from
Jun 2, 2021

Conversation

charlesmindee
Copy link
Collaborator

@charlesmindee charlesmindee commented May 25, 2021

This PR introduces a major feature: rotated bounding boxes.

Boxes were caracterized by a 4-dimension tuple: (xmin, ymin, xmax, ymax), and we introduce a new kind of rotated boxes caracterized by a 5-dimension tuple: (x, y, w, h, alpha), where (x, y) is the center of the box, (w, h) its spatial extension, and alpha the angle of rotation of the box (in degrees) to the horizontal (horizontal box: alpha = 0°).

For the moment, support is only provided for 2 quarters of the circle: only angles of rotation between -90° and +90° are supported (either up-rotated or down-rotated). As shown below, it is impossible to determine more than 1 quarter of 360° without any postprocessing tricks, however, we used the mean aspect ratio of boxes to extend this area to 180°.

However, this allows us to handle most of the rotated documents, which are usually slightly rotated with respect to the horizontal.

The following modules are modified accordingly:

  • datasets (load rotated boxes for both training & eval)
  • models (detection postprocessors)
  • utils (geometry, metrics, visualization)
  • tests
  • docs (new performances, due to a new detection postprocessing)

The 2 options for boxes geometry are available in each modules, and the "non-rotated" option is activated by default everywhere.
To instantiate a predictor with rotated boxes, please use:

predictor = models.OCRPredictor(rotated_detection_predictor, recognition_predictor, rotated_bbox=True)

where:

rotated_detection_predictor is a detection predictor instantiated with an architecture trained on rotated docs ('rotated_bbox' = True in cfg). However, you can also force any architecture to predict rotated boxes, passing a kwarg "rotated_bbox"=True in the constructor:

rotated_detection_predictor = detection.DetectionPredictor(PreProcessor(output_size=(512, 512), batch_size=batch_size), detection.db_resnet50(rotated_bbox=True, input_shape=(512, 512, 3)))

To plot rotated boxes, use:
page.show(img, rotation=True) or document.show([img1, img2, ...], rotation=False)

If you want to train or evaluate a model with rotated boxes, please pass the argument rotation = True in the scripts evaluate.py and detection/train.py, it will instantiate datasets and metrics properly.

Any feedback is welcome!

sample

@charlesmindee charlesmindee added topic: documentation Improvements or additions to documentation module: models Related to doctr.models module: utils Related to doctr.utils ext: tests Related to tests folder type: breaking change Introducing a breaking change module: datasets Related to doctr.datasets labels May 25, 2021
@charlesmindee charlesmindee added this to the 0.3.0 milestone May 25, 2021
@charlesmindee charlesmindee self-assigned this May 25, 2021
@charlesmindee charlesmindee changed the title Rotate feat: add rotated boxes May 25, 2021
Copy link
Contributor

@fg-mindee fg-mindee left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks a lot for the massive PR! A few things to change, let me know if you have questions!

doctr/datasets/cord.py Outdated Show resolved Hide resolved
doctr/datasets/cord.py Outdated Show resolved Hide resolved
doctr/datasets/detection.py Outdated Show resolved Hide resolved
doctr/datasets/detection.py Outdated Show resolved Hide resolved
doctr/datasets/funsd.py Show resolved Hide resolved
test/test_utils_metrics.py Outdated Show resolved Hide resolved
test/test_utils_geometry.py Outdated Show resolved Hide resolved
test/test_models_detection.py Outdated Show resolved Hide resolved
test/test_models_detection.py Show resolved Hide resolved
test/test_models.py Outdated Show resolved Hide resolved
Copy link
Contributor

@fg-mindee fg-mindee left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Few things to handle and we're good to go!

doctr/documents/elements.py Outdated Show resolved Hide resolved
doctr/models/detection/differentiable_binarization.py Outdated Show resolved Hide resolved
doctr/models/detection/differentiable_binarization.py Outdated Show resolved Hide resolved
doctr/models/detection/differentiable_binarization.py Outdated Show resolved Hide resolved
doctr/models/detection/differentiable_binarization.py Outdated Show resolved Hide resolved
doctr/models/detection/linknet.py Outdated Show resolved Hide resolved
doctr/models/detection/linknet.py Outdated Show resolved Hide resolved
doctr/utils/metrics.py Outdated Show resolved Hide resolved
references/detection/train.py Show resolved Hide resolved
Copy link
Contributor

@fg-mindee fg-mindee left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Almost there!

doctr/documents/elements.py Outdated Show resolved Hide resolved
Copy link
Contributor

@fg-mindee fg-mindee left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

All good!

@charlesmindee charlesmindee merged commit 6933091 into main Jun 2, 2021
@charlesmindee charlesmindee deleted the rotate branch June 2, 2021 11:25
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
ext: tests Related to tests folder module: datasets Related to doctr.datasets module: models Related to doctr.models module: utils Related to doctr.utils topic: documentation Improvements or additions to documentation type: breaking change Introducing a breaking change
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

2 participants