[prototype] compute orientation on segmentation map #1336

felixdittrich92 · 2023-10-02T14:51:42Z

This PR:

update page orientation detection (Now we calculate the orientation on the segmentation mask - much more robust - and will be more robust until all detection models are trained with rotation augmented samples)
update builder (now each page element contains the processed page image - easier for the user if they work with rotated documents for example to grab the orientation corrected page)
soft breaking change: now we can use the processed image directly for visualization - straightened pages are now also correctly displayed

Any feedback is super welcome 🤗

open todos:

check changed test again
profile memory usage if we store the raw image in the Page elements (everything looks fine)
improve builder tests

doctr/models/kie_predictor/pytorch.py

doctr/models/zoo.py

felixdittrich92 · 2023-10-04T14:34:05Z

@odulcy-mindee added for a first review (PR is not 100% final)

felixdittrich92 · 2023-10-04T14:37:04Z

@denivic #1283

codecov · 2023-10-04T14:54:27Z

Codecov Report

All modified and coverable lines are covered by tests ✅

Comparison is base (6d92df5) 95.81% compared to head (727099d) 95.76%.

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #1336      +/-   ##
==========================================
- Coverage   95.81%   95.76%   -0.05%     
==========================================
  Files         155      155              
  Lines        6948     6950       +2     
==========================================
- Hits         6657     6656       -1     
- Misses        291      294       +3

Flag	Coverage Δ
unittests	`95.76% <100.00%> (-0.05%)`	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

felixdittrich92 · 2023-10-05T11:15:08Z

doctr/models/detection/predictor/pytorch.py

+        seg_maps = [
+            pred.permute(1, 2, 0).detach().cpu().numpy() for batch in predicted_batches for pred in batch["out_map"]
+        ]
+        if return_maps:


return preds, seg_maps if return_maps else preds raises an issue

felixdittrich92 · 2023-10-05T11:18:15Z

doctr/models/kie_predictor/tensorflow.py

-        loc_preds = self.det_predictor(pages, **kwargs)
+            pages = [rotate_image(page, -angle, expand=False) for page, angle in zip(pages, origin_page_orientations)]
+            # Forward again to get predictions on straight pages
+            loc_preds = self.det_predictor(pages, **kwargs)  # type: ignore[assignment]


sec forward is more save atm
If we have a solution also for upside down and the results from the seg_maps leads to 99%+ correct results we can change this later on and rotate the loc preds instead of the forward

doctr/models/predictor/pytorch.py

felixdittrich92 · 2023-10-13T07:31:47Z

@charlesmindee @frgfm @odulcy-mindee
Some results:

TF:

orientation: 14.036239624023438, rnd: 14
orientation: -12.094757080078125, rnd: -12
orientation: -87.87889862060547, rnd: -88
orientation: -42.07927703857422, rnd: -42
orientation: 11.944175720214844, rnd: 12
orientation: -80.13418579101562, rnd: -80
orientation: 0, rnd: -2
orientation: 37.82828903198242, rnd: 38
orientation: 39.805572509765625, rnd: 40
orientation: 0, rnd: -2
orientation: 64.04123115539551, rnd: 64
orientation: 80.16643619537354, rnd: 80
orientation: 43.89118576049805, rnd: 44
orientation: -79.91940307617188, rnd: -80
orientation: -29.01908302307129, rnd: -29
orientation: 58.967790603637695, rnd: 59
orientation: -61.8629035949707, rnd: -62
orientation: 49.07487487792969, rnd: 49
orientation: -36.7241096496582, rnd: -37
orientation: -4.911787986755371, rnd: -5

PT:

orientation: -2.0901589393615723, rnd: -2
orientation: -45.0, rnd: -45
orientation: 0, rnd: 1
orientation: -38.1572265625, rnd: -38
orientation: 11.129188537597656, rnd: 11
orientation: -39.889583587646484, rnd: -40
orientation: -65.0372085571289, rnd: -66
orientation: 78.46537971496582, rnd: 78
orientation: 13.150947570800781, rnd: 13
orientation: -6.023993015289307, rnd: -6
orientation: -12.094757080078125, rnd: -12
orientation: -69.94390869140625, rnd: -70
orientation: 46.84761047363281, rnd: 47
orientation: -38.08047103881836, rnd: -38
orientation: 32.927860260009766, rnd: 33
orientation: 29.016590118408203, rnd: 29
orientation: 60.88385200500488, rnd: 61
orientation: 71.96571350097656, rnd: 72
orientation: -50.61758041381836, rnd: -51
orientation: 54.46232223510742, rnd: 54

felixdittrich92 · 2023-10-14T10:35:52Z

@felixdittrich92 todo: Update doc strings

felixdittrich92 · 2023-10-20T14:17:43Z

@charlesmindee @frgfm @odulcy-mindee Some results:

TF:

orientation: 14.036239624023438, rnd: 14
orientation: -12.094757080078125, rnd: -12
orientation: -87.87889862060547, rnd: -88
orientation: -42.07927703857422, rnd: -42
orientation: 11.944175720214844, rnd: 12
orientation: -80.13418579101562, rnd: -80
orientation: 0, rnd: -2
orientation: 37.82828903198242, rnd: 38
orientation: 39.805572509765625, rnd: 40
orientation: 0, rnd: -2
orientation: 64.04123115539551, rnd: 64
orientation: 80.16643619537354, rnd: 80
orientation: 43.89118576049805, rnd: 44
orientation: -79.91940307617188, rnd: -80
orientation: -29.01908302307129, rnd: -29
orientation: 58.967790603637695, rnd: 59
orientation: -61.8629035949707, rnd: -62
orientation: 49.07487487792969, rnd: 49
orientation: -36.7241096496582, rnd: -37
orientation: -4.911787986755371, rnd: -5

PT:

orientation: -2.0901589393615723, rnd: -2
orientation: -45.0, rnd: -45
orientation: 0, rnd: 1
orientation: -38.1572265625, rnd: -38
orientation: 11.129188537597656, rnd: 11
orientation: -39.889583587646484, rnd: -40
orientation: -65.0372085571289, rnd: -66
orientation: 78.46537971496582, rnd: 78
orientation: 13.150947570800781, rnd: 13
orientation: -6.023993015289307, rnd: -6
orientation: -12.094757080078125, rnd: -12
orientation: -69.94390869140625, rnd: -70
orientation: 46.84761047363281, rnd: 47
orientation: -38.08047103881836, rnd: -38
orientation: 32.927860260009766, rnd: 33
orientation: 29.016590118408203, rnd: 29
orientation: 60.88385200500488, rnd: 61
orientation: 71.96571350097656, rnd: 72
orientation: -50.61758041381836, rnd: -51
orientation: 54.46232223510742, rnd: 54

Changed to round see: #1283 (reply in thread)

felixdittrich92 · 2023-11-16T07:55:34Z

Only an example (if have tested it with different kinds of documents)

orientation angle: 21

after straighten:

doctr/models/_utils.py

doctr/models/kie_predictor/pytorch.py

doctr/models/predictor/pytorch.py

tests/tensorflow/test_models_zoo_tf.py

felixdittrich92 commented Oct 3, 2023

View reviewed changes

doctr/models/kie_predictor/pytorch.py Outdated Show resolved Hide resolved

felixdittrich92 commented Oct 3, 2023

View reviewed changes

doctr/models/kie_predictor/pytorch.py Outdated Show resolved Hide resolved

felixdittrich92 commented Oct 3, 2023

View reviewed changes

doctr/models/zoo.py Outdated Show resolved Hide resolved

felixdittrich92 requested a review from odulcy-mindee October 4, 2023 14:32

felixdittrich92 added this to the 0.7.1 milestone Oct 5, 2023

felixdittrich92 self-assigned this Oct 5, 2023

felixdittrich92 changed the title ~~[prototype/draft] compute orientation on seq map~~ [prototype] compute orientation on segmentation map Oct 5, 2023

felixdittrich92 commented Oct 5, 2023

View reviewed changes

doctr/models/predictor/pytorch.py Show resolved Hide resolved

felixdittrich92 requested review from frgfm and charlesmindee October 5, 2023 11:21

felixdittrich92 force-pushed the orientation-seq-prototype branch from 434c0b5 to 04bcc59 Compare October 12, 2023 09:01

felixdittrich92 force-pushed the orientation-seq-prototype branch from 04bcc59 to 54dd7b5 Compare October 13, 2023 13:34

felixT2K mentioned this pull request Oct 13, 2023

TF change antialias to true #1348

Merged

felixdittrich92 marked this pull request as ready for review October 14, 2023 10:34

felixdittrich92 mentioned this pull request Oct 30, 2023

[models] TF & PT: Retrain all detection models with random rotation augmentation #1364

Closed

11 tasks

felixdittrich92 force-pushed the orientation-seq-prototype branch from 4531652 to 4b30b29 Compare November 10, 2023 07:42

felixT2K mentioned this pull request Nov 14, 2023

Fixed DivisionByZeroError issue with contours having 0 height or width by excluding them #1380

Closed

felixdittrich92 added 11 commits November 15, 2023 10:00

[skip ci] compute orientation on seq map

dfe09eb

[skip ci] some updates and fixes

eeb63ec

update tests

a21c4af

update some parts

6183a44

more checks and tests

768e00e

correct median

2e2f397

rebase

13b90cd

round angle to int

0cbadb5

update doc string

54e9604

rebase

3fc93cf

rebase

d5c23b3

felixdittrich92 force-pushed the orientation-seq-prototype branch from 4b30b29 to d5c23b3 Compare November 15, 2023 09:19

odulcy-mindee reviewed Nov 16, 2023

View reviewed changes

felixdittrich92 requested a review from odulcy-mindee November 16, 2023 10:21

felixdittrich92 added 2 commits November 16, 2023 15:13

fix docstrings

375486d

apply suggestion

727099d

odulcy-mindee approved these changes Nov 17, 2023

View reviewed changes

felixdittrich92 merged commit e645ead into mindee:main Nov 17, 2023
66 of 67 checks passed

felixdittrich92 deleted the orientation-seq-prototype branch November 17, 2023 10:53

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[prototype] compute orientation on segmentation map #1336

[prototype] compute orientation on segmentation map #1336

felixdittrich92 commented Oct 2, 2023 •

edited

felixdittrich92 commented Oct 4, 2023

felixdittrich92 commented Oct 4, 2023 •

edited

codecov bot commented Oct 4, 2023 •

edited

felixdittrich92 Oct 5, 2023 •

edited

felixdittrich92 Oct 5, 2023

felixdittrich92 commented Oct 13, 2023

felixdittrich92 commented Oct 14, 2023

felixdittrich92 commented Oct 20, 2023

felixdittrich92 commented Nov 16, 2023

[prototype] compute orientation on segmentation map #1336

[prototype] compute orientation on segmentation map #1336

Conversation

felixdittrich92 commented Oct 2, 2023 • edited

felixdittrich92 commented Oct 4, 2023

felixdittrich92 commented Oct 4, 2023 • edited

codecov bot commented Oct 4, 2023 • edited

Codecov Report

felixdittrich92 Oct 5, 2023 • edited

Choose a reason for hiding this comment

felixdittrich92 Oct 5, 2023

Choose a reason for hiding this comment

felixdittrich92 commented Oct 13, 2023

felixdittrich92 commented Oct 14, 2023

felixdittrich92 commented Oct 20, 2023

felixdittrich92 commented Nov 16, 2023

felixdittrich92 commented Oct 2, 2023 •

edited

felixdittrich92 commented Oct 4, 2023 •

edited

codecov bot commented Oct 4, 2023 •

edited

felixdittrich92 Oct 5, 2023 •

edited