Use pdf tags for text extraction

[MartinThoma]

The comments in https://stackoverflow.com/a/75419068/562769 might be interesting @pubpub-zz

[pubpub-zz]

It is interesting about the /Span<</ActualText() >>
@naourass can you try to give an explanation about the cleaning:

When I decode the sequence I see no difference😯

s=b"\x06\x27\xFE\xDF\xFE\xE4\xFE\xEE\x06\x27\x06\x31\x06\x2F"
s.decode("utf-16-be")
'اﻟﻤﻮارد'
ss=b"\x06\x27\x06\x44\x06\x45\x06\x48\x06\x27\x06\x31\x06\x2F"
ss.decode("utf-16-be")
'الموارد'

[naourass]

@pubpub-zz
In Arabic, there's similar looking characters that has multiple unicodes. Look the letter Ba or Ra for example:

Often when these variants of the char are ligatured, they give the same visual result. And from what I experienced, sometimes just copying and pasting Arabic text in some editors may change the variant of some chars (maybe due to rich to plain text, normalization or some processing by the editor I'm not sure about the reason of the change).

[naourass]

BTW regarding the order, what I usually see in other text extraction/processing tools, is that they give you the logical order of the text and you need to call bidi.algorithm.get_display() to get the visual order.

https://unicode-org.github.io/icu/userguide/transforms/bidi.html
https://www.w3.org/International/articles/inline-bidi-markup/uba-basics

[naourass]

And for the cleaning, there's high chance that it could be just normalization.