InDesign finalizations

[rigaspapas]

Checklist (for the reviewer)

• Problem and solution are well-explained in the PR
• Change is covered by unit-tests
• Code is well documented
• Code is styled well and is following best practices
• Performs well when applied to big organizations/resources/etc from our production database
• Errors are handled properly
• All (conceivable) edge-cases are handled
• Code is instrumented to report unexpected failures or increases in the failure rate
• Commits have been squashed so that each one has a clear purpose (and green tests)
• Proper labels have been applied

Problem

1. Empty strings appeared in the editor (when only non-printable characters were included)
2. Spaces around the text where not preserved

Steps to reproduce

1. Use the ifood IDML file from this doc
2. Use the TrustYouBranding file from the same doc and the recent research results string

Solution

1. Ignore strings that don't contain letters, symbols or punctuation (as defined here)
2. Extend the regular expression to ignore wrapping spaces

[kouk]

@rigaspapas why not just check for content after stripping all non-printable characters?
ref: https://mayart.de/download/Indesign-IDML/special-idml-chars.pdf

one idea would be to use the unicodedata module to ignore all character not in printable categories:
https://www.unicode.org/reports/tr44/#General_Category_Values

[kouk]

The reason I'm saying this is who's to say we don't find another file that e.g. has a u'\u200c' character?

[rigaspapas]

@kouk what we want to achieve is to avoid exporting text that is not translatable. There is no standard way in Unicode characters to distinguish letters from symbols. Is there any?

You are right that any non-printable character can cause the same unwanted result. I considered ignoring one specific character, because I guess that's what InDesign insert in every document automatically (not a user input).

[coveralls]

Coverage increased (+0.02%) to 95.63% when pulling 6f70b68 on TX-9144-skip-special-content into 1374e95 on devel.

[kouk]

there's a bug here that needs fixing (see my comment below about the missing test case). Ignore the other comment (unless you like oneliners)

[kouk]

here's another test case:

u'\ufeff<Br/>;'

;-)

[rigaspapas]

You are right. We should first strip the special characters and then check for the translatable ones. Thanks!

[kouk]

just for fun:

from six.moves import map
return any(c[0] in ["L", "P", "S"] for c in map(unicodedata.category, string))

or even

from six.moves import map
return any(map(["L", "P", "S"].__contains__,
               map(itemgetter(0), map(unicodedata.category, string))) 

[rigaspapas]

@kouk I really like the first approach, which is very functional, but it would require a new dependency. So, should we leave it as is?

[kouk]

sure, it's fine like it is. But these examples don't really require the new dependency, you could do from itertools import imap. It's just that this way it's compatible with both python2 and python3.

[kouk]

LGTM