Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

po2prop: index out of range error with UTF-16 files #3706

Open
katanagari7c1 opened this issue Nov 16, 2017 · 1 comment
Open

po2prop: index out of range error with UTF-16 files #3706

katanagari7c1 opened this issue Nov 16, 2017 · 1 comment

Comments

@katanagari7c1
Copy link

I've just created a repo with the files required to reproduce the error:

https://github.com/katanagari7c1/po2prop-out-of-range

Note that the template file contains an UTF-16 character. This file was generated by XCode's ibtool and once I removed this character everything works fine.
Running this command with --errorlevel=traceback gives me this trace:

Traceback (most recent call last):

 File "/usr/local/Cellar/translate-toolkit/2.2.5/libexec/lib/python2.7/site-packages/translate/misc/optrecurse.py", line 536, in recursiveprocess
   fulltemplatepath)

 File "/usr/local/Cellar/translate-toolkit/2.2.5/libexec/lib/python2.7/site-packages/translate/convert/convert.py", line 190, in processfile
   fulltemplatepath)

 File "/usr/local/Cellar/translate-toolkit/2.2.5/libexec/lib/python2.7/site-packages/translate/misc/optrecurse.py", line 592, in processfile
   **passthroughoptions):

 File "/usr/local/Cellar/translate-toolkit/2.2.5/libexec/lib/python2.7/site-packages/translate/convert/po2prop.py", line 203, in convertstrings
   remove_untranslated=remove_untranslated)

 File "/usr/local/Cellar/translate-toolkit/2.2.5/libexec/lib/python2.7/site-packages/translate/convert/po2prop.py", line 230, in convertprop
   outputprop = convertor.convertstore(includefuzzy)

 File "/usr/local/Cellar/translate-toolkit/2.2.5/libexec/lib/python2.7/site-packages/translate/convert/po2prop.py", line 94, in convertstore
   outputstr = self.convertline(line)

 File "/usr/local/Cellar/translate-toolkit/2.2.5/libexec/lib/python2.7/site-packages/translate/convert/po2prop.py", line 152, in convertline
   delimiter_char, delimiter_pos = self.personality.find_delimiter(line)

 File "/usr/local/Cellar/translate-toolkit/2.2.5/libexec/lib/python2.7/site-packages/translate/storage/properties.py", line 283, in find_delimiter
   if cls.key_wrap_char != '' and line[start_pos] == cls.key_wrap_char:

IndexError: string index out of range

As you can see, I reproduced this bug with version 2.2.5. However, this didn't happen in version 1.12.0

@katanagari7c1 katanagari7c1 changed the title po2prop: index out of range error with utf-16 files po2prop: index out of range error with UTF-16 files Nov 16, 2017
@Artoria2e5
Copy link
Contributor

Artoria2e5 commented Dec 10, 2017

Note: the problematic file is Greeting.strings.

$ hexdump -C Greeting.strings.txt
00000000  ff fe 0a 00 2f 00 2a 00  20 00 4a 00 75 00 73 00  |..../.*. .J.u.s.|
00000010  74 00 20 00 61 00 20 00  67 00 72 00 65 00 65 00  |t. .a. .g.r.e.e.|
00000020  74 00 69 00 6e 00 67 00  20 00 2a 00 2f 00 0a 00  |t.i.n.g. .*./...|
00000030  22 00 74 00 65 00 73 00  74 00 5f 00 67 00 72 00  |".t.e.s.t._.g.r.|
00000040  65 00 65 00 74 00 69 00  6e 00 67 00 22 00 20 00  |e.e.t.i.n.g.". .|
00000050  3d 00 20 00 22 00 22 00  3b 00 0a 00              |=. .".".;...|

Looks like UTF-16LE with BOM. Perhaps the decoder was doing The Right Thing when it skipped the BOM, causing some trouble with indexing.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants