Hungarian dictionary contains invalid UTF-8 sequences

[dimztimz]

The issues can be of three types (put an X between the brackets):

• Bug reports
• Change request or feature request
• Others, questions

Reporting bugs

When reporting a bug you must tell us the state of your system and the
steps to reproduce the bug. For the state please specify the following:

key	value
OS, distro, version =	Linux, Ubuntu
Hunspell version =	1.6.2
Dictionary,package name,version =	Hungarian

Steps to reproduce

Open hu_HU_u8.aff in gedit

sudo apt install hunspell-hu
gedit /usr/share/hunspell/hu_HU.aff --encoding=UTF-8

Bugged behavior (output)

Gedit shows error. If by any chance it tries to interpret the file as ISO-8859-15 open the file with --encoding option in gedit.

Expected behavior (output)

No error should be shown by the text editor. Valid UTF-8 is expected.

Solution

Invalid UTF appears only in comments and in flag vectors.

Upstream is here https://sourceforge.net/projects/magyarispell/ , open the source tarball.

The fix is in the file bin/u8myspell. The following script should fix it completely.

#!/bin/bash
set -x
export LANG=en_US
export LC_ALL=C

case $# in
0|1|2) echo "u8myspell - converts MySpell dictionaries to UTF-8
usage: u8myspell source_name output_name source_charset"; exit 1;;
esac

i=$1
o=$2
charset=$3
localdir="$(dirname $0)"

iconv -f "$charset" -t UTF-8 "$i.dic" | sed -f "$localdir"/l1_u8.sed > "$o.dic"
iconv -f "$charset" -t UTF-8 "$i.aff" |
sed 's/^SET .*$/SET UTF-8\
FLAG UTF-8/' | sed -f "$localdir"/l1_u8.sed > "$o.aff"

Basically the latin2 is converted to utf8 and the command FLAG UTF-8 is additionally issued in .aff.

[PanderMusubi]

I have a list of many encoding problems in dictionaries, affix files and word list files. I will send those to Debian so they will land in Ubuntu too. I will keep this issue in the loop.

[dimztimz]

Better place to forward the issues is the Libreoffice bug tracker because Ubuntu pulls the dictionaries from the Libreoffice repo.

[PanderMusubi]

Then LibreOffice pulls them from Debian. I will make sure all involved are informed.

[PanderMusubi]

Confirmed that this issue is still in version 1.7b5 from 11 March 2018.

Unfortunately, in the bug tracker on the SourceForge project page it is not possible to report bugs. Hence, assigning here to @laszlonemeth

Please, open bug tracker on SourceForge again.

[PanderMusubi]

Some other minor issues:

1. Add recode and gawk as package dependencies in the README in order to run make properly
2. How is the dictionary count on the first line of the .dic file determined? It is a bit too high. That is nog a problem, but if there is an error in the calculation, that might also result in a number that is too low. Hence request for a review or explanation of how this is calculated. Since the source code is also in Hungarian it is hard to pinpoint this. The dic file is (excluding the first line) 93826 lines but in the first line 93837 is reported.
3. The dic file also has an unnecessary empty line at the end.

[PanderMusubi]

Reported also at LibreOffice in https://bugs.documentfoundation.org/show_bug.cgi?id=117324

[laszlonemeth]

hu_HU.dic and hu_HU.aff file are not UTF-8 encoded files.

They contain UTF-8 encoded dictionary items (words and morphemes), and the default 8-bit flags, see hunspell (5) manual page for dictionary format.

The suggested conversion duplicates the memory footprint of the flag vectors, and decoding of the UTF-8 encoded flags slows down the dictionary loading by 70% (plain dic.) or 50% (alias compressed dic.), resulting noticeable differences in the user interface of LibreOffice.

[dimztimz]

1. the in ram memory footprint stays exactly the same, because flags are stored as 16-bit values in any case.
2. the file grow for ~ 1k bytes, not big deal
3. i'm not really sure on the slow downs you mention. Dictionary loading time is dominated by IO, not by decoding utf8.
4. Keeping invalid utf-8 might end up as a security issue some day.

[laszlonemeth]

1. You are right. It seems, I dropped it or haven't optimized it after introducing long flag support and alias compression.

2–3. The problem is the following difference (on my laptop with the recent Hunspell):

default 8-bit flag -> FLAG UTF-8
0.56 s -> 0.96 s (hu_HU_u8_gen)
0.60 s -> 0.90 s (hu_HU_u8_gen_alias)

for the test
echo x | time hunspell -d hu_HU on my laptop. Maybe numerical flags can show better results.

(Indeed, dictionary loading is not optimal especially for mobile platforms, etc. that is why Google Chrome has changed this area in its Hunspell fork.

By the way, with Hunspell 1.3.2, loading times are much better:

0.22 s -> 0.32 s)

4. Hunspell's file format is not an UTF-8 encoded text file in the case of SET UTF-8 with the default 8-bit FLAG.

It would be more comfortable to use more flags for Hungarian, so I plan to change to one of the long flag formats in the future, but without these performance regressions.

[dimztimz]

Seeing the decode_flags function, switching to LONG flags in the Hungarian dict will have pretty much zero performance impact. If you do that please make sure both chars of the flag are in ASCII range (0-127) so when the hu_HU.aff gets opened in text editor, we don't see lots of question marks.

[PanderMusubi]

@laszlonemeth , another detail I've noticed is the following on line numbers 61032 and 61033:

...
SMS-ezik/918    11402
#/425
/36
újra    9776
...

Probably these lines starting with / and # can be removed.