Recode with ISO-8859-2 input specified calls iconv with CP912, unsupported by libiconv

[oreo639]

While running the enca tests with recode 3.7.15 with libiconv, I noticed a test failure in test-recode.sh.

enca: librecode probably damaged `STDIN'. No way to recover in a pipe.
1d0
< nápor po»ouchlých Ÿlutých èíòanù pøehlu¹il i zuøící orkán
FAIL test-recode.sh (exit status: 1)

Running ../src/enca -L cs -C librecode -x UTF-8 <test-recode.actual >test-recode.tmp with verbose:

Adding converter `librecode'
Explicitly specified target charset: UTF-8
Initializing language cs
enca: converting `STDIN': ISO-8859-2..UTF-8
    trying to convert `STDIN' using librecode

Upon further investigation, I found that ISO-8859-2 was being transformed into CP912 and passed to libiconv (this happens with both libiconv and glibc iconv, although glibc defines it as an alias in the /usr/lib/gconv/gconv-modules config file).

There is code for defining CP912 as an alias for ISO-8859-2 in strip-data.c (generated from tables.py), not sure why that doesn't seem to be working here?
https://github.com/rrthomas/recode/blob/f49992e71463ad9f8ee543ce357af80fc99d23c1/tables.py

(gdb) run -VVVV -L cs -C librecode -x UTF-8 <test-recode.actual >test-recode.tmp
The program being debugged has been started already.
Start it from the beginning? (y or n) y
`/home/oreo639/git/IPR/void-packages/masterdir-x86_64/builddir/enca-1.19/src/.libs/enca' has changed; re-reading symbols.
Starting program: /home/oreo639/git/IPR/void-packages/masterdir-x86_64/builddir/enca-1.19/src/.libs/enca -VVVV -L cs -C librecode -x UTF-8 <test-recode.actual >test-recode.tmp
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/usr/lib64/libthread_db.so.1".
Adding converter `librecode'
Explicitly specified target charset: UTF-8
Initializing language cs
enca: converting `STDIN': ISO-8859-2..UTF-8
    trying to convert `STDIN' using librecode
encreq: ISO-8859-2/..UTF-8

Breakpoint 2, recode_scan_request (request=0x55555564dcb0, 
    string=0x55555564dc90 "ISO-8859-2/..UTF-8") at ./src/request.c:1156
warning: 1156	./src/request.c: No such file or directory
(gdb) c
Continuing.

Breakpoint 1, libiconv_open (tocode=tocode@entry=0x55555556afb0 "UTF-8", 
    fromcode=0x7ffff7f2a62c "CP912") at ./iconv.c:235
warning: 235	./iconv.c: No such file or directory
(gdb) bt
#0  libiconv_open (tocode=tocode@entry=0x55555556afb0 "UTF-8", 
    fromcode=0x7ffff7f2a62c "CP912") at ./iconv.c:235
#1  0x00007ffff7eeab9a in recode_transform_with_iconv (subtask=0x7fffffffe0d0)
    at ./src/iconv.c:219
#2  0x00007ffff7ef11dd in recode_perform_task (task=0x55555564dd40)
    at ./src/task.c:396
#3  0x000055555555ae12 in convert_recode ()
#4  0x000055555555a459 in convert ()
#5  0x000055555555b8d1 in process_file ()
#6  0x0000555555559724 in main ()

[oreo639]

After further investigation, in scan_charset_before(), recode_find_alias() returns an alias object where:

alias->name: ISO8859-2
alias->symbol->name: ISO-8859-2
alias->symbol->iconv_name: CP912

(where ISO-8859-2 becomes ISO8859-2 in disambiguate_name())

The culprit appears to be here, it assumes that whatever alias appears at the top of the list must be recognized by iconv:

recode/src/iconv.c

Lines 263 to 281 in f49992e

	if (!recode_declare_iconv (outer, charset_name, *aliases))
	return false;
	/* Declare all aliases, given they bring something we do not already
	know. Even then, we still declare too many useless aliases, as the
	disambiguating tables are not recomputed as we go. FIXME! */
	for (cursor = aliases; *cursor; cursor++)
	{
	RECODE_ALIAS alias
	= recode_find_alias (outer, *cursor, ALIAS_FIND_AS_CHARSET);
	/* If there is a charset contradiction, call recode_declare_alias
	nevertheless, as the error processing will occur there. */
	if (!alias || alias->symbol->name != charset_name)
	if (!recode_declare_alias (outer, *cursor, charset_name))
	return false;
	}
	}

Replacing *aliases with charset_name in recode_declare_iconv() does allow the test to pass.

Not sure how you want to handle this? (e.g. if there should be an extra parameter added to tables for the canonical iconv name?)

[rrthomas]

Thanks for the report and investigation.

I can see one immediate problem which is fixable, and some other things I don't understand yet. Here's the immediate problem and fix:

src/iconvdecl.h is in the source tarball. (Perhaps your recode was built using it.) But it is generated from the system's iconv (for you libiconv's, but for me, glibc's), so it should not be shipped in the source distribution. Fixed in git. To check this is not your problem, you can delete the file and rebuild.

[rrthomas]

I think I have understood a bit better what is going on now. The list of character sets understood by iconv is just that, a flat list. The code is somewhat confusing here: it can handle a list of aliases for a single charset, but there is no such list in the generated file iconvdecl.h; that list is strictly a list of literal strings followed by NULL.

Finally, in your case, and assuming I'm right and your recode was built with a glibc-generated list of code pages, what happens is that recode simply looks for the first iconv name for ISO-8859-2, and that will be CP912, which is the first in the list. Not ideal, but nothing wrong there.

Originally iconvdecl.h was generated from libiconv's encodings.def file. I plan to go back to using libiconv, to avoid the system-specific differences in iconv implementations that have plagued "my" versions of Recode, so I may revert to that on the static-iconv branch.

[oreo639]

Ah, thanks. After removing iconvdecl.h and regenerating it with the correct iconv executable, the previous error is gone, although now the test fails as recode never ends up performing recode_transform_with_iconv() and outputs nothing.

Adding printfs to simplify_sequence() both with glibc's iconvdecl.h and libiconv's, shows that outer->iconv_pivot remains the same, but in[0].after/in[1].before are different with libiconv's iconvdecl.h:
glibc

outer->iconv_pivot: 0x55555556b2a0
	name: :iconv:
	iconv_name: (null)
in[0].after: 0x55555556b2a0
	name: :iconv:
	iconv_name: (null)
in[1].before: 0x55555556b2a0
	name: :iconv:
	iconv_name: (null)

libiconv

outer->iconv_pivot: 0x55555556b2a0
	name: :iconv:
	iconv_name: (null)
in[0].after: 0x55555556b250
	name: ISO-10646-UCS-2
	iconv_name: ISO-10646-UCS-2
in[1].before: 0x55555556b250
	name: ISO-10646-UCS-2
	iconv_name: ISO-10646-UCS-2
in[0].after: 0x5555555934d0
	name: UTF-8
	iconv_name: UTF-8
in[1].before: (nil)

Here is the libiconv iconvdecl.h for reference: https://gist.github.com/oreo639/76f9e1c5c3ea944481ee61a423b08cf6

This gets set from find_sequence() in scan_charset_after().

[rrthomas]

Recode doesn't have to use iconv, so not calling recode_transform_with_iconv is not a big deal. However, outputting nothing from the conversion is obviously a problem. I might be able to reproduce using the static-iconv branch.

[rrthomas]

I copied the test file from enca, and I ran the following command:

recode ISO-8859-2..UTF-8 < test-recode.actual > test-recode.tmp

with both recode 3.7.15 built on my Ubuntu system, and recode from the static-iconv branch. They gave identical and apparently correct results.

So if you're getting no output, it looks like something is going wrong in the use of librecode by Enca, and it might be worth seeing for starters whether the task set up by recode's CLI is different from that set up by Enca.

I'm sorry, I hope you'll understand that I really prefer issue reports that only involve Recode (so in this case, maybe construct a MWE?) and not other packages (here, Enca).

[oreo639]

So if you're getting no output, it looks like something is going wrong in the use of librecode by Enca, and it might be worth seeing for starters whether the task set up by recode's CLI is different from that set up by Enca.

Sorry, after double checking, there is an output and it is mostly fine but ../src/enca -L cs -C librecode -x UTF-8 <test-recode.actual >test-recode.tmp ends up outputting the string twice.

And yeah, when running recode ISO-8859-2..UTF-8 <test-recode.actual, recode ISO-8859-2..UTF-8 <test-recode.actual >test-recode.tmp, or ../src/enca -L cs -C librecode -x UTF-8 <test-recode.actual that doesn't seem to happen.

[rrthomas]

Since enca is using librecode rather than recode directly, it's still possible there's a recode bug here! I'll close this issue, and if you can find a librecode issue, by all means file away! In the mean time, I'll make another point release without the offending source file.

[oreo639]

I'll keep looking. Thanks for all the help so far.

Edit: Alright, yeah, enca just calls recode_perform_task() twice unconditionally: https://github.com/nijel/enca/blob/1.19/src/convert_recode.c#L141-L201
The first call by itself returns the string once, the second call by itself returns the string zero times, but both calls in succession returns the string 2 times.

[oreo639]

Somehow, adding fflush(stdout); after both recode_perform_task() calls seems to fix the duplication issue?
Similarly, if I add if (subtask->output.file) fflush (subtask->output.file); right before here, it seems to work: https://github.com/rrthomas/recode/blob/master/src/task.c#L435

[rrthomas]

Many thanks for the further investigation, I'll look into it. Could you possibly re-file your last couple of comments as a new issue, as it's not really the same problem as this now-closed one!