gen_binary_files adds non-determinism to pinyin_index.bin

[bmwiedemann]

While working on reproducible builds for openSUSE, I found that our libpinyin package had
a pinyin_index.bin and addon_pinyin_index.bin that vary for every build, unless ASLR is disabled.

Steps to reproduce:

cd ~/rpmbuild/BUILD/libpinyin-2.8.1/data && for i in 1 2 ; do
   rm -f addon_pinyin_index.bin && ../utils/storage/gen_binary_files --table-dir ../data && md5sum addon_pinyin_index.bin ;
done

Actual result:
produces different results with roughly 16 bits of randomness.

Expected result:
output should be fully deterministic.

Here are two sets of samples:
https://rb.zq1.de/temp/pinyin_index.tar.xz

[epico]

Do you know some good tools for binary compare?

Thanks!

[bmwiedemann]

I use filterdiff as
filterdiff "hexdump -C" $file1 $file2
or
filterdiff "od -tx1 -Ax" $file1 $file2

[epico]

I think libpinyin uses Kyoto Cabinet now.

I guess the difference between pinyin_index.bin and phrase_index.bin is that:

• pinyin_index.bin aligned with 2 bytes
• phrase_index.bin aligned with 4 bytes

BTW, does Kyoto Cabinet support reproducible builds?

[bmwiedemann]

That 2-byte-alignment could cause an added 2 bytes of padding that remain uninitialized.
Do you know where the structs that get written into pinyin_index.bin get allocated?

[epico]

Maybe the code is in the ChewingLargeTable2::add_index_internal method of the chewing_large_table2_kyotodb.cpp file.