GOST-34.11 tests fail on i686

[thmo]

In Fedora's latest mass rebuild with gcc7, the Botan package failed to rebuild on i686, due to a failing check. See https://bugzilla.redhat.com/show_bug.cgi?id=1423245 which has the full build logs attached.

This was with Botan 1.10.14, but the problem also shows up with 1.10.15.

This only happens on i686, builds on x86_64, ppc64, ppc64le, aarch64 and armv7hl succeeded.

Testing Hash Functions: ............GOST-34.11 test with provider core failed
ERROR: "GOST-34.11" failed test #1
GOST-34.11 test with provider core failed
ERROR: "GOST-34.11" failed test #2
GOST-34.11 test with provider core failed
ERROR: "GOST-34.11" failed test #3
GOST-34.11 test with provider core failed
ERROR: "GOST-34.11" failed test #4
GOST-34.11 test with provider core failed
ERROR: "GOST-34.11" failed test #5
GOST-34.11 test with provider core failed
ERROR: "GOST-34.11" failed test #6
GOST-34.11 test with provider core failed
ERROR: "GOST-34.11" failed test #7
GOST-34.11 test with provider core failed
ERROR: "GOST-34.11" failed test #8
GOST-34.11 test with provider core failed
ERROR: "GOST-34.11" failed test #9
GOST-34.11 test with provider core failed
ERROR: "GOST-34.11" failed test #10

[randombit]

Thanks for the report. I have not seen this issue before so it may be GCC 7 specific. I will look into it, but may be some time before I have a chance to build a GCC 7 binary from source and try things out.

[thmo]

Using mock, I can reproduce that locally on a Fedora system (basically mock sets up a chroot of fedora-rawhide-i386).

Any hints where to look first resp. how to get more verbose output from the testsuite?

[randombit]

Sorry it looks like the output you got is all that is available in the 1.10 tests.

Can you try to replicate using latest master? The code for GOST 34.11 has not really changed much between 1.10 and 2.0, so if this is a GCC 7 miscompilation problem it likely affects both versions, and the error output in newer versions should be more helpful.

Another useful test would be to build the hash example in 1.10, and try hashing an empty file using GOST-34.11 (expected hash 981E5F3CA30C841487830F84FB433E13AC1101569B9C13584AC483234CD656C0)

[thmo]

The hash is indeed wrong:

# ./hash GOST-34.11 empty
6C64FEE4914BC2AE5283B9AA6677FC2F76DCA508F504E475C5D8AA53999D5646  empty
# ./hash SHA-256 empty
E3B0C44298FC1C149AFBF4C8996FB92427AE41E4649B934CA495991B7852B855  empty

Will try to reproduce with 2.0 now.

[thmo]

Here's the output of botan-test.

For the record, that's 2.0.1, built with

make CXX="g++ -std=c++11  -O2 -g -pipe -Wall -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector-strong --param=ssp-buffer-size=4 -grecord-gcc-switches -specs=/usr/lib/rpm/redhat/redhat-hardened-cc1 -m32 -march=i686 -fasynchronous-unwind-tables"

If it helps, I can also try with HEAD.

BTW, src/lib/filters/threaded_fork.cpp is missing an #include <functional>, should I open a separate ticket for that?

[randombit]

Unfortunately I am unable to replicate with current master and a recent GCC 7 snapshot. Both 32-bit and 64-bit build passes all tests without problems. :/

$ g++-7-20170226 -v
Using built-in specs.
COLLECT_GCC=/usr/local/gcc-7-20170226/bin/g++-7-20170226
COLLECT_LTO_WRAPPER=/usr/local/gcc-7-20170226/libexec/gcc/x86_64-pc-linux-gnu/7.0.1/lto-wrapper
Target: x86_64-pc-linux-gnu
Configured with: ../gcc-7-20170226/configure --prefix=/usr/local/gcc-7-20170226 --program-suffix=-7-20170226 --enable-languages=c,c++
Thread model: posix
gcc version 7.0.1 20170226 (experimental) (GCC) 

[thmo]

Thanks for testing. I am still seeing the issue with this GCC

# LC_ALL=C g++ -v 
Using built-in specs.
COLLECT_GCC=g++
COLLECT_LTO_WRAPPER=/usr/libexec/gcc/i686-redhat-linux/7/lto-wrapper
Target: i686-redhat-linux
Configured with: ../configure --enable-bootstrap --enable-languages=c,c++,objc,obj-c++,fortran,ada,go,lto --prefix=/usr --mandir=/usr/share/man --infodir=/usr/share/info --with-bugurl=http://bugzilla.redhat.com/bugzilla --enable-shared --enable-threads=posix --enable-checking=release --enable-multilib --with-system-zlib --enable-__cxa_atexit --disable-libunwind-exceptions --enable-gnu-unique-object --enable-linker-build-id --with-gcc-major-version-only --with-linker-hash-style=gnu --enable-plugin --enable-initfini-array --with-isl --enable-libmpx --enable-gnu-indirect-function --with-tune=generic --with-arch=i686 --build=i686-redhat-linux
Thread model: posix
gcc version 7.0.1 20170225 (Red Hat 7.0.1-0.10) (GCC)

which is admittedly not as recent as yours ;) So I will wait for the next GCC update in Fedora and then re-check.

[thmo]

Still fails with gcc (GCC) 7.0.1 20170309 (Red Hat 7.0.1-0.12).

Further testing shows that the problem is only there if compiled with -O3. Tests work fine for -O2-compiled hash_gost3411.c.

[thmo]

There are 13 differences regarding the optimization flags between -O2 and -O3 for that version of GCC.

Didn't try all combinations, but it seems -ftree-slp-vectorize together with -fpeel-loops creates code that fails (cross-checked by using -O3 and switching these two flags off).

In order to file a bug with GCC we'd need to strip down the test case a little bit, I guess.

[randombit]

Thanks for tracking this down. Something about how the Arch distribution of GCC is built must be bypassing the bug you are encountering. The psi sub-algorithm in the GOST_34_11::compress_n function is IMO almost certainly where the miscompilation occurs.

I would bet to reproduce, take (from master) lines 123-210 of gost_3411.cpp, and copy xor_buf and copy_mem both in src/lib/utils/mem_ops.h, I suspect you will find a standalone function with just this code will produce different output depending on the compilation flags.

[thmo]

Sorry, was a bit short of time in the last months - will look into this again.

For now, sort of a status update, the issue is still present with gcc version 7.1.1 20170718 (Red Hat 7.1.1-6) (GCC) and Botan 2.1.0.

gost_3410_sign:
GOST 34.10-2001/EMSA1(GOST-34.11) signature generation ran 16 tests in 172.39 msec 4 FAILED
Failure 1: GOST 34.10-2001/EMSA1(GOST-34.11) signature generation KAT signature valid produced unexpected result '0' expected '1'
Failure 2: GOST 34.10-2001/EMSA1(GOST-34.11) signature generation unexpected result for generated signature matches KAT
Produced: 7A9A7CA4E232C6E03FB8BC8A2B5225A19C554F0BD6D32841C5D43D9B0BC83DD30965C38281A01D23FA5F5D332B339EB2602A564563C005335269A2E811520563
Expected: 60053488E6936975A4913083FE16A0CF620FA75732B563AB65B82D37A825BE7D0965C38281A01D23FA5F5D332B339EB2602A564563C005335269A2E811520563
XOR Diff: 1A9F482C04A1AF959B298C09D544856EFE5AE85CE4664BEAA06C10ACA3ED83AE0000000000000000000000000000000000000000000000000000000000000000 (121 bits different)
Failure 3: GOST 34.10-2001/EMSA1(GOST-34.11) signature generation KAT signature valid produced unexpected result '0' expected '1'
Failure 4: GOST 34.10-2001/EMSA1(GOST-34.11) signature generation unexpected result for generated signature matches KAT
Produced: 29AA2E1C61BEF70B1B85F3AD832C2CC4C0E688ECA4A764308472C14788D7BC071AFADA58E26D1FE08AB4D2E67BF0FF8508870B417D6EA1224FA98B3165D8A032
Expected: 5E39519E12B5208B3A0A1A1CDD44E2629F0F76A51DFED8484A8B67A2E46D5B871AFADA58E26D1FE08AB4D2E67BF0FF8508870B417D6EA1224FA98B3165D8A032
XOR Diff: 77937F82730BD780218FE9B15E68CEA65FE9FE49B959BC78CEF9A6E56CBAE7800000000000000000000000000000000000000000000000000000000000000000 (142 bits different)
Note 1: GOST 34.10-2001/EMSA1(GOST-34.11) signature generation Test #1 failed
Note 2: GOST 34.10-2001/EMSA1(GOST-34.11) signature generation Test #2 failed

[xyproto]

Trying to package the latest version of botan for Arch Linux (gcc 7), I get failing tests for gost_3411 and also these modules:

• emsa1 - wrong return value
• x509 - the test segfaults

[randombit]

@xyproto Can you post details? I develop on Arch (gcc 7.1.1) and haven't seen any problems.

[randombit]

@xyproto Also just tried an i386 build in case that was the issue, but everything passes there also. :(

[xyproto]

@randombit Sorry for the lack of details, I was in a rush. I will submit as a new issue and test on two different machines.

[randombit]

@thmo Can you try the change in 3e1312d? It fixes the problem for me on master and should also apply cleanly to 1.10

[thmo]

@randombit Indeed that workaround helps, thanks!

As noted earlier, I also tried to further isolate the issue, but that turned out to be not that easy, also due to time constraints.

[randombit]

Thanks for confirming, I will apply the fix to 1.10 branch also then.

[randombit]

@thmo I have backported to 1.10 branch. I am not bothering to push out a new release just for this but it will be fixed in a future release and you can ship the patch for now. Closing. Thanks for the report and sorry it took so long to track this down - I still don't really get why this seemingly only manifests on a 'native' 32-bit compiler but not a multilib style build, but whatever.