Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

32-bit test failure: test/crypto_tests.cpp(561): error: in "crypto_tests/sha256d64": check memcmp(out1, out2, 32 * i) == 0 has failed #14580

Closed
error10 opened this issue Oct 26, 2018 · 23 comments

Comments

@error10
Copy link
Contributor

commented Oct 26, 2018

A test in the bitcoin test suite is failing for me while running the sha256d64 test cases in crypto_tests.cpp, but only on 32-bit distro. It passes on 64-bit.

I expected the test suite to pass.

The test suite log is:

$ cat test/crypto_tests.cpp.log
Running 11 test cases...
Entering test module "Bitcoin Test Suite"
test/crypto_tests.cpp(23): Entering test suite "crypto_tests"
test/crypto_tests.cpp(217): Entering test case "ripemd160_testvectors"
test/crypto_tests.cpp(217): Leaving test case "ripemd160_testvectors"; testing time: 343157us
test/crypto_tests.cpp(233): Entering test case "sha1_testvectors"
test/crypto_tests.cpp(233): Leaving test case "sha1_testvectors"; testing time: 200719us
test/crypto_tests.cpp(249): Entering test case "sha256_testvectors"
test/crypto_tests.cpp(249): Leaving test case "sha256_testvectors"; testing time: 518373us
test/crypto_tests.cpp(271): Entering test case "sha512_testvectors"
test/crypto_tests.cpp(271): Leaving test case "sha512_testvectors"; testing time: 1071278us
test/crypto_tests.cpp(308): Entering test case "hmac_sha256_testvectors"
test/crypto_tests.cpp(308): Leaving test case "hmac_sha256_testvectors"; testing time: 10312us
test/crypto_tests.cpp(361): Entering test case "hmac_sha512_testvectors"
test/crypto_tests.cpp(361): Leaving test case "hmac_sha512_testvectors"; testing time: 22518us
test/crypto_tests.cpp(429): Entering test case "aes_testvectors"
test/crypto_tests.cpp(429): Leaving test case "aes_testvectors"; testing time: 3358us
test/crypto_tests.cpp(445): Entering test case "aes_cbc_testvectors"
test/crypto_tests.cpp(445): Leaving test case "aes_cbc_testvectors"; testing time: 4206us
test/crypto_tests.cpp(497): Entering test case "chacha20_testvector"
test/crypto_tests.cpp(497): Leaving test case "chacha20_testvector"; testing time: 3051us
test/crypto_tests.cpp(527): Entering test case "countbits_tests"
test/crypto_tests.cpp(527): Leaving test case "countbits_tests"; testing time: 63452us
test/crypto_tests.cpp(549): Entering test case "sha256d64"
test/crypto_tests.cpp(561): error: in "crypto_tests/sha256d64": check memcmp(out1, out2, 32 * i) == 0 has failed
test/crypto_tests.cpp(561): error: in "crypto_tests/sha256d64": check memcmp(out1, out2, 32 * i) == 0 has failed
test/crypto_tests.cpp(561): error: in "crypto_tests/sha256d64": check memcmp(out1, out2, 32 * i) == 0 has failed
test/crypto_tests.cpp(561): error: in "crypto_tests/sha256d64": check memcmp(out1, out2, 32 * i) == 0 has failed
test/crypto_tests.cpp(561): error: in "crypto_tests/sha256d64": check memcmp(out1, out2, 32 * i) == 0 has failed
test/crypto_tests.cpp(561): error: in "crypto_tests/sha256d64": check memcmp(out1, out2, 32 * i) == 0 has failed
test/crypto_tests.cpp(561): error: in "crypto_tests/sha256d64": check memcmp(out1, out2, 32 * i) == 0 has failed
test/crypto_tests.cpp(561): error: in "crypto_tests/sha256d64": check memcmp(out1, out2, 32 * i) == 0 has failed
test/crypto_tests.cpp(561): error: in "crypto_tests/sha256d64": check memcmp(out1, out2, 32 * i) == 0 has failed
test/crypto_tests.cpp(561): error: in "crypto_tests/sha256d64": check memcmp(out1, out2, 32 * i) == 0 has failed
test/crypto_tests.cpp(561): error: in "crypto_tests/sha256d64": check memcmp(out1, out2, 32 * i) == 0 has failed
test/crypto_tests.cpp(561): error: in "crypto_tests/sha256d64": check memcmp(out1, out2, 32 * i) == 0 has failed
test/crypto_tests.cpp(561): error: in "crypto_tests/sha256d64": check memcmp(out1, out2, 32 * i) == 0 has failed
test/crypto_tests.cpp(561): error: in "crypto_tests/sha256d64": check memcmp(out1, out2, 32 * i) == 0 has failed
test/crypto_tests.cpp(561): error: in "crypto_tests/sha256d64": check memcmp(out1, out2, 32 * i) == 0 has failed
test/crypto_tests.cpp(561): error: in "crypto_tests/sha256d64": check memcmp(out1, out2, 32 * i) == 0 has failed
test/crypto_tests.cpp(561): error: in "crypto_tests/sha256d64": check memcmp(out1, out2, 32 * i) == 0 has failed
test/crypto_tests.cpp(561): error: in "crypto_tests/sha256d64": check memcmp(out1, out2, 32 * i) == 0 has failed
test/crypto_tests.cpp(561): error: in "crypto_tests/sha256d64": check memcmp(out1, out2, 32 * i) == 0 has failed
test/crypto_tests.cpp(561): error: in "crypto_tests/sha256d64": check memcmp(out1, out2, 32 * i) == 0 has failed
test/crypto_tests.cpp(561): error: in "crypto_tests/sha256d64": check memcmp(out1, out2, 32 * i) == 0 has failed
test/crypto_tests.cpp(561): error: in "crypto_tests/sha256d64": check memcmp(out1, out2, 32 * i) == 0 has failed
test/crypto_tests.cpp(561): error: in "crypto_tests/sha256d64": check memcmp(out1, out2, 32 * i) == 0 has failed
test/crypto_tests.cpp(561): error: in "crypto_tests/sha256d64": check memcmp(out1, out2, 32 * i) == 0 has failed
test/crypto_tests.cpp(561): error: in "crypto_tests/sha256d64": check memcmp(out1, out2, 32 * i) == 0 has failed
test/crypto_tests.cpp(561): error: in "crypto_tests/sha256d64": check memcmp(out1, out2, 32 * i) == 0 has failed
test/crypto_tests.cpp(561): error: in "crypto_tests/sha256d64": check memcmp(out1, out2, 32 * i) == 0 has failed
test/crypto_tests.cpp(561): error: in "crypto_tests/sha256d64": check memcmp(out1, out2, 32 * i) == 0 has failed
test/crypto_tests.cpp(561): error: in "crypto_tests/sha256d64": check memcmp(out1, out2, 32 * i) == 0 has failed
test/crypto_tests.cpp(561): error: in "crypto_tests/sha256d64": check memcmp(out1, out2, 32 * i) == 0 has failed
test/crypto_tests.cpp(561): error: in "crypto_tests/sha256d64": check memcmp(out1, out2, 32 * i) == 0 has failed
test/crypto_tests.cpp(561): error: in "crypto_tests/sha256d64": check memcmp(out1, out2, 32 * i) == 0 has failed
test/crypto_tests.cpp(549): Leaving test case "sha256d64"; testing time: 5115us
test/crypto_tests.cpp(23): Leaving test suite "crypto_tests"; testing time: 2246048us
Leaving test module "Bitcoin Test Suite"; testing time: 2246167us

*** 32 failures are detected in the test module "Bitcoin Test Suite"

This is occurring on Fedora 29 i686 while building bitcoin. The builder has 4GB RAM. This environment includes boost 1.66.0 and openssl 1.1.1. The test passes while building bitcoin on Fedora 29 x86_64 on the same machine. The test also passed on Fedora 28 i686, which has boost 1.66.0 and openssl 1.1.0i.

This is bitcoin 0.17.0 obtained from thhe github release tagged v0.17.0.

I glanced at the test but I didn't see anything obvious that might cause it to fail on a 32-bit system. I would appreciate a hint as to where to look next.

@fanquake fanquake added the Tests label Oct 26, 2018

@MarcoFalke

This comment has been minimized.

Copy link
Member

commented Oct 26, 2018

Could you also check if branch 16 and branch master are affected, please?

@error10

This comment has been minimized.

Copy link
Contributor Author

commented Oct 26, 2018

Could you also check if branch 16 and branch master are affected, please?

The test passes on v0.16.3, and fails identically as above on master. But it turns out this test is not present in 0.16.3 anyway.

@sipa

This comment has been minimized.

Copy link
Member

commented Oct 26, 2018

Do you have the same issue with release binaries?

@error10

This comment has been minimized.

Copy link
Contributor Author

commented Oct 26, 2018

Do you have the same issue with release binaries?

Good call. I downloaded bitcoin-0.17.0-i686-pc-linux-gnu.tar.gz, and its included bin/test_bitcoin passes. So, I guess I don't have the same issue with release binaries.

@MarcoFalke

This comment has been minimized.

Copy link
Member

commented Nov 2, 2018

I believe this also happens 32bit Ubuntu Cosmic.

@error10

This comment has been minimized.

Copy link
Contributor Author

commented Nov 2, 2018

I believe this also happens 32bit Ubuntu Cosmic.

I don't just believe. I tested it, and sure enough it blows up on Ubuntu 18.10 i386, identically as above.

Cosmic also has openssl 1.1.1, so perhaps it's something specific to 1.1.1? A 32-bit openssl bug maybe?

@luke-jr

This comment has been minimized.

@pkepler

This comment has been minimized.

Copy link

commented Jan 16, 2019

I am running 0.15.1 that won't upgrade on a raspberry pi 3 B. If I can help in any way with testing I would be glad to.

@error10

This comment has been minimized.

Copy link
Contributor Author

commented Jan 16, 2019

I am running 0.15.1 that won't upgrade on a raspberry pi 3 B. If I can help in any way with testing I would be glad to.

You can just run a 64-bit distro on the RPi 3. Until Raspbian goes 64-bit, Fedora is probably your best bet. And then you can just pick up my aarch64 repo if you want.

@MarcoFalke

This comment has been minimized.

Copy link
Member

commented Feb 13, 2019

As a workaround you could build from our ./depends (which comes with an older openssl)

@error10

This comment has been minimized.

Copy link
Contributor Author

commented Feb 13, 2019

If 32-bit support were officially just dropped, I wouldn't complain. Aside from ancient Raspberry Pi 1 and 2s and the occasional antique netbook, there's virtually no hardware out there these days that requires a 32-bit OS. And the resource requirements of Bitcoin seem unlikely to shrink in the future.

@MarcoFalke

This comment has been minimized.

Copy link
Member

commented Feb 13, 2019

Oh, we didn't drop support for 32-bit. This is just a bug introduced in a newer version of openssl, but no one looked into it to understand what happened or how it could be worked around.

@MarcoFalke

This comment has been minimized.

Copy link
Member

commented May 2, 2019

I am seeing this running the gitian build (which is built with depends) on ubuntu_xenial_armhf ... 😩

@MarcoFalke

This comment has been minimized.

Copy link
Member

commented May 2, 2019

Steps to reproduce on fresh Ubuntu Bionic 64-bit:

dpkg --add-architecture i386
apt update && apt install wget libc6:i386 libncurses5:i386 libstdc++6:i386
wget https://bitcoincore.org/bin/bitcoin-core-0.18.0/bitcoin-0.18.0-i686-pc-linux-gnu.tar.gz
tar -xzf bitcoin-0.18.0-i686-pc-linux-gnu.tar.gz
./bitcoin-0.18.0/bin/test_bitcoin -t crypto_tests
Running 11 test cases...
test/crypto_tests.cpp(561): error: in "crypto_tests/sha256d64": check memcmp(out1, out2, 32 * i) == 0 has failed
test/crypto_tests.cpp(561): error: in "crypto_tests/sha256d64": check memcmp(out1, out2, 32 * i) == 0 has failed
test/crypto_tests.cpp(561): error: in "crypto_tests/sha256d64": check memcmp(out1, out2, 32 * i) == 0 has failed
test/crypto_tests.cpp(561): error: in "crypto_tests/sha256d64": check memcmp(out1, out2, 32 * i) == 0 has failed
test/crypto_tests.cpp(561): error: in "crypto_tests/sha256d64": check memcmp(out1, out2, 32 * i) == 0 has failed
test/crypto_tests.cpp(561): error: in "crypto_tests/sha256d64": check memcmp(out1, out2, 32 * i) == 0 has failed
test/crypto_tests.cpp(561): error: in "crypto_tests/sha256d64": check memcmp(out1, out2, 32 * i) == 0 has failed
test/crypto_tests.cpp(561): error: in "crypto_tests/sha256d64": check memcmp(out1, out2, 32 * i) == 0 has failed
test/crypto_tests.cpp(561): error: in "crypto_tests/sha256d64": check memcmp(out1, out2, 32 * i) == 0 has failed
test/crypto_tests.cpp(561): error: in "crypto_tests/sha256d64": check memcmp(out1, out2, 32 * i) == 0 has failed
test/crypto_tests.cpp(561): error: in "crypto_tests/sha256d64": check memcmp(out1, out2, 32 * i) == 0 has failed
test/crypto_tests.cpp(561): error: in "crypto_tests/sha256d64": check memcmp(out1, out2, 32 * i) == 0 has failed
test/crypto_tests.cpp(561): error: in "crypto_tests/sha256d64": check memcmp(out1, out2, 32 * i) == 0 has failed
test/crypto_tests.cpp(561): error: in "crypto_tests/sha256d64": check memcmp(out1, out2, 32 * i) == 0 has failed
test/crypto_tests.cpp(561): error: in "crypto_tests/sha256d64": check memcmp(out1, out2, 32 * i) == 0 has failed
test/crypto_tests.cpp(561): error: in "crypto_tests/sha256d64": check memcmp(out1, out2, 32 * i) == 0 has failed
test/crypto_tests.cpp(561): error: in "crypto_tests/sha256d64": check memcmp(out1, out2, 32 * i) == 0 has failed
test/crypto_tests.cpp(561): error: in "crypto_tests/sha256d64": check memcmp(out1, out2, 32 * i) == 0 has failed
test/crypto_tests.cpp(561): error: in "crypto_tests/sha256d64": check memcmp(out1, out2, 32 * i) == 0 has failed
test/crypto_tests.cpp(561): error: in "crypto_tests/sha256d64": check memcmp(out1, out2, 32 * i) == 0 has failed
test/crypto_tests.cpp(561): error: in "crypto_tests/sha256d64": check memcmp(out1, out2, 32 * i) == 0 has failed
test/crypto_tests.cpp(561): error: in "crypto_tests/sha256d64": check memcmp(out1, out2, 32 * i) == 0 has failed
test/crypto_tests.cpp(561): error: in "crypto_tests/sha256d64": check memcmp(out1, out2, 32 * i) == 0 has failed
test/crypto_tests.cpp(561): error: in "crypto_tests/sha256d64": check memcmp(out1, out2, 32 * i) == 0 has failed
test/crypto_tests.cpp(561): error: in "crypto_tests/sha256d64": check memcmp(out1, out2, 32 * i) == 0 has failed
test/crypto_tests.cpp(561): error: in "crypto_tests/sha256d64": check memcmp(out1, out2, 32 * i) == 0 has failed
test/crypto_tests.cpp(561): error: in "crypto_tests/sha256d64": check memcmp(out1, out2, 32 * i) == 0 has failed
test/crypto_tests.cpp(561): error: in "crypto_tests/sha256d64": check memcmp(out1, out2, 32 * i) == 0 has failed
test/crypto_tests.cpp(561): error: in "crypto_tests/sha256d64": check memcmp(out1, out2, 32 * i) == 0 has failed
test/crypto_tests.cpp(561): error: in "crypto_tests/sha256d64": check memcmp(out1, out2, 32 * i) == 0 has failed
test/crypto_tests.cpp(561): error: in "crypto_tests/sha256d64": check memcmp(out1, out2, 32 * i) == 0 has failed
test/crypto_tests.cpp(561): error: in "crypto_tests/sha256d64": check memcmp(out1, out2, 32 * i) == 0 has failed

*** 32 failures are detected in the test module "Bitcoin Test Suite"
@MarcoFalke

This comment has been minimized.

Copy link
Member

commented May 2, 2019

Why is the travis 32-bit not failing?

https://travis-ci.org/bitcoin/bitcoin/jobs/527334881#L2853

@MarcoFalke

This comment has been minimized.

Copy link
Member

commented May 2, 2019

So I guess it has nothing to do with openssl

@MarcoFalke

This comment has been minimized.

Copy link
Member

commented May 2, 2019

I can reproduce also on vanilla fedora 30 - 32 bit

Schermafdruk van 2019-05-02 17-09-39

@sipa

This comment has been minimized.

Copy link
Member

commented May 2, 2019

@MarcoFalke What does the "Using the '%s' SHA256 implementation" say on that system/build? Also what does cat /proc/cpuinfo | fgrep flags say?

@MarcoFalke

This comment has been minimized.

Copy link
Member

commented May 2, 2019

$ ./src/bitcoind -printtoconsole -regtest | grep SHA
2019-05-02T21:19:53Z Using the 'standard,sse41(4way),avx2(8way)' SHA256 implementation


$ cat /proc/cpuinfo | fgrep flags | uniq -c
      8 flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ss ht syscall nx pdpe1gb rdtscp lm constant_tsc arch_perfmon xtopology cpuid tsc_known_freq pni pclmulqdq vmx ssse3 fma cx16 sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand hypervisor lahf_lm abm 3dnowprefetch cpuid_fault ssbd ibrs ibpb tpr_shadow vnmi flexpriority ept vpid ept_ad fsgsbase tsc_adjust bmi1 avx2 smep bmi2 erms invpcid mpx rdseed adx smap clflushopt xsaveopt xsavec xgetbv1 xsaves arat umip
@sipa

This comment has been minimized.

Copy link
Member

commented May 5, 2019

I believe this is a GCC bug, filed as https://gcc.gnu.org/bugzilla/show_bug.cgi?id=90348

@practicalswift

This comment has been minimized.

Copy link
Member

commented May 5, 2019

@sipa Excellent work! This sure looks like a miscompilation bug in gcc: the test case does not contain any UB that would give gcc the leeway to behave this way.

laanwj added a commit that referenced this issue May 16, 2019
Merge #15950: Do not construct out-of-bound pointers in SHA2 code
c01c065 Do not construct out-of-bound pointers in SHA512/SHA1/RIPEMD160 code (Pieter Wuille)

Pull request description:

  This looks like an issue in the current SHA256/512 code, where a pointer outside of the area pointed to may be constructed (this is UB in theory, though in practice every supported platform treats pointers as integers).

  I discovered this while investigating #14580. Sadly, it does not fix it.

ACKs for commit c01c06:
  practicalswift:
    utACK c01c065

Tree-SHA512: 47660e00f164f38c36a1ab46e52dd91cd33cfda6a6048d67541c2f8e73c050d4d9d81b5c149bfad281212d52f204f57bebf5b19879dc7a6a5f48aa823fbc2c02
sidhujag pushed a commit to syscoin/syscoin that referenced this issue May 18, 2019
Merge bitcoin#15950: Do not construct out-of-bound pointers in SHA2 code
c01c065 Do not construct out-of-bound pointers in SHA512/SHA1/RIPEMD160 code (Pieter Wuille)

Pull request description:

  This looks like an issue in the current SHA256/512 code, where a pointer outside of the area pointed to may be constructed (this is UB in theory, though in practice every supported platform treats pointers as integers).

  I discovered this while investigating bitcoin#14580. Sadly, it does not fix it.

ACKs for commit c01c06:
  practicalswift:
    utACK c01c065

Tree-SHA512: 47660e00f164f38c36a1ab46e52dd91cd33cfda6a6048d67541c2f8e73c050d4d9d81b5c149bfad281212d52f204f57bebf5b19879dc7a6a5f48aa823fbc2c02
@practicalswift

This comment has been minimized.

Copy link
Member

commented May 26, 2019

Can be closed now that #15983 has been merged?

@error10, thanks a lot for reporting! This turned out to be a very valuable bug report.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
8 participants
You can’t perform that action at this time.