openssl@3 3.2.0 build failing in make test, breaking build.

[apjanke]

Starting today, Sunday 2024-01-28, the build is breaking due to a test failure in the openssl@3 3.2.0_1 build's make test step, at least on Apple Silicon.

Test Summary Report
-------------------
15-test_ec.t                          (Wstat: 256 Tests: 15 Failed: 1)
  Failed test:  2
  Non-zero exit status: 1
30-test_evp.t                         (Wstat: 256 Tests: 80 Failed: 1)
  Failed test:  74
  Non-zero exit status: 1
30-test_evp_extra.t                   (Wstat: 256 Tests: 3 Failed: 1)
  Failed test:  1
  Non-zero exit status: 1
Files=294, Tests=3562, 296 wallclock secs ( 2.82 usr  0.45 sys + 173.76 cusr 49.28 csys = 226.31 CPU)
Result: FAIL
make[1]: *** [run_tests] Error 1
make: *** [tests] Error 2
[...]

Error: openssl@3 3.2.0 did not build

This is blocking the Octave.app 8.4.0 release, and most any further work on it.

Probably not much I can do here except wait until it gets fixed upstream? Or add support for building from a fixed older commit of the Homebrew core formulae and roll back to an older working version of openssl 3?

Symptoms

When trying to build an Octave.app 8.4.0 beta1 release on eilonwy (M1 Max MBP), macOS 14:

Tail of build output:

99-test_fuzz_x509.t ..................... ok

Test Summary Report
-------------------
15-test_ec.t                          (Wstat: 256 Tests: 15 Failed: 1)
  Failed test:  2
  Non-zero exit status: 1
30-test_evp.t                         (Wstat: 256 Tests: 80 Failed: 1)
  Failed test:  74
  Non-zero exit status: 1
30-test_evp_extra.t                   (Wstat: 256 Tests: 3 Failed: 1)
  Failed test:  1
  Non-zero exit status: 1
Files=294, Tests=3562, 296 wallclock secs ( 2.82 usr  0.45 sys + 173.76 cusr 49.28 csys = 226.31 CPU)
Result: FAIL
make[1]: *** [run_tests] Error 1
make: *** [tests] Error 2

==> Formula
Path: /Users/janke/Library/Caches/Homebrew/downloads/776dcdce0844a14d82d10a323f91e9b5626c99597ad36b1dd8d846ab8999a90d--openssl@3.rb
==> Configuration
HOMEBREW_VERSION: 4.2.5-94-gcfac516
ORIGIN: https://github.com/Homebrew/brew
HEAD: cfac516ee045d869be34598152e80a11ef2befe7
Last commit: 50 minutes ago
Core tap JSON: 28 Jan 16:43 UTC
Core cask tap JSON: 28 Jan 16:43 UTC
HOMEBREW_PREFIX: /Applications/Octave-8.4.0_beta1.app/Contents/Resources/usr
HOMEBREW_REPOSITORY: /Applications/Octave-8.4.0_beta1.app/Contents/Resources/usr/Homebrew
HOMEBREW_CELLAR: /Applications/Octave-8.4.0_beta1.app/Contents/Resources/usr/Cellar
HOMEBREW_CASK_OPTS: []
HOMEBREW_EDITOR: code
HOMEBREW_MAKE_JOBS: 10
HOMEBREW_NO_AUTO_UPDATE: set
HOMEBREW_NO_ENV_HINTS: set
Homebrew Ruby: 3.1.4 => /Applications/Octave-8.4.0_beta1.app/Contents/Resources/usr/Homebrew/Library/Homebrew/vendor/portable-ruby/3.1.4/bin/ruby
CPU: 10-core 64-bit arm_firestorm_icestorm
Clang: 15.0.0 build 1500
Git: 2.39.3 => /Applications/Xcode.app/Contents/Developer/usr/bin/git
Curl: 8.4.0 => /usr/bin/curl
macOS: 14.3-arm64
CLT: 15.1.0.0.1.1700200546
Xcode: 15.2
Rosetta 2: false
==> ENV
HOMEBREW_CC: clang
HOMEBREW_CXX: clang++
MAKEFLAGS: -j10
CMAKE_PREFIX_PATH: /Applications/Octave-8.4.0_beta1.app/Contents/Resources/usr
CMAKE_INCLUDE_PATH: /Library/Developer/CommandLineTools/SDKs/MacOSX14.sdk/System/Library/Frameworks/OpenGL.framework/Versions/Current/Headers
CMAKE_LIBRARY_PATH: /Library/Developer/CommandLineTools/SDKs/MacOSX14.sdk/System/Library/Frameworks/OpenGL.framework/Versions/Current/Libraries
PKG_CONFIG_LIBDIR: /usr/lib/pkgconfig:/Applications/Octave-8.4.0_beta1.app/Contents/Resources/usr/Homebrew/Library/Homebrew/os/mac/pkgconfig/14
HOMEBREW_GIT: git
HOMEBREW_SDKROOT: /Library/Developer/CommandLineTools/SDKs/MacOSX14.sdk
ACLOCAL_PATH: /Applications/Octave-8.4.0_beta1.app/Contents/Resources/usr/share/aclocal
PATH: /Applications/Octave-8.4.0_beta1.app/Contents/Resources/usr/Homebrew/Library/Homebrew/shims/mac/super:/usr/bin:/bin:/usr/sbin:/sbin


[...]
ACLOCAL_PATH: /Applications/Octave-8.4.0_beta1.app/Contents/Resources/usr/share/aclocal
PATH: /Applications/Octave-8.4.0_beta1.app/Contents/Resources/usr/Homebrew/Library/Homebrew/shims/mac/super:/usr/bin:/bin:/usr/sbin:/sbin

Error: openssl@3 3.2.0 did not build
Logs:
     /Users/janke/Library/Logs/Homebrew/openssl@3/00.options.out
     /Users/janke/Library/Logs/Homebrew/openssl@3/01.perl
     /Users/janke/Library/Logs/Homebrew/openssl@3/01.perl.cc
     /Users/janke/Library/Logs/Homebrew/openssl@3/02.make
     /Users/janke/Library/Logs/Homebrew/openssl@3/02.make.cc
     /Users/janke/Library/Logs/Homebrew/openssl@3/03.make
     /Users/janke/Library/Logs/Homebrew/openssl@3/04.make
Do not report this issue to Homebrew/brew or Homebrew/homebrew-core!

$

Scrolled up and found these. Looks like the output is a little messy or out of order, I suspect due to the progress summary being on stdout (which is buffered) and the error details on stderr (unbuffered), and no newline being output at the end of the failure details.

15-test_dsa.t ........................... ok
15-test_dsaparam.t ...................... ok

        # Curve SM2
        # ERROR: (memory) 'b1 == b2' failed @ test/ectest.c:2783
        # --- b1
        # +++ b2
        # 0000:-0445aee733a34f77 4594c9abd9a73832 7249284e0c928b68 01293fab3c919e57
        # 0000:+04884d165f541e4b c4f954fd79fbab90 0a2ee53d26472b3e 10874170294e72cf
        #         ^^^^^^^^^^^^^^ ^^^^^^^^^^^^^^^^ ^^^^^^^^^^^^^^^^ ^^^^^^^^^^^^^^^^
        # 0020:-e4e79ff051cf1783 e003ce1240771de0 377f94db71305e20 1bddf090cf497c11
        # 0020:+f4761a87f11a05ec c52f0a3a7549ec67 83091a1dd07342e2 1683bee2a5e23a74
        #       ^^^^^^^^^^^^^^^^ ^^^^^^^^^^^^^^^^ ^^^^^^^^^^^^^^^^ ^^^^^^^^^^^^^^^^
        # 0040:-80
        # 0040:+9b
        #       ^^
        #
        # OPENSSL_TEST_RAND_SEED=1706460726
        not ok 751 - iteration 82
# ------------------------------------------------------------------------------
    # OPENSSL_TEST_RAND_SEED=1706460726
    not ok 18 - custom_generator_test
# ------------------------------------------------------------------------------
../../util/wrap.pl ../../test/ectest => 1
not ok 2 - running ectest
# ------------------------------------------------------------------------------
# Looks like you failed 1 test of 15.15-test_ec.t ............................
Dubious, test returned 1 (wstat 256, 0x100)
Failed 1/15 subtests
	(less 1 skipped subtest: 13 okay)
15-test_ecdsa.t ......................... ok
15-test_ecparam.t ....................... ok
[...]
30-test_engine.t ........................ ok

        # INFO:  @ test/testutil/stanza.c:21
        # Reading ../../test/recipes/30-test_evp_data/evppkey_sm2.txt
        # INFO:  @ test/testutil/stanza.c:122
        # Starting "SM2 tests" tests at line 19
        # INFO:  @ test/evp_test.c:3698
        # ../../test/recipes/30-test_evp_data/evppkey_sm2.txt:57: Source of above error; unexpected error KEYOP_ERROR
        # 409C04DB01000000:error:1A800066:SM2 routines:ossl_sm2_decrypt:reason(102):crypto/sm2/sm2_crypt.c:384:
        # INFO:  @ test/testutil/stanza.c:122
        # Starting "SM2 key generation tests" tests at line 78
        # INFO:  @ test/testutil/stanza.c:33
        # Completed 10 tests with 1 errors and 0 skipped
        # OPENSSL_TEST_RAND_SEED=1706460843
        not ok 1 - iteration 1
# ------------------------------------------------------------------------------
    # OPENSSL_TEST_RAND_SEED=1706460843
    not ok 1 - run_file_tests
# ------------------------------------------------------------------------------
../../util/wrap.pl ../../test/evp_test -config ../../test/default-and-legacy.cnf ../../test/recipes/30-test_evp_data/evppkey_sm2.txt => 1
not ok 74 - running evp_test -config ../../test/default-and-legacy.cnf evppkey_sm2.txt
# ------------------------------------------------------------------------------
# Looks like you failed 1 test of 80.30-test_evp.t ...........................
Dubious, test returned 1 (wstat 256, 0x100)
Failed 1/80 subtests
	(less 3 skipped subtests: 76 okay)

    # ERROR: (int) 'EVP_DigestVerifyFinal(mctx, signature, sizeof(signature)) > 0' failed @ test/evp_extra_test.c:2115
    # [0] compared to [0]
    # OPENSSL_TEST_RAND_SEED=1706460843
    not ok 16 - test_EVP_SM2_verify
# ------------------------------------------------------------------------------
../../util/wrap.pl ../../test/evp_extra_test => 1
not ok 1 - running evp_extra_test
# ------------------------------------------------------------------------------
# Looks like you failed 1 test of 3.30-test_evp_extra.t .....................
Dubious, test returned 1 (wstat 256, 0x100)
Failed 1/3 subtests
30-test_evp_fetch_prov.t ................ ok

I'm getting a test failure doing a build-from-source installation in my main Homebrew installation on this machine, too. Not as many tests failed, but the one that did fail, 30-test_evp_extra.t, is one that failed in my build.

$ brew reinstall --build-from-source openssl@3

Warning: building from source is not supported!
You're on your own. Failures are expected so don't create any issues, please!
[...]
==> make
==> make install MANDIR=/opt/homebrew/Cellar/openssl@3/3.2.0_1/share/man MANSUFFIX=ssl
==> make test
Last 15 lines from /Users/janke/Library/Logs/Homebrew/openssl@3/04.make:
99-test_fuzz_quic_client.t .............. ok
99-test_fuzz_server.t ................... ok
99-test_fuzz_smime.t .................... ok
99-test_fuzz_v3name.t ................... ok
99-test_fuzz_x509.t ..................... ok

Test Summary Report
-------------------
30-test_evp_extra.t                   (Wstat: 256 Tests: 3 Failed: 1)
  Failed test:  2
  Non-zero exit status: 1
Files=294, Tests=3562, 298 wallclock secs ( 2.79 usr  0.46 sys + 175.17 cusr 49.23 csys = 227.65 CPU)
Result: FAIL
make[1]: *** [run_tests] Error 1
make: *** [tests] Error 2

READ THIS: https://docs.brew.sh/Troubleshooting

Trying a non-parallel build with HOMEBREW_MAKE_JOBS=1 brew reinstall --verbose --build-from-source openssl@3 and running it a couple times to see if the errors are consistent.

[apjanke]

WIP

TODO:

• Re-try openssl@3 build-from-source in main Homebrew a few times
• Try openssl@3 build-from-source on crumpet and other hosts too, in case it's a weird host-specific error

Build command:

• brew reinstall --verbose --build-from-source openssl@3
• HOMEBREW_MAKE_JOBS=1 brew reinstall --verbose --build-from-source openssl@3

[apjanke]

Well well well. The failures only happen on my one Mac eilonwy, and are intermittent - fails only about 15% of the time, and different (related) tests fail on different runs, and the specific unexpected values are different in those failures. Looks like I may have a hardware issue, maybe even some bad RAM.

Couldn't reproduce this failure on any other machine.

Just doing the build on a different machine for now. I think this also means all previous Apple Silicon 8.4.0 prerelease builds I uploaded are suspect: if it's bad RAM, it may have corrupted built files in a way that did not show up in tests.

[apjanke]

Closing this as NoRepro. Looks like this is a problem with my local build environment, not octapp.