openssl: doesn't build with enabled tests [Windows] #761

Closed
JMoVS opened this Issue Aug 13, 2016 · 21 comments

Projects

None yet

3 participants

@JMoVS
JMoVS commented Aug 13, 2016

Please follow the general troubleshooting steps first:

  • Ran brew update and retried your prior step?
  • Ran brew doctor, fixed as many issues as possible and retried your prior step?
  • If you're seeing permission errors tried running sudo chown -R $(whoami) $(brew --prefix)?

OpenSSL doesn't build on windows subsystem for linux currently. It fails in the make depend area.

https://gist.github.com/6a18df98b469ce93b5f22867efb62e37

Any ideas how to solve this?

@sjackman sjackman changed the title from openssl: Doesn't build on windows subsystem for linux to openssl: makedepend: ELF not found [Windows] Aug 13, 2016
@sjackman
Member
sjackman commented Aug 13, 2016 edited

This issue may be the same issue as Linuxbrew/brew#95
One hypothesis I have is that the Windows kernel is not able to load some (but it seems not all) executables that have been fixed up by patchelf. Please report the output of

brew install hello
brew test hello
brew install makedepend
brew test makedepend
brew reinstall -s makedepend
brew test makedepend
brew install openssl
@JMoVS
JMoVS commented Aug 13, 2016 edited

as I already startet, I did the following:

brew reinstall makedepend -s
brew install openssl -sv

it didn't compile, but it is a different error now:

here the full brew gist-logs:
https://gist.github.com/910c6d8d27d2266390cd1bf200089a1f

https://gist.github.com/JMoVS/231af5bc440595cda53198d8cd9bdb5f

aside from that brew test hello results in:

brew install hello
==> Downloading https://linuxbrew.bintray.com/bottles/hello-2.10.x86_64_linux.bottle.tar.gz
######################################################################## 100,0%
==> Pouring hello-2.10.x86_64_linux.bottle.tar.gz
🍺  /home/justin/.linuxbrew/Cellar/hello/2.10: 52 files, 231.3K
justin@PC69EP5:~$ brew test hello
Testing hello
==> /home/justin/.linuxbrew/Cellar/hello/2.10/bin/hello --greeting=brew
Last 15 lines from /home/justin/.cache/Homebrew/Logs/hello/01.hello:
2016-08-13 16:20:06 +0200

/home/justin/.linuxbrew/Cellar/hello/2.10/bin/hello
--greeting=brew

/home/justin/.linuxbrew/Cellar/hello/2.10/bin/hello: 1: /home/justin/.linuxbrew/Cellar/hello/2.10/bin/hello: �ELF����: not found
/home/justin/.linuxbrew/Cellar/hello/2.10/bin/hello: 2: /home/justin/.linuxbrew/Cellar/hello/2.10/bin/hello: @���@�?@�?�0Q�td������?�?�����8: not found
/home/justin/.linuxbrew/Cellar/hello/2.10/bin/hello: 3: /home/justin/.linuxbrew/Cellar/hello/2.10/bin/hello: Syntax error: "(" unexpected
Error: hello: failed
Failed executing: /home/justin/.linuxbrew/Cellar/hello/2.10/bin/hello --greeting=brew
/home/justin/.linuxbrew/Library/Homebrew/formula.rb:1508:in `block in system'
/home/justin/.linuxbrew/Library/Homebrew/formula.rb:1445:in `open'
/home/justin/.linuxbrew/Library/Homebrew/formula.rb:1445:in `system'
/home/justin/.linuxbrew/Library/Taps/homebrew/homebrew-core/Formula/hello.rb:26:in `block in <class:Hello>'
/home/justin/.linuxbrew/Library/Homebrew/formula.rb:1350:in `block in run_test'
/home/justin/.linuxbrew/Library/Homebrew/extend/fileutils.rb:14:in `block in mktemp'
/home/justin/.linuxbrew/Library/Homebrew/extend/fileutils.rb:75:in `block in run'
/home/justin/.linuxbrew/Library/Homebrew/extend/fileutils.rb:75:in `chdir'
/home/justin/.linuxbrew/Library/Homebrew/extend/fileutils.rb:75:in `run'
/home/justin/.linuxbrew/Library/Homebrew/extend/fileutils.rb:13:in `mktemp'
/home/justin/.linuxbrew/Library/Homebrew/formula.rb:1344:in `run_test'
/home/justin/.linuxbrew/Library/Homebrew/test.rb:33:in `block in <main>'
/usr/lib/ruby/1.9.1/timeout.rb:69:in `timeout'
/home/justin/.linuxbrew/Library/Homebrew/test.rb:32:in `<main>'

as I already had it handy, here is the output of self-compiled makedepend:

justin@myPC:~$ brew test makedepend

Testing makedepend
==> /home/justin/.linuxbrew/Cellar/makedepend/1.0.5/bin/makedepend

and from bottled version:

~$ brew test makedepend
Testing makedepend
==> /home/justin/.linuxbrew/Cellar/makedepend/1.0.5/bin/makedepend
Last 15 lines from /home/justin/.cache/Homebrew/Logs/makedepend/01.makedepend:
2016-08-13 16:23:54 +0200

/home/justin/.linuxbrew/Cellar/makedepend/1.0.5/bin/makedepend

/home/justin/.linuxbrew/Cellar/makedepend/1.0.5/bin/makedepend: 1: /home/justin/.linuxbrew/Cellar/makedepend/1.0.5/bin/makedepend: �ELF����: not found
/home/justin/.linuxbrew/Cellar/makedepend/1.0.5/bin/makedepend: 3: /home/justin/.linuxbrew/Cellar/makedepend/1.0.5/bin/makedepend: Syntax error: "(" unexpected
Error: makedepend: failed
Failed executing: /home/justin/.linuxbrew/Cellar/makedepend/1.0.5/bin/makedepend
/home/justin/.linuxbrew/Library/Homebrew/formula.rb:1508:in `block in system'
/home/justin/.linuxbrew/Library/Homebrew/formula.rb:1445:in `open'
/home/justin/.linuxbrew/Library/Homebrew/formula.rb:1445:in `system'
/home/justin/.linuxbrew/Library/Taps/homebrew/homebrew-core/Formula/makedepend.rb:52:in `block in <class:Makedepend>'
/home/justin/.linuxbrew/Library/Homebrew/formula.rb:1350:in `block in run_test'
/home/justin/.linuxbrew/Library/Homebrew/extend/fileutils.rb:14:in `block in mktemp'
/home/justin/.linuxbrew/Library/Homebrew/extend/fileutils.rb:75:in `block in run'
/home/justin/.linuxbrew/Library/Homebrew/extend/fileutils.rb:75:in `chdir'
/home/justin/.linuxbrew/Library/Homebrew/extend/fileutils.rb:75:in `run'
/home/justin/.linuxbrew/Library/Homebrew/extend/fileutils.rb:13:in `mktemp'
/home/justin/.linuxbrew/Library/Homebrew/formula.rb:1344:in `run_test'
/home/justin/.linuxbrew/Library/Homebrew/test.rb:33:in `block in <main>'
/usr/lib/ruby/1.9.1/timeout.rb:69:in `timeout'
/home/justin/.linuxbrew/Library/Homebrew/test.rb:32:in `<main>'
@JMoVS
JMoVS commented Aug 13, 2016

also, building mkvtoolnix fails with:

==> ./configure --disable-debug --prefix=/home/justin/.linuxbrew/Cellar/mkvtoolnix/9.3.1_1 --without-curl --with-boost=/home/justin/.linuxbrew/opt/boost --with-extra-includes=/home/justin/.linuxbrew/opt/libogg/include;/home/justin/.linuxbrew/opt/libvorbis/include;/home/justin/.linuxbrew/opt/libebml/include;/home/justin/.linuxbrew/opt/libmatroska/include --with-extra-libs=/home/justin/.linuxbrew/opt/libogg/lib;/home/justin/.linuxbrew/opt/libvorbis/lib;/home/justin/.linuxbrew/opt/libebml/lib;/home/justin/.linuxbrew/opt/libmatroska/lib --disable-qt
Failed to execute: ./configure

Is that related? Should I essentially start rebuilding all packages from source in homebrew? ;-)

https://gist.github.com/d1ea8e5e65483a65d41038748f206052

@sjackman
Member

Interesting that brew install hello doesn't work for you. I believe Linuxbrew/brew#95 stated that brew install hello did work for them. I've asked them to reconfirm.

Should I essentially start rebuilding all packages from source in homebrew? ;-)

If bottles are not working on Windows (as it seems), always build from source using:

export HOMEBREW_BUILD_FROM_SOURCE=1
@sjackman
Member

also, building mkvtoolnix fails with:
Failed to execute: ./configure

Not sure what's going on here. Try

brew install -dv mkvtoolnix

Then select shell from the interactive menu, and try executing ./configure.

@JMoVS
JMoVS commented Aug 13, 2016

It's also me who reported Linuxbrew/brew#95 - well, installing worked, but I didn't run it (I oversaw this or maybe was faster than you could edit your comment) ;-)

@JMoVS
JMoVS commented Aug 13, 2016

BTW: I updated the comment with the gist from the openssl installation attempt

@JMoVS
JMoVS commented Aug 13, 2016

mkvtoolnix issue was due to xz not working but failing silently, so there was no extraction happening. I rebuilt xz from source and mkvtoolnix is hitting some other errors now, more on that later

@sjackman
Member

Thanks for the clarification regarding hello. Progress. Good stuff.

@sjackman sjackman self-assigned this Aug 13, 2016
@JMoVS
JMoVS commented Aug 14, 2016

I now rm-ed all packages and am currently reinstalling with homebrew_build_from_source set to 1 so I will see into what errors I'll run...

@JMoVS
JMoVS commented Aug 14, 2016

could you have a look at https://gist.github.com/63a03fdb21c0432664c592b284d67745 ? This is the next attempt with all dependencies built from source

@JMoVS
JMoVS commented Aug 14, 2016 edited

this doesn't seem to cause a problem: https://groups.google.com/forum/#!topic/mailing.openssl.dev/ePwMzh74GOE

so it's probably still

make[2]: Leaving directory `/tmp/openssl-20160814-19037-ge33vm/openssl-1.0.2h'
../util/shlib_wrap.sh ./destest
ERROR: ld.so: object '../util/../libcrypto.so.1.0.0' from LD_PRELOAD cannot be preloaded (cannot enable executable stack as shared object requires): ignored.
./destest: error while loading shared libraries: libcrypto.so.1.0.0: cannot enable executable stack as shared object requires: Invalid argument
make[1]: *** [test_des] Fehler 127
make[1]: Verzeichnis »/tmp/openssl-20160814-19037-ge33vm/openssl-1.0.2h/test« wird verlassen
make: *** [tests] Fehler 2
@sjackman
Member

Try

brew install openssl --without-test

See

brew options openssl
@JMoVS
JMoVS commented Aug 15, 2016

ok, this worked, cool!

@JMoVS JMoVS changed the title from openssl: makedepend: ELF not found [Windows] to openssl: doesn't build with enabled tests [Windows] Aug 15, 2016
@JMoVS
JMoVS commented Sep 20, 2016

hmm, is this "solution" a thing that could be incorporated into linux/win formulas to account for this executable stack nonsense, @sjackman ?
Microsoft/BashOnWindows#286 (comment)

@sjackman
Member
sjackman commented Oct 7, 2016 edited

Possibly, yes. For installed programs, it looks as though execstack -c $HOMEBREW_PREFIX/bin/* would workaround the issue by enabling the executable stack on all programs. It defeats the intended security benefit of disabling executable stack. There's probably a limited number of programs that need an executable stack. We could fix this on a case-by-case basis.

For the OpenSSL test it's trickier. I think that's an upstream bug that needs to be raised with the upstream OpenSSL developers. In the mean time, in Linuxbrew we could disable the OpenSSL tests (--without-test) by default on Windows.

@JMoVS
JMoVS commented Oct 7, 2016

hmm, starting with this formula which requires executable stack as well, how could I include the execstack command into the formula (also, execstack seems to require root, so probably interesting to include it into a brew formula) of openssl so I can build with tests? Would it just go at a specific position in the formula?

@sjackman
Member
sjackman commented Oct 8, 2016

I believe you'd have to somehow tell OpenSSL to use execstack when it runs its tests. You'd need help from the developers of OpenSSL for that.

@JMoVS
JMoVS commented Oct 10, 2016

Hmm, I was wondering about this executable stack as it seems that sometimes, this is already needed at brew install time. But I'm not sure.

@yorickdowne
yorickdowne commented Jan 10, 2017 edited

OpenSSL does not require execstack. Debian worked around the issue with -Wa,--noexecstack in flags. See https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=430583

A patch to have OpenSSL not request execstack was rejected in 2005, for portability issues. See http://openssl.6102.n7.nabble.com/PATCH-libcrypto-without-executable-stack-td30772.html
I had an idea on how to create a portable patch, but it didn't pan out.

The question is why brew doesn't use -Wa,--noexecstack, as that's an OpenSSL default, see below.

Edit: The plot thickens. @sjackman , can you comment on how linuxbrew behaves? I've just run ./config --shared manually on a 1.0.2j source tarball, and here are the CFLAGS:

CFLAG =-fPIC -DOPENSSL_PIC -DOPENSSL_THREADS -D_REENTRANT -DDSO_DLFCN -DHAVE_DLFCN_H -Wa,--noexecstack -m64 -DL_ENDIAN -O3 -Wall -DOPENSSL_IA32_SSE2 -DOPENSSL_BN_ASM_MONT -DOPENSSL_BN_ASM_MONT5 -DOPENSSL_BN_ASM_GF2m -DRC4_ASM -DSHA1_ASM -DSHA256_ASM -DSHA512_ASM -DMD5_ASM -DAES_ASM -DVPAES_ASM -DBSAES_ASM -DWHIRLPOOL_ASM -DGHASH_ASM -DECP_NISTZ256_ASM

Note the -Wa,--noexecstack. That's supposed to avoid execstack being set. And it does. I did a make, followed by make test. That's successful. I ran scanelf and this is the output:

tbehrens@WSL-Insider-Test:~/openssl-1.0.2j$ find . -name 'libcrypto*'
./libcrypto.a
./libcrypto.pc
./libcrypto.so
./libcrypto.so.1.0.0
tbehrens@WSL-Insider-Test:~/openssl-1.0.2j$ scanelf -qeR .
tbehrens@WSL-Insider-Test:~/openssl-1.0.2j$

No ELF binaries requesting execstack anywhere in the build tree. The OpenSSL build does clear execstack on libcrypto. The next question is: Why doesn't it build that way when using brew? And what can be done to fix that?

@sjackman
Member

Without looking at the openssl code, it's likely a bug in their Makefile that when brew sets CFLAGS it overrides -Wa,--noexecstack, which is not desirable behaviour of a Makefile. I've merged PR #1510 which should resolve this issue.

@sjackman sjackman closed this Jan 10, 2017
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment