make check and make pytest fail on MacOS

[willcl-ark]

Issue and Steps to Reproduce

• MacOS: 10.15
• Clang: Apple clang version 11.0.3 (clang-1103.0.32.29)
• CLightning @ 185fe72

Running

make -j3 full-check PYTEST_PAR=8 DEVELOPER=1 VALGRIND=0

...works successfully: make_full-check.txt

However trying either make check or make pytest seems to fail, e.g.:

make -j3 check PYTEST_PAR=8 DEVELOPER=1 VALGRIND=0

make_check.txt

make -j3 pytest PYTEST_PAR=8 DEVELOPER=1 VALGRIND=0

make_pytest.txt

Trying to run pytests manually using:

PYTHONPATH=contrib/pylightning:contrib/pyln-client:contrib/pyln-testing:contrib/pyln-proto py.test -v tests/

...appears to also fail with the same errors.

It's possible I'm "using them wrong", but I've tried to follow the documentation where possible... Pretty sure I have all required packages installed: pip_installs.txt

[cdecker]

It seems that pytest is failing on the python-bitcoinlib dependency:

Current thread 0x00000001061dfdc0 (most recent call first):
  File "/Users/will/.pyenv/versions/3.7.7/lib/python3.7/ctypes/__init__.py", line 364 in __init__
  File "/Users/will/.pyenv/versions/3.7.7/lib/python3.7/ctypes/__init__.py", line 442 in LoadLibrary
  File "/Users/will/.pyenv/versions/3.7.7/envs/lightning-3.7.7/lib/python3.7/site-packages/bitcoin/core/key.py", line 33 in <module>
...

Which seems to indicate a failure here, which in turn indicates that we fail to load libssl which is a dependency of python-bitcoinlib.

full-check seems to be failing in the shellcheck step, thus never even starting pytest, otherwise I'm sure it's behavior would be the same.

@willcl-ark can you confirm that you don't have libssl installed? We could report this upstream to have python-bitcoinlib add an installation check to see if it can load libssl and abort installation otherwise.

[willcl-ark]

Hello @cdecker 👋🏼 I have openssl@1.1 installed via brew, allegedly the (direct) MacOS equivalent, and also using pyenv which compiles my python. It seems from a bit of digging on that side (e.g. here) that perhaps pyenv doesn't find openssl correctly when compiling new python version?...

Have now re-compiled python 3.7.7 linked directly to openssl:

checking for openssl/ssl.h in /usr/local/opt/openssl@1.1... yes
checking whether compiling and linking against OpenSSL works... yes

Then tried a re-configure and re-make to no avail.

Had I checked python-bitcoinlib's repo first however I would have found this: petertodd/python-bitcoinlib#238 !

I was able to fix as Sergi did by hardcoding into python-bitcoinlib/keys.py:

#_ssl = ctypes.cdll.LoadLibrary(ctypes.util.find_library('ssl') or 'libeay32')
 _ssl = ctypes.cdll.LoadLibrary("/usr/local/opt/openssl@1.1/lib/libcrypto.dylib")

So it seems it certainly is one for upstream, although I'm not sure if Peter is merging (any?) changes into this project any more? Anyway, thanks for the tip! At least when it happens to me (or someone else) again, we'll be able to find this issue :)

[niftynei]

i believe @kanzure has taken over maintenance of python-bitcoinlib.

[kanzure]

I haven't dived into that problem yet. Is this urgent?

[sr-gi]

petertodd/python-bitcoinlib#238 was closed by petertodd/python-bitcoinlib#240 a couple of months ago, so this may be fixed now.

Tried to run tests on pyln-proto using python-bitcoinlib==0.11.0 and it works for me. Maybe @willcl-ark can verify.

[willcl-ark]

Hello @sr-gi, thanks for the tag.

Yes great success in running the tests now on Catalina with petertodd/python-bitcoinlib#240 patch for python-bitcoinlib.

Unfortunately the patch is not in python-bitcoinlib==0.11.0 release so have to install manually from source for now, but that's OK.

I haven't run tests to completion (nor will I tonight), but it looks like a fair few tests fail in the first 5%, e.g. 4 failed, 14 passed, 8 warnings, 14 errors in 188.67s (0:03:08), but I guess unrelated to this issue...