Support LibreSSL (instead of OpenSSL): make RAND_egd optional

[EddBarrett]

BPO	21356
Nosy	@pitrou, @vstinner, @giampaolo, @tiran, @florentx, @mgorny, @koobs, @dstufft
Files	patch-Lib_ssl.py: Make RAND_egd support automatic patch-Modules__ssl.c: Make RAND_egd support automatic patch-configure.ac: Make RAND_egd support automatic test_ssl.log: Output of test_ssl.py

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2015-01-06.13:01:58.158>
created_at = <Date 2014-04-26.14:21:36.056>
labels = ['build']
title = 'Support LibreSSL (instead of OpenSSL): make RAND_egd optional'
updated_at = <Date 2015-01-06.13:01:58.157>
user = 'https://bugs.python.org/EddBarrett'

bugs.python.org fields:

activity = <Date 2015-01-06.13:01:58.157>
actor = 'vstinner'
assignee = 'none'
closed = True
closed_date = <Date 2015-01-06.13:01:58.158>
closer = 'vstinner'
components = ['Build']
creation = <Date 2014-04-26.14:21:36.056>
creator = 'Edd.Barrett'
dependencies = []
files = ['37300', '37301', '37302', '37304']
hgrepos = []
issue_num = 21356
keywords = []
message_count = 25.0
messages = ['217198', '217199', '226355', '226357', '226819', '226832', '231426', '231428', '231462', '231471', '231797', '231798', '231799', '231801', '231802', '231803', '231804', '231807', '231808', '231809', '231812', '231814', '231839', '233535', '233537']
nosy_count = 15.0
nosy_names = ['janssen', 'pitrou', 'vstinner', 'giampaolo.rodola', 'christian.heimes', 'flox', 'polymorphm', 'mgorny', 'python-dev', 'rpointel', 'oberstet', 'koobs', 'dstufft', 'Edd.Barrett', 'spil']
pr_nums = []
priority = 'normal'
resolution = 'fixed'
stage = None
status = 'closed'
superseder = None
type = None
url = 'https://bugs.python.org/issue21356'
versions = ['Python 2.7', 'Python 3.4', 'Python 3.5']

[EddBarrett]

Hi,

I'm sure you have heard about OpenBSD's LibreSSL fork of OpenSSL. There has been a lot of code reorganisation and removal. One function which was removed RAND_egd() breaks the CPython build. CPython no longer builds on OpenBSD.

I have submitted a patch against PyPy already. The application library part of the change can probably be re-used since PyPy borrows CPython's application-level standard library (including the ssl and socket module). However, for the interpreter level change, the build system will probably have to be hacked. We need to check for the existence of RAND_egd() at configure time and only build in support if the function is found.

The PyPy patch (and some discussion) is here:
https://bitbucket.org/pypy/pypy/pull-request/233/fix-translation-for-libressl-and-fix-ssl/diff#comment-1744605

I may have a go at doing this myself (for Python-2.7 at least) if no-one steps up in the meantime; for now just making the CPython devs aware.

Thanks

[pitrou]

This should wait until the LibreSSL API stabilizes.

Regardless, I think we should consider deprecating RAND_egd(). The Entropy Gathering Daemon doesn't seem to have seen a release for more than 10 years... (http://sourceforge.net/projects/egd/files/)

[vstinner]

The PyPy patch (and some discussion) is here:

Your patch checks at runtime if libssl comes with RAND_egd:

   HAVE_OPENSSL_RAND_EGD = rffi_platform.Has('RAND_egd')

In CPython, the _ssl module is compiled in C. How can we check if libssl provides RAND_egd() or not at compile time?

Is there a way to check if libssl is OpenSSL or LibreSSL?

[vstinner]

Related discussion:
http://marc.info/?l=openbsd-tech&m=140512043210089&w=2

The answer for Python is:
"your package maintainers and ask them to configure these software without egd support."

[mgorny]

In CPython, the _ssl module is compiled in C. How can we check if libssl provides RAND_egd() or not at compile time?

How about... checking whether the function is provided? Unless I'm missing some major point, AC_CHECK_FUNC should be good enough.

Is there a way to check if libssl is OpenSSL or LibreSSL?

Why would you want to do that? Do you want to make silly assumptions on API depending on provider name, and then add extra conditionals for versions?

[pitrou]

Unless I'm missing some major point, AC_CHECK_FUNC should be good enough.

Building extension modules such as ssl doesn't involve autoconf.

Do you want to make silly assumptions on API depending on provider name, and then add extra conditionals for versions?

Arguably it would be better if LibreSSL exposed the same API as OpenSSL. We're not responsible for the discrepancy here.

[spil]

EGD was only necessary for some commercial UNIX systems, versions that needed it all reached end of life. It no longer makes sense to have any code referring to it.

    EGD needed until        OS release date

IRIX 6.5.19 feb 2003
Solaris 2.6 jul 1997
AIX 5.2 oct 2002
Tru64 5.1B sep 2002
HP-UX 11i v2 sep 2003

Please check OpenBSD's patches to remove EGD support from Python for many versions.
http://cvsweb.openbsd.org/cgi-bin/cvsweb/ports/lang/python/2.7/patches/patch-Lib_ssl_py
http://cvsweb.openbsd.org/cgi-bin/cvsweb/ports/lang/python/3.4/patches/patch-Lib_ssl_py
http://cvsweb.openbsd.org/cgi-bin/cvsweb/ports/lang/python/3.4/patches/patch-Lib_ssl_py

Alternatively see Gentoo's LibreSSL changes https://github.com/Sp1l/libressl/tree/master/dev-lang/python

[vstinner]

We don't drop feature in minor releases, we are working hard to maintain the backward compatibility.

We may only disable RAND_egd if Python is compiled/linked to LibreSSL. So the check should probably be dynamic.

[pitrou]

We're still willing to fix this if someone tells us how to test for LibreSSL in C code.

[spil]

Hi,

I think this can be found in LibreSSL's opensslv.h
An ifdef LIBRESSL_VERSION_NUMBER should work

See https://github.com/libressl-portable/openbsd/blob/master/src/lib/libssl/src/crypto/opensslv.h

_ssl.c includes crypto.h which in turn includes opensslv.h so checking for LIBRESSL_VERSION_NUMBER should provide the correct check.

Attached patch does this in C whereas it should be checked for in configure and disabled with a HAS_RAND_egd
Have not figured out how to do this conditionally in Lib/ssl.py yet

[spil]

When configure is called with correct LDFLAGS and CPPFLAGS for LibreSSL these patches to configure, Modules/_ssl.c and Lib/_ssl.py will detect not having RAND_egd support in OpenSSL and make the build succeed.

[vstinner]

patch-configure.ac:
-AC_DEFINE(__BSD_VISIBLE, 1, [Define on FreeBSD to activate all library features])

Why do you remove this define?

[pitrou]

I thikn RAND_egd() should probably raise NotImplementedError if the function isn't exposed by the ssl library.