Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Make Python compatible with OpenSSL 3.0.0 #83001

Closed
tiran opened this issue Nov 16, 2019 · 29 comments
Closed

Make Python compatible with OpenSSL 3.0.0 #83001

tiran opened this issue Nov 16, 2019 · 29 comments
Assignees
Labels
3.9 only security fixes 3.10 only security fixes 3.11 bug and security fixes topic-SSL type-feature A feature request or enhancement

Comments

@tiran
Copy link
Member

tiran commented Nov 16, 2019

BPO 38820
Nosy @tiran, @mcepl, @ambv, @stratakis, @hroncok, @miss-islington, @iritkatriel
PRs
  • bpo-38820: OpenSSL 3.0.0 compatibility. #17190
  • [3.8] bpo-38820: OpenSSL 3.0.0 compatibility. (GH-17190) #17499
  • [3.7] bpo-38820: OpenSSL 3.0.0 compatibility. (GH-17190) #17500
  • bpo-38820: Run CI with OpenSSL 3.0.0-alpha14 #25316
  • bpo-38820: OpenSSL 3.0.0: Use supported hashing algos in doc test (GH-25319) #25319
  • [3.9] bpo-38820: OpenSSL 3.0.0: Use supported hashing algos in doc test (GH-25319) #25320
  • [3.8] bpo-38820: OpenSSL 3.0.0: Use supported hashing algos in doc test (GH-25319) #25321
  • bpo-38820: Test with OpenSSL 3.0.0-alpha15 (GH-25537) #25537
  • bpo-38820: Old OpenSSL 3.0.0 releases are in /old/3.0/ (GH-25624) #25624
  • [3.9] bpo-38820: Old OpenSSL 3.0.0 releases are in /old/3.0/ (GH-25624) #25626
  • [3.8] bpo-38820: Old OpenSSL 3.0.0 releases are in /old/3.0/ (GH-25624) #25627
  • bpo-38820: Add ssl, hashlib, and hmac changes to whatsnew 3.10 (GH-25817) #25817
  • bpo-38820: Test with OpenSSL 3.0.0-alpha16 (GH-25942) #25942
  • [3.10] bpo-38820: Test with OpenSSL 3.0.0-alpha16 (GH-25942) #25943
  • [3.9] bpo-38820: Test with OpenSSL 3.0.0-alpha16 (GH-25942) #25944
  • bpo-38820: Test with OpenSSL 3.0.0-alpha17 #26266
  • [3.10] bpo-38820: Test with OpenSSL 3.0.0-alpha17 (GH-26266) #26269
  • bpo-38820: Test with OpenSSL 3.0.0-beta1 (GH-26769) #26769
  • [3.10] bpo-38820: Test with OpenSSL 3.0.0-beta1 (GH-26769) #26799
  • bpo-38820: Test with OpenSSL 3.0.0 final #28205
  • [3.10] bpo-38820: Test with OpenSSL 3.0.0 final (GH-28205) #28216
  • [3.9] bpo-38820: Test with OpenSSL 3.0.0 final (GH-28205) #28217
  • Dependencies
  • bpo-43788: OpenSSL 3.0.0: Make ssl_data.h version specific
  • bpo-43789: OpenSSL 3.0.0: password callback called multiple times
  • bpo-43791: OpenSSL 3.0.0: TLS 1.0 / 1.1 connections fail with TLSV1_ALERT_INTERNAL_ERROR
  • bpo-43794: OpenSSL 3.0.0: Handle UNEXPECTED_EOF_WHILE_READING / wrap SSL_OP_IGNORE_UNEXPECTED_EOF
  • bpo-43799: OpenSSL 3.0.0: define OPENSSL_API_COMPAT 1.1.1
  • bpo-43811: Run GHA CI with multiple OpenSSL versions
  • bpo-43920: OpenSSL 3.0.0: handle empty cadata consistently
  • Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.

    Show more details

    GitHub fields:

    assignee = 'https://github.com/tiran'
    closed_at = None
    created_at = <Date 2019-11-16.15:06:59.538>
    labels = ['expert-SSL', 'type-feature', '3.9', '3.10', '3.11']
    title = 'Make Python compatible with OpenSSL 3.0.0'
    updated_at = <Date 2021-09-08.17:05:14.504>
    user = 'https://github.com/tiran'

    bugs.python.org fields:

    activity = <Date 2021-09-08.17:05:14.504>
    actor = 'lukasz.langa'
    assignee = 'christian.heimes'
    closed = False
    closed_date = None
    closer = None
    components = ['SSL']
    creation = <Date 2019-11-16.15:06:59.538>
    creator = 'christian.heimes'
    dependencies = ['43788', '43789', '43791', '43794', '43799', '43811', '43920']
    files = []
    hgrepos = []
    issue_num = 38820
    keywords = ['patch']
    message_count = 27.0
    messages = ['356750', '356759', '357979', '357980', '357981', '378790', '378807', '390485', '390650', '390662', '390664', '390666', '391688', '391887', '391892', '391940', '392701', '393098', '394031', '396120', '396123', '397326', '401311', '401355', '401407', '401408', '401409']
    nosy_count = 8.0
    nosy_names = ['christian.heimes', 'mcepl', 'lukasz.langa', 'cstratak', 'hroncok', 'miss-islington', 'iritkatriel', 'bweeks']
    pr_nums = ['17190', '17499', '17500', '25316', '25319', '25320', '25321', '25537', '25624', '25626', '25627', '25817', '25942', '25943', '25944', '26266', '26269', '26769', '26799', '28205', '28216', '28217']
    priority = 'high'
    resolution = None
    stage = 'patch review'
    status = 'open'
    superseder = None
    type = 'enhancement'
    url = 'https://bugs.python.org/issue38820'
    versions = ['Python 3.9', 'Python 3.10', 'Python 3.11']

    @tiran
    Copy link
    Member Author

    tiran commented Nov 16, 2019

    OpenSSL 3.0.0 is currently development [1]. I'm expecting a first beta release in December. Final release is scheduled for Q2 2020. OpenSSL 3.0.0 is API and feature compatible to OpenSSL 1.1.0 and 1.1.1. Only minor changes are required:

    • OpenSSL version number is >= 3.0.0, which breaks test_openssl_version
    • GENERAL_NAME_print() no longer adds trailing newline to IPv6 address strings.
    • ERR_func_error_string is deprecated

    [1] https://www.openssl.org/blog/blog/2019/11/07/3.0-update/

    @tiran tiran added 3.7 (EOL) end of life 3.8 only security fixes 3.9 only security fixes labels Nov 16, 2019
    @tiran tiran self-assigned this Nov 16, 2019
    @tiran tiran added topic-SSL type-feature A feature request or enhancement labels Nov 16, 2019
    @tiran
    Copy link
    Member Author

    tiran commented Nov 16, 2019

    PR #61392 fixes test_openssl_version and removes the trailing newline from IPv6 addresses on all OpenSSL versions. I prefer to have the output consistent on all OpenSSL versions. The newline was silly any way.

    @miss-islington
    Copy link
    Contributor

    New changeset 2b7de66 by Miss Islington (bot) (Christian Heimes) in branch 'master':
    bpo-38820: OpenSSL 3.0.0 compatibility. (GH-17190)
    2b7de66

    @miss-islington
    Copy link
    Contributor

    New changeset 9d3cacd by Miss Islington (bot) in branch '3.8':
    [3.8] bpo-38820: OpenSSL 3.0.0 compatibility. (GH-17190) (GH-17499)
    9d3cacd

    @miss-islington
    Copy link
    Contributor

    New changeset a197f8a by Miss Islington (bot) in branch '3.7':
    [3.7] bpo-38820: OpenSSL 3.0.0 compatibility. (GH-17190) (GH-17500)
    a197f8a

    @iritkatriel
    Copy link
    Member

    Can this be closed?

    @methane methane closed this as completed Oct 17, 2020
    @tiran
    Copy link
    Member Author

    tiran commented Oct 17, 2020

    No, this is still work in progress.

    @tiran tiran reopened this Oct 17, 2020
    @hroncok
    Copy link
    Mannequin

    hroncok mannequin commented Apr 7, 2021

    Python 3.10.0a7 with OpenSSL 3.0 from https://copr.fedorainfracloud.org/coprs/saprasad/openssl-3.0/ in https://copr.fedorainfracloud.org/coprs/g/python/openssl-3.0/package/python3.10/ (full logs available there).

    3 tests failed:
    test_imaplib test_ssl test_urllib2_localnet

    Many:

    ssl.SSLError: [SSL: KRB5_S_TKT_NYV] unexpected eof while reading (_ssl.c:2628)

    Also:

    Traceback (most recent call last):
      File "/builddir/build/BUILD/Python-3.10.0a7/Lib/test/test_ssl.py", line 1413, in test_load_cert_chain
        ctx.load_cert_chain(CERTFILE_PROTECTED, password=getpass_huge)
    SystemError: _PyEval_EvalFrameDefault returned a result with an exception set

    And:

    ssl.SSLCertVerificationError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: self-signed certificate (_ssl.c:1122)

    ssl.SSLError: [SSL: SSLV3_ALERT_BAD_CERTIFICATE] sslv3 alert bad certificate (_ssl.c:1122)

    ssl.SSLError: [SSL: TLSV1_ALERT_UNKNOWN_CA] tlsv1 alert unknown ca (_ssl.c:1122)

    @hroncok hroncok mannequin added the 3.10 only security fixes label Apr 7, 2021
    @tiran tiran removed the 3.7 (EOL) end of life label Apr 9, 2021
    @tiran
    Copy link
    Member Author

    tiran commented Apr 9, 2021

    Miro,

    I have pushed several fixes for OpenSSL 3.0.0

    • bpo-43788 addresses wrong library and error reason codes (e.g. KRB5_S_TKT_NYV)
    • bpo-43789 fixes an issue with exception state in password callbacks (_PyEval_EvalFrameDefault returned a result with an exception set)
    • bpo-43791 disables TLS 1.0 and 1.1 testing with OpenSSL 3.0.0. I'll have to talk to upstream and figure out a better solution.
    • bpo-43794 adds OP_IGNORE_UNEXPECTED_EOF and sets it by default. This makes the code behave like OpenSSL 1.1.0 and 1.0.2.

    I'll look into the other issues next week.

    @tiran
    Copy link
    Member Author

    tiran commented Apr 9, 2021

    New changeset 2d7fdc9 by Christian Heimes in branch 'master':
    bpo-38820: OpenSSL 3.0.0: Use supported hashing algos in doc test (GH-25319)
    2d7fdc9

    @miss-islington
    Copy link
    Contributor

    New changeset ffb05bb by Miss Islington (bot) in branch '3.8':
    bpo-38820: OpenSSL 3.0.0: Use supported hashing algos in doc test (GH-25319)
    ffb05bb

    @miss-islington
    Copy link
    Contributor

    New changeset 7c8796a by Miss Islington (bot) in branch '3.9':
    bpo-38820: OpenSSL 3.0.0: Use supported hashing algos in doc test (GH-25319)
    7c8796a

    @tiran
    Copy link
    Member Author

    tiran commented Apr 23, 2021

    New changeset dcf6581 by Christian Heimes in branch 'master':
    bpo-38820: Test with OpenSSL 3.0.0-alpha15 (GH-25537)
    dcf6581

    @tiran
    Copy link
    Member Author

    tiran commented Apr 26, 2021

    New changeset 3c586ca by Christian Heimes in branch 'master':
    bpo-38820: Old OpenSSL 3.0.0 releases are in /old/3.0/ (GH-25624)
    3c586ca

    @miss-islington
    Copy link
    Contributor

    New changeset 10ee266 by Miss Islington (bot) in branch '3.8':
    [3.8] bpo-38820: Old OpenSSL 3.0.0 releases are in /old/3.0/ (GH-25624) (GH-25627)
    10ee266

    @miss-islington
    Copy link
    Contributor

    New changeset 3b917d1 by Miss Islington (bot) in branch '3.9':
    [3.9] bpo-38820: Old OpenSSL 3.0.0 releases are in /old/3.0/ (GH-25624) (GH-25626)
    3b917d1

    @tiran
    Copy link
    Member Author

    tiran commented May 2, 2021

    New changeset d8389e3 by Christian Heimes in branch 'master':
    bpo-38820: Add ssl, hashlib, and hmac changes to whatsnew 3.10 (GH-25817)
    d8389e3

    @miss-islington
    Copy link
    Contributor

    New changeset f8778f9 by Miss Islington (bot) in branch '3.10':
    bpo-38820: Test with OpenSSL 3.0.0-alpha16 (GH-25942)
    f8778f9

    @miss-islington
    Copy link
    Contributor

    New changeset 36843f7 by Miss Islington (bot) in branch '3.10':
    bpo-38820: Test with OpenSSL 3.0.0-alpha17 (GH-26266)
    36843f7

    @tiran
    Copy link
    Member Author

    tiran commented Jun 19, 2021

    New changeset 44fb551 by Christian Heimes in branch 'main':
    bpo-38820: Test with OpenSSL 3.0.0-beta1 (GH-26769)
    44fb551

    @miss-islington
    Copy link
    Contributor

    New changeset c6cd2ec by Miss Islington (bot) in branch '3.10':
    [3.10] bpo-38820: Test with OpenSSL 3.0.0-beta1 (GH-26769) (GH-26799)
    c6cd2ec

    @ambv
    Copy link
    Contributor

    ambv commented Jul 12, 2021

    New changeset c92b391 by Christian Heimes in branch '3.9':
    [3.9] bpo-38820: Test with OpenSSL 3.0.0-alpha16 (GH-25942) (bpo-25944)
    c92b391

    @ambv
    Copy link
    Contributor

    ambv commented Sep 7, 2021

    New changeset cc7c680 by Christian Heimes in branch 'main':
    bpo-38820: Test with OpenSSL 3.0.0 final (GH-28205)
    cc7c680

    @miss-islington
    Copy link
    Contributor

    New changeset 2fe15db by Miss Islington (bot) in branch '3.10':
    bpo-38820: Test with OpenSSL 3.0.0 final (GH-28205)
    2fe15db

    @ambv
    Copy link
    Contributor

    ambv commented Sep 8, 2021

    New changeset 7a6178a by Łukasz Langa in branch '3.9':
    [3.9] bpo-38820: Test with OpenSSL 3.0.0 final (GH-28205) (GH-28217)
    7a6178a

    @ambv
    Copy link
    Contributor

    ambv commented Sep 8, 2021

    Christian, Python is now tested with 3.0.0 final in 3.9, 3.10, and 3.11. Looks like we can close this!

    Thank you for this big body of work ✨ 🍰 ✨

    @ambv ambv added 3.11 bug and security fixes and removed 3.8 only security fixes labels Sep 8, 2021
    @ambv
    Copy link
    Contributor

    ambv commented Sep 8, 2021

    (I'll let you close this yourself when you determine that the two remaining open dependencies can be closed as well.)

    @ezio-melotti ezio-melotti transferred this issue from another repository Apr 10, 2022
    @Steve123p
    Copy link

    Is this going to be officially closed?

    @kumaraditya303
    Copy link
    Contributor

    I am closing this as it is fixed, if required it can reopened.

    Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
    Labels
    3.9 only security fixes 3.10 only security fixes 3.11 bug and security fixes topic-SSL type-feature A feature request or enhancement
    Projects
    None yet
    Development

    No branches or pull requests

    7 participants