Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add AF_ALG (Linux Kernel crypto) to socket module #71931

Closed
tiran opened this issue Aug 12, 2016 · 22 comments
Closed

Add AF_ALG (Linux Kernel crypto) to socket module #71931

tiran opened this issue Aug 12, 2016 · 22 comments
Labels
extension-modules C modules in the Modules dir type-feature A feature request or enhancement

Comments

@tiran
Copy link
Member

tiran commented Aug 12, 2016

BPO 27744
Nosy @vstinner, @tiran, @vadmium, @Lukasa, @zhangyangyu
Files
  • AF_ALG-kernel-crypto-support-for-socket-module.patch
  • AF_ALG-kernel-crypto-support-for-socket-module-1.patch
  • AF_ALG-kernel-crypto-support-for-socket-module-2.patch
  • AF_ALG-kernel-crypto-support-for-socket-module-3.patch
  • AF_ALG-kernel-crypto-support-for-socket-module-4.patch
  • AF_ALG-kernel-crypto-support-for-socket-module-5.patch
  • Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.

    Show more details

    GitHub fields:

    assignee = None
    closed_at = <Date 2021-10-12.09:36:25.702>
    created_at = <Date 2016-08-12.10:21:30.746>
    labels = ['extension-modules', 'type-feature']
    title = 'Add AF_ALG (Linux Kernel crypto) to socket module'
    updated_at = <Date 2021-10-12.09:36:25.702>
    user = 'https://github.com/tiran'

    bugs.python.org fields:

    activity = <Date 2021-10-12.09:36:25.702>
    actor = 'christian.heimes'
    assignee = 'none'
    closed = True
    closed_date = <Date 2021-10-12.09:36:25.702>
    closer = 'christian.heimes'
    components = ['Extension Modules']
    creation = <Date 2016-08-12.10:21:30.746>
    creator = 'christian.heimes'
    dependencies = []
    files = ['44118', '44140', '44248', '44268', '44295', '44359']
    hgrepos = []
    issue_num = 27744
    keywords = ['patch']
    message_count = 22.0
    messages = ['272516', '272746', '272918', '273018', '273838', '274365', '274442', '274445', '274453', '274458', '274508', '274516', '274520', '274543', '274553', '274557', '275702', '275824', '276010', '287497', '287502', '287503']
    nosy_count = 7.0
    nosy_names = ['vstinner', 'christian.heimes', 'python-dev', 'martin.panter', 'Lukasa', 'xiang.zhang', 'Christian H']
    pr_nums = []
    priority = 'normal'
    resolution = 'fixed'
    stage = 'resolved'
    status = 'closed'
    superseder = None
    type = 'enhancement'
    url = 'https://bugs.python.org/issue27744'
    versions = ['Python 3.6']

    @tiran
    Copy link
    Member Author

    tiran commented Aug 12, 2016

    Linux has a netlink-based user-space interface for Kernel cryptography. Kernel based crypto has a couple of advantages that are explained at http://www.chronox.de/libkcapi/html/ch01s02.html . The document doesn't mention that a crypto socket also supports splicing and sendfile. Files no longer have to be copied to user-space.

    My experimental branch https://github.com/tiran/cpython/commits/feature/af_alg implements af_alg support. Example:

    from socket import socket, AF_ALG, SOCK_SEQPACKET, SOL_ALG, ALG_SET_KEY
    from binascii import hexlify
    with socket(AF_ALG, SOCK_SEQPACKET, 0) as alg:
        alg.bind(('hash', 'hmac(sha512)'))
        alg.setsockopt(SOL_ALG, ALG_SET_KEY, b'key')
        op, _ = alg.accept()
        with open('/etc/passwd', 'rb') as f:
            op.sendfile(f)
        print(hexlify(op.recv(64)))
        op.close()

    @tiran tiran added extension-modules C modules in the Modules dir type-feature A feature request or enhancement labels Aug 12, 2016
    @tiran
    Copy link
    Member Author

    tiran commented Aug 15, 2016

    Working patch with tests and documentation.

    socket.algset() isn't strictly necessary but makes the feature much more pleasant to use. I accept ideas for a better name, though.

    @vstinner
    Copy link
    Member

    vstinner commented Aug 17, 2016

    I reviewed AF_ALG-kernel-crypto-support-for-socket-module.patch.

    @tiran
    Copy link
    Member Author

    tiran commented Aug 18, 2016

    Thanks for your review, Victor. I have addressed most of your remarks.

    • algset() is now called sendmsg_afalg(). It behaves more like a specialized version of sendmsg() and can optionally handle an array of iovec.

    • I had to add another variant of setsockopt that sends NULL, int instead of (char*)int, sizeof(int) to get the AEAD GCM tests working. AEAD expects ALG_SET_AEAD_AUTHSIZE as (NULL, taglen). algo.setsockopt(SOL_ALG, ALG_SET_AEAD_AUTHSIZE, (None, taglen)) sends optval=NULL, optlen=taglen.

    • Added tests for AES-CBC decryption, AEAD AES-GCM and DRBG.

    @tiran
    Copy link
    Member Author

    tiran commented Aug 28, 2016

    New patch with setsockopt(socket.SOL_ALG, socket.ALG_SET_AEAD_AUTHSIZE, None, taglen) instead of (None, taglen).

    @tiran
    Copy link
    Member Author

    tiran commented Sep 4, 2016

    I have removed binascii.(un)hexlify().

    @python-dev
    Copy link
    Mannequin

    python-dev mannequin commented Sep 5, 2016

    New changeset 74ce062a0397 by Christian Heimes in branch 'default':
    Issue bpo-27744: Add AF_ALG (Linux Kernel crypto) to socket module.
    https://hg.python.org/cpython/rev/74ce062a0397

    @python-dev
    Copy link
    Mannequin

    python-dev mannequin commented Sep 5, 2016

    New changeset 52404f9596b5 by Christian Heimes in branch 'default':
    Issue bpo-27744: correct comment and markup
    https://hg.python.org/cpython/rev/52404f9596b5

    @python-dev
    Copy link
    Mannequin

    python-dev mannequin commented Sep 5, 2016

    New changeset ee32af890e27 by Christian Heimes in branch 'default':
    bpo-27744: Check for AF_ALG support in Kernel
    https://hg.python.org/cpython/rev/ee32af890e27

    @python-dev
    Copy link
    Mannequin

    python-dev mannequin commented Sep 5, 2016

    New changeset 4ebe3ade6922 by Christian Heimes in branch 'default':
    bpo-27744: AES-CBC and DRBG need Kernel 3.19+
    https://hg.python.org/cpython/rev/4ebe3ade6922

    @vadmium
    Copy link
    Member

    vadmium commented Sep 6, 2016

    Despite the last changes, test_aes_cbc() hangs for fifteen minutes:
    http://buildbot.python.org/all/builders/x86-64%20Ubuntu%2015.10%20Skylake%20CPU%203.x/builds/1298/steps/test/logs/stdio
    running: test_socket (900 sec)
    0:28:47 [332/402] test_socket crashed
    Timeout (0:15:00)!
    Thread 0x00007f2843d94700 (most recent call first):
    File "/home/buildbot/buildarea/3.x.intel-ubuntu-skylake/build/Lib/test/test_socket.py", line 5389 in test_aes_cbc
    File "/home/buildbot/buildarea/3.x.intel-ubuntu-skylake/build/Lib/test/support/init.py", line 523 in wrapper
    File "/home/buildbot/buildarea/3.x.intel-ubuntu-skylake/build/Lib/unittest/case.py", line 600 in run
    . . .

    @vadmium
    Copy link
    Member

    vadmium commented Sep 6, 2016

    Also, the Gentoo buildbots fail:
    http://buildbot.python.org/all/builders/x86%20Gentoo%20Non-Debug%20with%20X%203.x/builds/1368/steps/test/logs/stdio
    ======================================================================
    ERROR: test_aead_aes_gcm (test.test_socket.LinuxKernelCryptoAPI)
    ----------------------------------------------------------------------

    Traceback (most recent call last):
      File "/buildbot/buildarea/3.x.ware-gentoo-x86.nondebug/build/Lib/test/support/__init__.py", line 523, in wrapper
        return func(*args, **kw)
      File "/buildbot/buildarea/3.x.ware-gentoo-x86.nondebug/build/Lib/test/test_socket.py", line 5428, in test_aead_aes_gcm
        with self.create_alg('aead', 'gcm(aes)') as algo:
      File "/buildbot/buildarea/3.x.ware-gentoo-x86.nondebug/build/Lib/test/test_socket.py", line 5346, in create_alg
        sock.bind((typ, name))
    FileNotFoundError: [Errno 2] No such file or directory

    Similar failures for test_aes_cbc test_drbg_pr_sha256 test_hmac_sha1 test_sha256.

    @vstinner
    Copy link
    Member

    vstinner commented Sep 6, 2016

    Also, the Gentoo buildbots fail:
    http://buildbot.python.org/all/builders/x86%20Gentoo%20Non-Debug%20with%20X%203.x/builds/1368/steps/test/logs/stdio
    ======================================================================
    ERROR: test_aead_aes_gcm (test.test_socket.LinuxKernelCryptoAPI)

    It's Linux 4.4.6. configure says "checking for sockaddr_alg... yes".

    @python-dev
    Copy link
    Mannequin

    python-dev mannequin commented Sep 6, 2016

    New changeset e3b83bfa02c5 by Christian Heimes in branch 'default':
    bpo-27744: skip test if AF_ALG socket bind fails
    https://hg.python.org/cpython/rev/e3b83bfa02c5

    @tiran
    Copy link
    Member Author

    tiran commented Sep 6, 2016

    Some distributions mess with the Kernel or disable user-space crypto. I have added some tweaks and fixed a couple of buildbots. I don't know what is going on with x86-64 Ubuntu 15.10 Skylake CPU. It's a Kernel 4.2 machine and should support AES-CBC.

    @zhangyangyu
    Copy link
    Member

    zhangyangyu commented Sep 6, 2016

    My PC is Ubuntu15.10, kernel 4.2, though CPU not Skylake. Everything works fine.

    test_aead_aes_gcm (test.test_socket.LinuxKernelCryptoAPI) ... skipped "('[Errno 2] No such file or directory', 'aead', 'gcm(aes)')"
    test_aes_cbc (test.test_socket.LinuxKernelCryptoAPI) ... ok
    test_drbg_pr_sha256 (test.test_socket.LinuxKernelCryptoAPI) ... ok
    test_hmac_sha1 (test.test_socket.LinuxKernelCryptoAPI) ... ok
    test_sendmsg_afalg_args (test.test_socket.LinuxKernelCryptoAPI) ... ok
    test_sha256 (test.test_socket.LinuxKernelCryptoAPI) ... ok

    @tiran
    Copy link
    Member Author

    tiran commented Sep 10, 2016

    x86-64 Ubuntu 15.10 Skylake CPU 3.x is still blocking. It looks like I have to add another workaround for a Ubuntu quirk.

    @python-dev
    Copy link
    Mannequin

    python-dev mannequin commented Sep 11, 2016

    New changeset 55d77f5a7cb3 by Christian Heimes in branch 'default':
    bpo-27744: test_aes_cbc is blocking x86-64 Ubuntu 15.10 Skylake CPU 3.x for a while. Require Kernel 4.3+ for now
    https://hg.python.org/cpython/rev/55d77f5a7cb3

    @python-dev
    Copy link
    Mannequin

    python-dev mannequin commented Sep 12, 2016

    New changeset a951f8f30922 by Victor Stinner in branch 'default':
    Cleanup socketmodule.c
    https://hg.python.org/cpython/rev/a951f8f30922

    New changeset 3a6917c73857 by Victor Stinner in branch 'default':
    socket: Fix memory leak in sendmsg() and sendmsg_afalg()
    https://hg.python.org/cpython/rev/3a6917c73857

    @vstinner
    Copy link
    Member

    vstinner commented Feb 10, 2017

    What is the status of this issue? test_aead_aes_gcm() fails on my Fedora 25 (Python: default branch).

    haypo@selma$ cat /etc/fedora-release
    Fedora release 25 (Twenty Five)
    haypo@selma$ uname -r
    4.9.5-200.fc25.x86_64

    test test_socket failed -- Traceback (most recent call last):
      File "/home/haypo/prog/python/default/Lib/test/support/__init__.py", line 556, in wrapper
        return func(*args, **kw)
      File "/home/haypo/prog/python/default/Lib/test/test_socket.py", line 5515, in test_aead_aes_gcm
        res = op.recv(assoclen + len(plain) + taglen)
    OSError: [Errno 22] Invalid argument

    @tiran
    Copy link
    Member Author

    tiran commented Feb 10, 2017

    I'll look into the matter and push a fix after the migration to github today.

    @tiran
    Copy link
    Member Author

    tiran commented Feb 10, 2017

    By the way problem with AES-GCM is tracked in https://bugs.python.org/issue29324 . It was caused in a Kernel API change. Jan has provided a fix. I need to find some spare time to dig into Kernel sources and verify the patch.

    @tiran tiran closed this as completed Oct 12, 2021
    @ezio-melotti ezio-melotti transferred this issue from another repository Apr 10, 2022
    Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
    Labels
    extension-modules C modules in the Modules dir type-feature A feature request or enhancement
    Projects
    None yet
    Development

    No branches or pull requests

    4 participants