Trouble porting my code to caio

[FFY00]

Hi, I am having some trouble using caio in my project. I am pretty new to async so I may be doing something stupid 😅

I tried moving my code to async since it can be a bit IO intensive and lower delays do make a difference in my application.

WIP patch: FFY00/python-uhid@b2b49aa
Code: https://github.com/FFY00/python-uhid/tree/async

I am testing with the following snippet:

#!/usr/bin/env python

import uhid

async def create(n = 0):
    return await uhid.UHIDDevice.initialize(
            0x0000, n, f'Test Device {n}', [
            # Generic mouse report descriptor
            0x05, 0x01,  # Usage Page (Generic Desktop)        0
            0x09, 0x02,  # Usage (Mouse)                       2
            0xa1, 0x01,  # Collection (Application)            4
            0x09, 0x02,  # .Usage (Mouse)                      6
            0xa1, 0x02,  # .Collection (Logical)               8
            0x09, 0x01,  # ..Usage (Pointer)                   10
            0xa1, 0x00,  # ..Collection (Physical)             12
            0x05, 0x09,  # ...Usage Page (Button)              14
            0x19, 0x01,  # ...Usage Minimum (1)                16
            0x29, 0x03,  # ...Usage Maximum (3)                18
            0x15, 0x00,  # ...Logical Minimum (0)              20
            0x25, 0x01,  # ...Logical Maximum (1)              22
            0x75, 0x01,  # ...Report Size (1)                  24
            0x95, 0x03,  # ...Report Count (3)                 26
            0x81, 0x02,  # ...Input (Data,Var,Abs)             28
            0x75, 0x05,  # ...Report Size (5)                  30
            0x95, 0x01,  # ...Report Count (1)                 32
            0x81, 0x03,  # ...Input (Cnst,Var,Abs)             34
            0x05, 0x01,  # ...Usage Page (Generic Desktop)     36
            0x09, 0x30,  # ...Usage (X)                        38
            0x09, 0x31,  # ...Usage (Y)                        40
            0x15, 0x81,  # ...Logical Minimum (-127)           42
            0x25, 0x7f,  # ...Logical Maximum (127)            44
            0x75, 0x08,  # ...Report Size (8)                  46
            0x95, 0x02,  # ...Report Count (2)                 48
            0x81, 0x06,  # ...Input (Data,Var,Rel)             50
            0xc0,        # ..End Collection                    52
            0xc0,        # .End Collection                     53
            0xc0,        # End Collection                      54
        ],
    )


async def main():
    wait = []
    for i in range(5):
        wait.append(create(i))

    await asyncio.gather(*wait)


import asyncio
loop = asyncio.get_event_loop()
#loop.run_until_complete(main())
loop.run_until_complete(create())

But I get the following error:

Traceback (most recent call last):
  File "/home/anubis/.virtualenvs/caio/lib/python3.8/site-packages/caio/asyncio_base.py", line 43, in step
    self.context.submit(*operations.keys())
SystemError: (22, 'Invalid argument')

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "./test.py", line 56, in <module>
    loop.run_until_complete(create())
  File "/usr/lib/python3.8/asyncio/base_events.py", line 616, in run_until_complete
    return future.result()
  File "./test.py", line 9, in create
    return await uhid.UHIDDevice.initialize(
  File "/home/anubis/git/python-uhid/uhid/__init__.py", line 239, in initialize
    await device.create()
  File "/home/anubis/git/python-uhid/uhid/__init__.py", line 316, in create
    await self._uhid.create(
  File "/home/anubis/git/python-uhid/uhid/__init__.py", line 220, in create
    await self._send_event(_EventType.UHID_CREATE2, _Create2Req(
  File "/home/anubis/git/python-uhid/uhid/__init__.py", line 186, in _send_event
    n = await self._ctx.write(bytes(data), self._uhid, offset=0)
  File "/home/anubis/.virtualenvs/caio/lib/python3.8/site-packages/caio/asyncio_base.py", line 87, in submit
    await future
  File "/home/anubis/.virtualenvs/caio/lib/python3.8/site-packages/caio/asyncio_base.py", line 48, in step
    self.context.submit(operation)
SystemError: (22, 'Invalid argument')

Do you have any idea of what may be happening here?

[honglei]

I've got the same problem when using aiofile.

[mosquito]

I think It's fixed in caio==0.6.0

[honglei]

aiofile 3.0.0 requires caio~=0.5.3, but you'll have caio 0.6.0 which is incompatible.

[mosquito]

Oh. My bad

[mosquito]

@honglei aiofile==3.1.0 has been released.

[mosquito]

@honglei actually you should use Linux kernel >=4.18 it's was not a bug that's was the fixes avoiding compatibility issue. io_submit system call supports all required features since Linux 4.18.

The thread-based implementation will be used otherwise.

[honglei]

My kernel is 4.4.15 and I cannot update it, when using fsync , it raised "Bad file descriptor" Error.

[honglei]

@mosquito I think caio should choose to use theaded-mode if it cannot work correctly before linux kernel 4.18

[mosquito]

@honglei that's will happen transparently for you. When caio importing selects preferred implementation.

[honglei]

@mosquito But under Linux 4.4.15/Python3.8/caio 0.60, caio.preferred is caio.linux_aio

[mosquito]

@mosquito But under Linux 4.4.15/Python3.8/caio 0.60, caio.preferred is caio.linux_aio

@honglei Please provide uname -r output.

[honglei]

4.4.15-deepin-wutip-421-inspur-monitor-native-debugless

[honglei]

>>> os.uname()
posix.uname_result(sysname='Linux', nodename='lccy6', release='4.4.15-deepin-wutip-421-inspur-monitor-native-debugless', version='#298 SMP Mon Jun 29 14:14:12 CST 2020', machine='sw_64')

[FFY00]

Hi! I am running Linux 5.8.3 and just updated to caio 0.6.0, still the same thing 😕

$ uname -a
Linux genesis 5.8.3-arch1-1 #1 SMP PREEMPT Fri, 21 Aug 2020 16:54:16 +0000 x86_64 GNU/Linux

$ python -c 'import caio; print(caio.__version__)'
0.6.0

[mosquito]

@FFY00 @honglei Thanks for your persistence, it's really my bug. I found a solution and soon will release it.

[honglei]

Maybe this way:

import os
if os.name =='posix':
    os_version = os.uname().release.split("-")[0]
    version = tuple( [int(i) for i in os_version.split(".")])
    if version<(4,18):
        print("thread_aio")
    else:
        print("linux_aio")

[mosquito]

@honglei actually something like this but on c. Anyway, 0.6.1 has been released. I will happy if you try to check this out.

[FFY00]

Unfortunately this still doesn't fix my issue. My kernel is very recent, do you have any idea why this might not be working?

[mosquito]

@FFY00 it's happening when the kernel is not able to perform the asynchronous operation with this file. I found a problem with fsync/fdsync operations on old kernels, and avoid this. I guess that's isn't the last issue with old kernels, but when you bring some details e.g. environment, kernel version Linux distro, etc. I might reproduce this on VM, I guess, and try to resolve it, or restrict libaio based implementation for this kernel as a bad case.

[mosquito]

@FFY00 So after fast reading the documentation about UHID I found out this:

Non-blocking operations are supported by setting O_NONBLOCK.

Probably you do not need use caio for this (I guess kernel is not able to handle this file descriptor) try something like this:

import os, asyncio, fcntl

f = open(...)

rv = fcntl.fcntl(f, fcntl.F_SETFL, os.O_NONBLOCK)


def ready_to_read(fp):
    pass


def ready_to_write(fp):
    pass


loop = asyncio.get_event_loop()
loop.add_reader(f.fileno(), ready_to_read, f)
loop.add_writer(f.fileno(), ready_to_write, f)

[honglei]

Still linux_aio under kernel 4.4.15

lccy@lccy6:/opt/python38/bin$ ./python3
Python 3.8.4rc1 (default, Jul  1 2020, 11:44:28)
[GCC 5.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import caio
<frozen importlib._bootstrap>:219: RuntimeWarning: Linux supports fsync/fdsync with io_submit since 4.18 but current kernel 4.4.15-deepin-wutip-421-inspur-monitor-native-debugless doesn't support it. Related calls will have no effect.
>>> caio.preferred
<module 'caio.linux_aio' from '/opt/python38/lib/python3.8/site-packages/caio/linux_aio.cpython-38.so'>
>>> caio.__version__
'0.6.1'
>>> os.uname()
posix.uname_result(sysname='Linux', nodename='lccy6', release='4.4.15-deepin-wutip-421-inspur-monitor-native-debugless', version='#298 SMP Mon Jun 29 14:14:12 CST 2020', machine='sw_64')

[honglei]

@mosquito I think the last modification only give a warning, but have no effect for caio.preferred selection.

[mosquito]

@honglei please see Troubleshooting part in README.rst