(yajl2_c) TypeError when reading from an aiofiles file

[MKuranowski]

Running Python 3.8.2 (on Linux), ijson 3.1.post0, backend yajl2_c, aiofiles 0.5.0.

When trying to asynchronously get items from a simple json file, a TypeError: _get_read() missing 1 required positional argument: 'f' is raised.

Code:

import aiofiles
import asyncio
import ijson

ijson_backend = ijson.get_backend("yajl2_c")

async def main():
    async with aiofiles.open("test.json", "r", encoding="utf-8") as buff:
        async for i in ijson_backend.items_async(buff, "item"):
            print(i)

asyncio.run(main())

test.json:

["1", "2", "3", "4", "5", "6", "7", "8", "9", "10"]

Output:

Traceback (most recent call last):
  File "test_ijson.py", line 12, in <module>
    asyncio.run(main())
  File "/usr/lib/python3.8/asyncio/runners.py", line 43, in run
    return loop.run_until_complete(main)
  File "/usr/lib/python3.8/asyncio/base_events.py", line 616, in run_until_complete
    return future.result()
  File "ignore_test_ijson.py", line 9, in main
    async for t in ijson_backend.items_async(buff, "item"):
TypeError: _get_read() missing 1 required positional argument: 'f'

I would expect normal operation that would print all items from the test.json file.

---

Synchronous code works correctly:

import ijson

ijson_backend = ijson.get_backend("yajl2_c")


def noasync_main():
    with open("test.json", "r", encoding="utf-8") as buff:
        for i in ijson_backend.items(buff, "item"):
            print(i)


noasync_main()

Output:

1
2
3
4
5
6
7
8
9
10

---

Update: the yajl2_cffi backend works correctly:

Code:

import aiofiles
import asyncio
import ijson

ijson_backend = ijson.get_backend("yajl2_cffi")

async def main():
    async with aiofiles.open("test.json", "r", encoding="utf-8") as buff:
        async for i in ijson_backend.items_async(buff, "item"):
            print(i)

asyncio.run(main())

Output:

1
2
3
4
5
6
7
8
9
10

[rtobar]

@MKuranowski thanks for reporting. I can locally reproduce too, which is good (in a way!), but oddly it didn't happen with the unit tests. I'll have to study a bit closer what's going on.

[rtobar]

Found the problems (there were two actually), both of course in the yajl2_c backend only:

• There was an issue if the first call to await f.read() resulted on a yield to the scheduler. This wasn't caught by our unit tests because the simple class we use for testing doesn't yield execution when read is called. After adding an await asyncio.sleep(0) in there the unit tests starting failing in the same way your test script does. This is now fixed, and I pushed the fix to the master branch.
• After fixing that, a second problem came up. This one has to do with the fact that aiofiles implements the read operation (and most of them, I think) using @types.coroutines decorations rather than async def function definitions. The ijson C code wasn't prepared for that case (they are intrinsically different at this level), and I'll need to have a closer inspection yet into how to properly support both. I have something that seems to be working, but is still not fully stable, so I'd rather not push that fix yet.

I don't think I'll be able to tackle this second until early next week though. In the meanwhile you can workaround this by defining your own thin wrapper around aiofiles.open so it defines an async def read coroutine, that should work.

[rtobar]

I just pushed the last set of changes to the master branch that fix the second problem (correctly dealing with @type.coroutine awaitable read functions in the C backend), plus a new set of unit tests that ensure this will not creep up again. @MKuranowski please give this a try to confirm that things are working as intended.

A new release will follow probably tomorrow, time permitting.

[rtobar]

A new 3.1.1 release of ijson including these fixes is now available on PyPI.

[MKuranowski]

Thank you very much, everything's working as intended :)