YAJL backend not working in Win/Conda

[JoejynWan]

I am trying to implement ijson, and have troubles using the yajl2/yajl2_cffi/yajl2_c backends. I have no problems when using the default python backend, but this is working exceptionally slowly so when I try to implement the other backends, I am facing the following errors:

Implementing the yajl2_cffi backend:

>>> import ijson.backends.yajl2_cffi as ijson
Traceback (most recent call last):
  File "C:\Users\Joejyn\.conda\envs\sin_classifier\lib\site-packages\ijson\backends\__init__.py", line 51, in find_yajl_cffi
    yajl = ffi.dlopen(os.getenv('YAJL_DLL') or 'yajl')
  File "C:\Users\Joejyn\.conda\envs\sin_classifier\lib\site-packages\cffi\api.py", line 150, in dlopen
    lib, function_cache = _make_ffi_library(self, name, flags)
  File "C:\Users\Joejyn\.conda\envs\sin_classifier\lib\site-packages\cffi\api.py", line 832, in _make_ffi_library
    backendlib = _load_backend_lib(backend, libname, flags)
  File "C:\Users\Joejyn\.conda\envs\sin_classifier\lib\site-packages\cffi\api.py", line 827, in _load_backend_lib
    raise OSError(msg)
OSError: ctypes.util.find_library() did not manage to locate a library called 'yajl'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "C:\Users\Joejyn\.conda\envs\sin_classifier\lib\site-packages\ijson\backends\yajl2_cffi.py", line 64, in <module>
    yajl = backends.find_yajl_cffi(ffi, 2)
  File "C:\Users\Joejyn\.conda\envs\sin_classifier\lib\site-packages\ijson\backends\__init__.py", line 53, in find_yajl_cffi
    raise YAJLImportError('Unable to load YAJL.')
ijson.backends.YAJLImportError: Unable to load YAJL.`

Implementing the yajl2_c backend:

>>> import ijson.backends.yajl2_c as ijson
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "C:\Users\Joejyn\.conda\envs\sin_classifier\lib\site-packages\ijson\backends\yajl2_c.py", line 13, in <module>
    from . import _yajl2
ImportError: DLL load failed while importing _yajl2: The specified module could not be found.
>>> import ijson.backends.yajl2_c as ijson
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "C:\Users\Joejyn\.conda\envs\sin_classifier\lib\site-packages\ijson\backends\yajl2_c.py", line 13, in <module>
    from . import _yajl2
ImportError: DLL load failed while importing _yajl2: The specified module could not be found.

Implementing the yajl2 backend:

>>> import ijson.backends.yajl2 as ijson
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "C:\Users\Joejyn\.conda\envs\sin_classifier\lib\site-packages\ijson\backends\yajl2.py", line 11, in <module>
    yajl = _yajl2_ctypes_common.get_yajl(2)
  File "C:\Users\Joejyn\.conda\envs\sin_classifier\lib\site-packages\ijson\backends\_yajl2_ctypes_common.py", line 46, in get_yajl
    yajl = backends.find_yajl_ctypes(version)
  File "C:\Users\Joejyn\.conda\envs\sin_classifier\lib\site-packages\ijson\backends\__init__.py", line 37, in find_yajl_ctypes
    raise YAJLImportError('YAJL shared object not found.')
ijson.backends.YAJLImportError: YAJL shared object not found.

I believe that I do have the required packages installed within the Anaconda package. Is this issue due to ctypes not being able to find the yajl package within the conda environment? Would greatly appreciate some assistance. Thank you.

Execution information:

• Python version 3.9.9
• ijson version 3.1.4
• yajl version 2.1.0
• cffi version 1.15.0
• conda version 4.13.0
• Windows 11

[rtobar]

@JoejynWan thanks for the detailed report.

Sadly I won't be able to replicate your environment, so I'm working solely in your information and on hopefully educated guesses.

How did you install ijson? Was that using cinda itself, or via pip? What I'd basically like to know is whether you got the ijson binary wheel that we publish in PyPI (that should include the yajl2_c backend) or not. When using pip you should have gotten that (we have binary wheels for wib32/64 for python 3.9), but I'm not sure what happens when "cinda installing" ijson (do they provide their own compiled-or-not-actually-conpiled package?). In other words, try installibg via pip as that should give you a compiled package

Regarding ctyoes bit finding your yajl library: I'm not expert on how Windows finds DLLs, even less how cobda organised them. However ijson has an option to point to a yajl DLL directly:

ijson/ijson/backends/__init__.py

Line 35 in 3100b96

so_name = os.getenv('YAJL_DLL') or util.find_library('yajl')

Give that a try and hopefully you'll be able to use the ctypes-and cffi-based backends.

[JoejynWan]

@rtobar Thank you so much for your quick reply.

I had previously installed ijson with conda, and that does seem to be the problem. I uninstalled it and installed ijson with pip (though yajl is still installed with conda since I somehow could not get it to install with pip) and it worked.

Thanks so much again!

[rtobar]

@JoejynWan thanks for confirming that it's all good now, I'm happy I could be of assistance.