Unable to run wrapper when installed in custom directory for Ren'Py

[fantonhike]

Hello,
apologies if this is not the right place for this question, but I can't figure it out.

I'm trying to use the wrapper for a game made with Ren'Py. Normally it can use 3rd party modules from a folder called 'python-packages' in the game's directory (it works fine with the keyboard module, for example). However when I try to run it, it gives me the error for the line ahk = AHK() :

  File "python-packages/ahk/_sync/engine.py", line 130, in __init__
  File "python-packages/ahk/_sync/transport.py", line 610, in __init__
  File "python-packages/jinja2/loaders.py", line 323, in __init__
ValueError: The 'ahk' package was not installed in a way that PackageLoader understands.

I tried with 1.1.0, .2 and .3 but all 3 gave the same result. Re-installing ahk and python didn't do anything either.

Thanks

[spyoungtech]

Hmm. Unfortunately, I'm not familiar with making Ren'Py applications, so I'm not able to reproduce this issue. Did you use pip install --target path/to/python-packages/ ahk to install? Or some other method?

As a debugging measure, you can try to modify the relevant code in jinja (in your python-packages/jinja2/loaders.py file) to see why it cannot find the template root and let me know if you find out something useful. I'm guessing that something is unexpected with the value returned by spec.submodule_search_locations and/or spec.origin.

A similar issue was reported in jinja. The author of the issue discussed some code that identifies a problem (submodule_search_locations is None unexpectedly) and suggests a possible patch that was never applied in jinja (evidently, the problem was solved another way). As I understand it, submodule_search_locations is obtained from the package's __path__ attribute. Maybe something Ren'Py is [not] doing causes this to be None unexpectedly?

Anyhow. I believe it's possible for me to make a patch in ahk that gets around this problem, but it would be ideal if I had enough details about your problem to make sure I can reproduce the exact same problem you're having and test the fix.

[spyoungtech]

@fantonhike I've created (what I hope is) a fix and released it as a prerelease under version 1.1.3rc1. Can you try installing that and see if your problem is resolved?

First, remove ahk from your Ren'Py python-packages/ then reinstall using pip with the --pre flag and version 1.1.3rc1 like:

pip install --pre --target python-packages/ "ahk==1.1.3rc1"

Let me know if that changes anything with your issue.

[fantonhike]

Thank you so much for the advice, unfortunately the patch doesn't fix the error (although based on what you say it seems like it aught to), and the same message comes up.

I did indeed install with pip install --target path/to/python-packages/ ahk, with powershell on Windows 10.

And I have looked into the loaders.py file in jinja, but I have so far not managed to find out why it's not finding the template root. I'll keep you updated if I do.

In case it helps, here's the full traceback (using the new patch):

I'm sorry, but an uncaught exception occurred.

While running game code:
  File "game/script.rpy", line 1, in script
    init python:
  File "game/script.rpy", line 1, in script
    init python:
  File "game/script.rpy", line 3, in <module>
    ahk = AHK()
ValueError: The 'ahk' package was not installed in a way that PackageLoader understands.

-- Full Traceback ------------------------------------------------------------

Full traceback:
  File "C:\Users\fanto\Desktop\renpy-8.0.3-sdk\renpy\bootstrap.py", line 277, in bootstrap
    renpy.main.main()
  File "C:\Users\fanto\Desktop\renpy-8.0.3-sdk\renpy\main.py", line 558, in main
    renpy.game.context().run(node)
  File "game/script.rpy", line 1, in script
    init python:
  File "/home/tom/ab/renpy-build/tmp/install.linux-x86_64/lib/python3.9/site-packages/future/utils/__init__.py", line 441, in raise_
  File "game/script.rpy", line 1, in script
    init python:
  File "C:\Users\fanto\Desktop\renpy-8.0.3-sdk\renpy\ast.py", line 1131, in execute
    renpy.python.py_exec_bytecode(self.code.bytecode, self.hide, store=self.store)
  File "C:\Users\fanto\Desktop\renpy-8.0.3-sdk\renpy\python.py", line 1061, in py_exec_bytecode
    exec(bytecode, globals, locals)
  File "game/script.rpy", line 3, in <module>
    ahk = AHK()
  File "python-packages/ahk/_sync/engine.py", line 130, in __init__
  File "python-packages/ahk/_sync/transport.py", line 630, in __init__
  File "python-packages/ahk/_sync/transport.py", line 320, in __init__
  File "python-packages/ahk/_hotkey.py", line 128, in __init__
  File "python-packages/jinja2/loaders.py", line 323, in __init__
ValueError: The 'ahk' package was not installed in a way that PackageLoader understands.

Windows-10-10.0.19044 AMD64
Ren'Py 8.0.3.22090809
test_for_ahk 1.0
Wed May 17 12:53:55 2023

I've also read that PackageLoader in jinja can run into difficulty in some cases, and it may be possible to use FileSystemLoader instead? I don't know if that's at all relevant here though.

Thank you for the help

[spyoungtech]

Thank you for assisting in fixing the issue and thanks for the full traceback, that helps a lot. It looks like there's more than one place where I was trying to use PackageLoader that I did not patch. I've released another version: 1.1.3rc2 that fixes that issue.

Give that a shot @fantonhike

[spyoungtech]

I believe I'm also getting closer to understanding why Ren'Py applications have this behavior. Ren'Py initializes Python and does some hacks to modify sys.meta_path to be able to load third-party packages from python-packages/ -- I suspect their implementation is not fully compatible with normal Python package finders/loaders (per PEP302) and a missing behavior in their loader breaks how jinja finds package data files.

If you wouldn't mind helping me out with some additional information, can you add these lines of code before you call AHK() and share the output from the print calls when trying to run your app?

import importlib.util
from importlib import import_module

import_module('ahk')
spec = importlib.util.find_spec('ahk')
print('origin', spec.origin)
print('submodule_search_locations', spec.submodule_search_locations)

[fantonhike]

Thank you so much!! You have my immense gratitude.

The new patch fixes the issue, it seems to work completely fine now.

I'm more than happy to help with getting more information on how Ren'Py works. I added the code but it seems that it doesn't manage to retrieve anything. (Or maybe it's a problem with how I'm outputting the print function? I saved the print output to a text file from the game's script, as I didn't figure out how to run it from a command line)
Anyway, here's the output:

origin None
submodule_search_locations None
origin None
submodule_search_locations None

[spyoungtech]

Thanks. Glad the workaround is working for you. I will publish version 1.1.3 with this fix shortly.

As for the Ren'Py issues... It seems like Ren'Py's RenpyImporter causes the module spec to be formulated incorrectly. When ahk is installed normally and imported, the spec is present with proper submodule search locations set:

In [2]: import_module('ahk')
spec = importlib.util.find_spec('ahk')

In [3]: spec.origin
Out[3]: 'c:\\users\\spencer\\venv\\lib\\site-packages\\ahk\\__init__.py'

In [4]: spec.submodule_search_locations
Out[4]: ['c:\\users\\spencer\\venv\\lib\\site-packages\\ahk']

In [5]: ahk.__spec__
Out[5]: ModuleSpec(name='ahk', loader=<_frozen_importlib_external.SourceFileLoader object at 0x000001E7E98FAA00>, origin='c:\\users\\spencer\\venv\\lib\\site-packages\\ahk\\__init__.py', submodule_search_locations=['c:\\users\\spencer\\venv\\lib\\site-packages\\ahk'])

As observed when Ren'Py loads the package, the spec is not correct:

>>> ahk.__spec__
ModuleSpec(name='ahk', loader=<__main__.RenpyImporter object at 0x000001B32188E890>)

I'm not familiar enough Ren'Py to know how to fix this issue, but it's probably worth filing an issue with the Ren'Py project.

In any case, I think this change was needed for the intended fallback system to work anyhow, so I'm glad that you brought this up. I'm sure it will help others who are packaging ahk in various ways that break package data loading.

[spyoungtech]

fix released in v1.1.3 🎉

[spyoungtech]

I've filed an issue with Ren'Py about this and will followup as necessary. renpy/renpy#4644