Add better support for LUA scripts

[JCHacking]

Is your feature request related to a problem? Please describe.
I am trying to mock the redi backend for asgi-ratelimit and I am encountering the following error:

Traceback (most recent call last):
  File "C:\Users\jcmencia\PycharmProjects\NGCSQALIBS\py_api\venv\Lib\site-packages\redis\commands\core.py", line 5172, in __call__
    return await client.evalsha(self.sha, len(keys), *args)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\jcmencia\PycharmProjects\NGCSQALIBS\py_api\venv\Lib\site-packages\redis\asyncio\client.py", line 610, in execute_command
    return await conn.retry.call_with_retry(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\jcmencia\PycharmProjects\NGCSQALIBS\py_api\venv\Lib\site-packages\redis\asyncio\retry.py", line 59, in call_with_retry
    return await do()
           ^^^^^^^^^^
  File "C:\Users\jcmencia\PycharmProjects\NGCSQALIBS\py_api\venv\Lib\site-packages\redis\asyncio\client.py", line 584, in _send_command_parse_response
    return await self.parse_response(conn, command_name, **options)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\jcmencia\PycharmProjects\NGCSQALIBS\py_api\venv\Lib\site-packages\redis\asyncio\client.py", line 631, in parse_response
    response = await connection.read_response()
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\jcmencia\PycharmProjects\NGCSQALIBS\py_api\venv\Lib\site-packages\fakeredis\aioredis.py", line 171, in read_response
    raise response
redis.exceptions.NoScriptError: No matching script. Please use EVAL.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "C:\Users\jcmencia\PycharmProjects\NGCSQALIBS\py_api\venv\Lib\site-packages\starlette\middleware\errors.py", line 164, in __call__
    await self.app(scope, receive, _send)
  File "C:\Users\jcmencia\PycharmProjects\NGCSQALIBS\py_api\venv\Lib\site-packages\ratelimit\core.py", line 94, in __call__
    retry_after = await self.backend.retry_after(path, user, rule)
                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\jcmencia\PycharmProjects\NGCSQALIBS\py_api\venv\Lib\site-packages\ratelimit\backends\redis.py", line 51, in retry_after
    await self.lua_script(keys=list(ruleset.keys()), args=[json.dumps(ruleset)])
  File "C:\Users\jcmencia\PycharmProjects\NGCSQALIBS\py_api\venv\Lib\site-packages\redis\commands\core.py", line 5178, in __call__
    return await client.evalsha(self.sha, len(keys), *args)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\jcmencia\PycharmProjects\NGCSQALIBS\py_api\venv\Lib\site-packages\redis\asyncio\client.py", line 610, in execute_command
    return await conn.retry.call_with_retry(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\jcmencia\PycharmProjects\NGCSQALIBS\py_api\venv\Lib\site-packages\redis\asyncio\retry.py", line 59, in call_with_retry
    return await do()
           ^^^^^^^^^^
  File "C:\Users\jcmencia\PycharmProjects\NGCSQALIBS\py_api\venv\Lib\site-packages\redis\asyncio\client.py", line 584, in _send_command_parse_response
    return await self.parse_response(conn, command_name, **options)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\jcmencia\PycharmProjects\NGCSQALIBS\py_api\venv\Lib\site-packages\redis\asyncio\client.py", line 631, in parse_response
    response = await connection.read_response()
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\jcmencia\PycharmProjects\NGCSQALIBS\py_api\venv\Lib\site-packages\fakeredis\aioredis.py", line 171, in read_response
    raise response
redis.exceptions.ResponseError: Error running script (call to f_283a893bdb53a688600eff30001e6fb4e4ebd4f5): @user_script:?: [string "<python>"]:2: attempt to index a nil value (global 'cjson')
stack traceback:
	[string "<python>"]:2: in main chunk

Describe the solution you'd like
Add support for cjson? I don't know if this is only the problem.
I don't know exactly if this would be for lupa or for fakeredis, if you can give me some guidance I would appreciate it.

Related issues:

• Support LUA scripting jamesls/fakeredis#25
• Lua environmental differences jamesls/fakeredis#183

Additional context
I have tried this code:

from fakeredis.aioredis import FakeRedis
from fastapi import FastAPI
from ratelimit import RateLimitMiddleware, Rule
from ratelimit.backends.redis import RedisBackend
from ratelimit.types import Scope

app = FastAPI(debug=True)


async def authenticate(scope: Scope) -> tuple[str, str]:
    return str(scope["client"]), "default"


app.add_middleware(
    RateLimitMiddleware,
    authenticate=authenticate,
    backend=RedisBackend(
        FakeRedis.from_url("redis://redis")
    ),
    config={
        "": [Rule(minute=1)]
    }
)


@app.get("/")
async def test():
    return "A"

Upvote & Fund

• We're using Polar.sh so you can upvote and help fund this issue.
• We receive the funding once the issue is completed & confirmed by you.
• Thank you in advance for helping prioritize & fund our backlog.

[cunla]

Did you install fakeredis with lua support?
https://fakeredis.readthedocs.io/en/latest/#installation

[JCHacking]

Did you install fakeredis with lua support?
https://fakeredis.readthedocs.io/en/latest/#installation

Yes

[cunla]

I checked, to load cjson, it should be compiled for the machine where fakeredis is running.

I might add an environment variable LUA_MODULES stating which modules should be loaded.

[JCHacking]

So from what you say, to avoid this error I have to compile it on the machine that is going to run it with the environment variable LUA_MODULES=cjson right?

Or could this be a bug in the lua script?
It looks like this is the lua script running underneath. I understand that the failure is that it does not find the cjson module.

local ruleset = cjson.decode(ARGV[1])

-- Set limits
for i, key in pairs(KEYS) do
    redis.call('SET', key, ruleset[key][1], 'EX', ruleset[key][2], 'NX')
end

-- Check limits
for i = 1, #KEYS do
    local value = redis.call('GET', KEYS[i])
    if value and tonumber(value) < 1 then
        return ruleset[KEYS[i]][2]
    end
end

-- Decrease limits
for i, key in pairs(KEYS) do
    redis.call('DECR', key)
end
return 0

[cunla]

Well, I would need to implement this, but yeah, eventually, that would be the way it works.

[JCHacking]

I use poetry to manage dependencies, so when I get a whl from pypi I will download that one. I understand that in that case I will not use anything from LUA_MODULES because I will download it already compiled.

[cunla]

does cjson have a pypi package?

[JCHacking]

It seems that there are packages like ujson that give this functionality (I have not tested them).

[cunla]

We need something where this would run:

import lupa

lupa.LuaRuntime().execute("require('cjson')")

I don't think ujson meets this requirement

[JCHacking]

Trying with this code I get this error:

lupa.lua54.LuaError: [string "<python>"]:1: module 'cjson' not found:
	no field package.preload['cjson']
	no file 'C:\Users\jcmencia\AppData\Local\Programs\Python\Python312\lua\cjson.lua'
	no file 'C:\Users\jcmencia\AppData\Local\Programs\Python\Python312\lua\cjson\init.lua'
	no file 'C:\Users\jcmencia\AppData\Local\Programs\Python\Python312\cjson.lua'
	no file 'C:\Users\jcmencia\AppData\Local\Programs\Python\Python312\cjson\init.lua'
	no file 'C:\Users\jcmencia\AppData\Local\Programs\Python\Python312\..\share\lua\5.4\cjson.lua'
	no file 'C:\Users\jcmencia\AppData\Local\Programs\Python\Python312\..\share\lua\5.4\cjson\init.lua'
	no file '.\cjson.lua'
	no file '.\cjson\init.lua'
	no file 'C:\Users\jcmencia\AppData\Local\Programs\Python\Python312\cjson.dll'
	no file 'C:\Users\jcmencia\AppData\Local\Programs\Python\Python312\..\lib\lua\5.4\cjson.dll'
	no file 'C:\Users\jcmencia\AppData\Local\Programs\Python\Python312\loadall.dll'
	no file '.\cjson.dll'
stack traceback:
	[string "<python>"]:1: in main chunk
	[C]: in function 'require'

A little closer to making it work we are!

[JCHacking]

This is probably something that should be implemented in the lupa package.

[cunla]

lupa supports importing modules, it simply does not have cjson ootb.

I managed to install cjson:

• First, install luarocks (I am using mac so brew install luarocks, but there is an installation for windows).
• Then, install cjson using luarocks install lua-cjson
• Then, copy cjson.so (or in your case I assume cjson.dll) to the project homedir.
• Then, the code I wrote should work.

[JCHacking]

I have been researching and there is a "cleaner" way to do this:

import lupa

lupa.LuaRuntime().require("cjson")

[cunla]

Does it work without installing cjson first?

[JCHacking]

Does it work without installing cjson first?

No, as you said it seems that lupa does not bring the cjson module with it.

This makes more "complex", in my case, mock redis for asgi-ratelimit since I want for my CI to use python base only, and it doesn't seem right to upload cjson.dll or cjson.so to the repository either.

[cunla]

yeah, I am not sure why asgi-ratelimit chose to implement a ratelimit using a script that uses cjson - it seems a bit cumbersome.
But regardless, fakeredis now supports having LUA modules