Skip to content
This repository has been archived by the owner on Sep 11, 2019. It is now read-only.

Support StrictRedis.eval for Lua scripts #9

Merged
merged 57 commits into from
Feb 13, 2018
Merged
Show file tree
Hide file tree
Changes from 11 commits
Commits
Show all changes
57 commits
Select commit Hold shift + click to select a range
b8d997d
Limited Lua support
blfoster Feb 5, 2018
da8a820
Bump patch version
blfoster Feb 5, 2018
0845484
Remove nightly Python build
blfoster Feb 5, 2018
7ea39cd
Return result from Lua
blfoster Feb 5, 2018
74389b7
Fix table handling
blfoster Feb 6, 2018
58d0763
Fix unit tests
blfoster Feb 6, 2018
77685dc
Fix test
blfoster Feb 6, 2018
0dd661f
Decode non-byte strings
blfoster Feb 6, 2018
507a125
Add eval
blfoster Feb 6, 2018
06e02b5
Restore nightly build
blfoster Feb 6, 2018
0fd83e6
Revert "Restore nightly build"
blfoster Feb 6, 2018
f2086fd
Update readme
blfoster Feb 6, 2018
6e0c1b3
Remove eval from unimplemented functions
blfoster Feb 6, 2018
73cc1d7
Code review fallout
blfoster Feb 7, 2018
87c2686
pep8:
blfoster Feb 7, 2018
e0e7718
Install lupa for CI
blfoster Feb 7, 2018
121de8c
Code review fallout
blfoster Feb 7, 2018
9de8754
pep8:
blfoster Feb 7, 2018
9e6b32f
Install lupa for CI
blfoster Feb 7, 2018
15232fa
Merge branch 'master' of ../fakeredis
blfoster Feb 7, 2018
9bf4173
More fallout
blfoster Feb 8, 2018
bfa38c9
Remove lua from .travis.yml
blfoster Feb 8, 2018
63949ba
More tests
blfoster Feb 8, 2018
975f88c
pcall
blfoster Feb 8, 2018
71a2c1e
Test nested ok response
blfoster Feb 8, 2018
1def833
pep8
blfoster Feb 8, 2018
dd23fb1
Fix broken tst
blfoster Feb 8, 2018
c91f0b1
More fallout
blfoster Feb 8, 2018
ee53358
Remove lua from .travis.yml
blfoster Feb 8, 2018
a30fdef
More tests
blfoster Feb 8, 2018
b426f0f
pcall
blfoster Feb 8, 2018
6153787
Test nested ok response
blfoster Feb 8, 2018
7fb69f4
pep8
blfoster Feb 8, 2018
49cf337
Merge branch 'master' of ../fakeredis
blfoster Feb 8, 2018
392206c
Remove extra braces
blfoster Feb 8, 2018
d5e4ce2
Merge branch 'master' of ../fakeredis
blfoster Feb 8, 2018
fe9065d
Fix type
blfoster Feb 8, 2018
296c2ff
Fix decoding
blfoster Feb 8, 2018
95762e2
Merge branch 'master' of ../fakeredis
blfoster Feb 8, 2018
fde2b75
Mess with string types some more
blfoster Feb 8, 2018
773f0f4
Merge branch 'master' of ../fakeredis
blfoster Feb 8, 2018
b1a0402
Fix test
blfoster Feb 8, 2018
856a031
Merge branch 'master' of ../fakeredis
blfoster Feb 8, 2018
acc44b1
Test for nil in table
blfoster Feb 8, 2018
23a43af
More tests
blfoster Feb 9, 2018
525a29b
Fix test
blfoster Feb 9, 2018
267e338
Remove unnecessary exception handler
blfoster Feb 9, 2018
6377669
WIP: fallout
blfoster Feb 12, 2018
244056b
Remove docstring
blfoster Feb 12, 2018
8f869b4
More fallout
blfoster Feb 12, 2018
b2b6786
Catch LuaError
blfoster Feb 12, 2018
fa252bc
Check number of keys
blfoster Feb 12, 2018
f917498
More fallout
blfoster Feb 12, 2018
82e7e65
More edge cases for numkeys
blfoster Feb 12, 2018
fdf9ecb
Remove braces
blfoster Feb 12, 2018
a6a7cd6
flake8
blfoster Feb 12, 2018
9f4d910
to_bytes
blfoster Feb 12, 2018
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 5 additions & 1 deletion README.rst
Original file line number Diff line number Diff line change
Expand Up @@ -203,7 +203,6 @@ scripting
* script kill
* script load
* evalsha
* eval
* script exists


Expand Down Expand Up @@ -266,6 +265,11 @@ they have all been tagged as 'slow' so you can skip them by running::
Revision history
================

0.9.5
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can you change this to "Development version". There is some possibility of re-merging with fakeredis, so I don't want to make assumptions about version numbers yet.

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done.

-----
Add support for StrictRedis.eval for Lua scripts
- `#9 <https://github.com/ska-sa/fakenewsredis/pull/9>`_
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can you put the description into the bullet point (to make the format of the other changelog entries).

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done.


0.9.4
-----
This is a minor bugfix and optimization release:
Expand Down
52 changes: 52 additions & 0 deletions fakenewsredis.py
Original file line number Diff line number Diff line change
Expand Up @@ -14,6 +14,9 @@
import types
import re
import functools
from itertools import count

from lupa import LuaRuntime, lua_type

import redis
from redis.exceptions import ResponseError
Expand Down Expand Up @@ -701,6 +704,55 @@ def sort(self, name, start=None, num=None, by=None, get=None, desc=False,
except KeyError:
return []

def eval(self, script, numkeys, *keys_and_args):
"""
Execute the Lua ``script``, specifying the ``numkeys`` the script
will touch and the key names and argument values in ``keys_and_args``.
Returns the result of the script.
In practice, use the object returned by ``register_script``. This
function exists purely for Redis API completion.
"""
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You can delete the docstring. It's assumed that people will refer to redis-py for docs.

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done.

lua_runtime = LuaRuntime(unpack_returned_tuples=True)

raw_lua = """
function(KEYS, ARGV, callback)
redis = {{}}
redis.call = callback
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Also need redis.pcall, redis.error_reply, redis.status_reply?

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The latter two are done.

{body}
end
""".format(body=script)
keys = (None,) + keys_and_args[:numkeys]
args = (None,) + keys_and_args[numkeys:]

lua_func = lua_runtime.eval(raw_lua)
result = lua_func(
keys,
args,
self._lua_callback
)
if lua_type(result) == 'table':
# Convert Lua tables into lists, starting from index 1, mimicking the behavior of StrictRedis.
result_list = []
for index in count(1):
if index not in result:
break
item = result[index]
result_list.append(
item.encode() if isinstance(item, str) and not isinstance(item, bytes) else item
)
return result_list
return result

def _lua_callback(self, op, *args):
special_cases = {
'del': self.delete,
'decrby': self.decr,
'incrby': self.incr
}
op = op.lower()
func = special_cases[op] if op in special_cases else getattr(self, op)
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There are presumably attributes of self that shouldn't be visible to Lua - for a start, anything with an underscore prefix, plus things like delete and from_url. Also, when self is a FakeRedis rather than a FakeStrictRedis, it presumably should use the overload from the FakeStrictRedis base class.

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Probably yes. This is a bit of a hack, but I figure the tradeoff is that it doesn't have a huge mapping from Redis commands to Python functions that needs to be maintained as more Redis functions get added. I'll make it a bit more selective.

return func(*args)

def _retrive_data_from_sort(self, data, get):
if get is not None:
if isinstance(get, string_types):
Expand Down
1 change: 1 addition & 0 deletions requirements.txt
Original file line number Diff line number Diff line change
Expand Up @@ -2,3 +2,4 @@
flake8<3.0.0
nose==1.3.4
redis==2.10.6
lupa==1.6
2 changes: 1 addition & 1 deletion setup.py
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,7 @@

setup(
name='fakenewsredis',
version='0.9.4',
version='0.9.5',
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Don't update the version yet; I'll do it when I make a new release.

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done.

description="Fake implementation of redis API for testing purposes.",
long_description=open(os.path.join(os.path.dirname(__file__),
'README.rst')).read(),
Expand Down
53 changes: 53 additions & 0 deletions test_fakenewsredis.py
Original file line number Diff line number Diff line change
Expand Up @@ -3046,6 +3046,59 @@ def test_set_existing_key_persists(self):
self.redis.set('foo', 'foo')
self.assertEqual(self.redis.ttl('foo'), -1)

def test_eval_delete(self):
self.redis.set('foo', 'bar')
val = self.redis.get('foo')
self.assertEqual(val, b'bar')
val = self.redis.eval('redis.call("DEL", KEYS[1])', 1, 'foo')
self.assertIsNone(val)

def test_eval_set_value_to_arg(self):
self.redis.eval('redis.call("SET", KEYS[1], ARGV[1])', 1, 'foo', 'bar')
val = self.redis.get('foo')
self.assertEqual(val, b'bar')

def test_eval_conditional(self):
lua = """
local val = redis.call("GET", KEYS[1])
if val == ARGV[1] then
redis.call("SET", KEYS[1], ARGV[2])
else
redis.call("SET", KEYS[1], ARGV[1])
end
"""
self.redis.eval(lua, 1, 'foo', 'bar', 'baz')
val = self.redis.get('foo')
self.assertEqual(val, b'bar')
self.redis.eval(lua, 1, 'foo', 'bar', 'baz')
val = self.redis.get('foo')
self.assertEqual(val, b'baz')

def test_eval_lrange(self):
self.redis.lpush("foo", "bar")
val = self.redis.eval('return redis.call("LRANGE", KEYS[1], 0, 1)', 1, 'foo')
self.assertEqual(val, [b'bar'])

def test_eval_table(self):
lua = """
local a = {}
a[1] = "foo"
a[2] = "bar"
a[17] = "baz"
return a
"""
val = self.redis.eval(lua, 0)
self.assertEqual(val, [b'foo', b'bar'])

def test_eval_table_with_numbers(self):
lua = """
local a = {}
a[1] = 42
return a
"""
val = self.redis.eval(lua, 0)
self.assertEqual(val, [42])


class TestFakeRedis(unittest.TestCase):
decode_responses = False
Expand Down