Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Lua stack and prefetch variables #434

Closed

Conversation

ethervoid
Copy link
Contributor

@ethervoid ethervoid commented Oct 12, 2021

We've found that with a great number of delayed messages, using the max LUA stack in the prefetch for delayed messages and multiple workers you can end up having stack overflow in Redis and crashing the LUA script so this PR intends to minimize that situation:

  • Adding configurable environment variable for the delayed queue so we can define the maximum amount of delayed messages fetched
  • Adding configurable environment variable to define the max LUA stack value
  • Set the maximum unpacking value to 1/2 of the max LUA stack

Related to #433

@ethervoid ethervoid changed the title Variable delayed queue prefetch Lua stack and prefetch variables Oct 12, 2021
@ethervoid ethervoid marked this pull request as ready for review October 12, 2021 14:46
Delayed queue is not tuneable so you could end up fetching the almost
the limit of the LUA stack size and have a server crash in redis because
when redis cannot perform lua_checkstack it ends with a serverPanic as
you can see [h

Signed-off-by: Mario de Frutos <mario@defrutos.org>
We want to be able to define an static value for our LUA stack so we
don't need to rely on dynamic calculations although if we don't pass
that new env. variable we're going to check the max stack value
dynamically

Signed-off-by: Mario de Frutos <mario@defrutos.org>
There are some possible stack overflow problems if we have a high number
of workers and all of the unpack with the maximum limit of the stack.
This change reduces to half that value so we don't incur in the stack
overflow.

See Bogdanp#433

Signed-off-by: Mario de Frutos <mario@defrutos.org>
Signed-off-by: Mario de Frutos <mario@defrutos.org>
@ethervoid ethervoid force-pushed the variable_delayed_queue_prefetch branch from 4a5ccea to 8d3050d Compare October 12, 2021 14:50
@ethervoid
Copy link
Contributor Author

@Bogdanp I've tried to run tox and pass the test but it's failing for me in Apple M1 and also in docker with ubuntu. Is there anything special I have to do?

I've been getting this error in Mac and Ubuntu inside docker:

py38-cpython installed: alabaster==0.7.12,attrs==21.2.0,Babel==2.9.1,backports.entry-points-selectable==1.1.0,bleach==4.1.0,bump2version==1.0.1,certifi==2021.10.8,charset-normalizer==2.0.7,colorama==0.4.4,coverage==6.0.2,distlib==0.3.3,docutils==0.17.1,dramatiq==1.11.0,filelock==3.3.0,flake8==4.0.1,gevent==21.8.0,greenlet==1.1.2,idna==3.2,imagesize==1.2.0,importlib-metadata==4.8.1,iniconfig==1.1.1,Jinja2==3.0.2,keyring==23.2.1,MarkupSafe==2.0.1,mccabe==0.6.1,packaging==21.0,pkginfo==1.7.1,platformdirs==2.4.0,pluggy==1.0.0,pockets==0.9.1,prometheus-client==0.11.0,py==1.10.0,py-cpuinfo==8.0.0,pycodestyle==2.8.0,pyflakes==2.4.0,pygal==2.4.0,pygaljs==1.0.2,Pygments==2.10.0,pyparsing==2.4.7,pytest==6.2.5,pytest-benchmark==3.4.1,pytest-cov==3.0.0,pytz==2021.3,readme-renderer==30.0,redis==3.5.3,requests==2.26.0,requests-toolbelt==0.9.1,rfc3986==1.5.0,six==1.16.0,snowballstemmer==2.1.0,Sphinx==1.7.9,sphinxcontrib-napoleon==0.7,sphinxcontrib-serializinghtml==1.1.5,sphinxcontrib-websupport==1.2.4,toml==0.10.2,tomli==1.2.1,tox==3.24.4,tqdm==4.62.3,twine==3.4.2,urllib3==1.26.7,virtualenv==20.8.1,watchdog==2.1.6,watchdog-gevent==0.1.1,webencodings==0.5.1,zipp==3.6.0,zope.event==4.5.0,zope.interface==5.4.0
py38-cpython run-test-pre: PYTHONHASHSEED='3419722698'
py38-cpython run-test: commands[0] | py.test --benchmark-skip
ImportError while loading conftest '/Users/ethervoid/development/dramatiq/tests/conftest.py'.
tests/conftest.py:7: in <module>
    import pylibmc
E   ModuleNotFoundError: No module named 'pylibmc'
ERROR: InvocationError for command /Users/ethervoid/development/dramatiq/.tox/py38-cpython/bin/py.test --benchmark-skip (exited with code 4)

and looks like tox is not installing the necessary dependencies. Anything I'm missing?

@Bogdanp
Copy link
Owner

Bogdanp commented Oct 17, 2021

Thanks! I've merged the change with rebase.

Re. tox, did you run it with a specific environment (eg. tox -e py38-cpython)? When you do that, it should automatically install everything.

@Bogdanp Bogdanp closed this Oct 17, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

2 participants