locking depends on redis

[paweljasinski]

This is more of the documentation issue. It is not clear that redis is a hard dependency.
As a workaround, I switched my celery to single worker and commented out active part of locking.py

[codingjoe]

Hi @paweljasinski,

yes, Redis is a hard dependency. I salute your workaround, but I am afraid, I wouldn't really be an option in a production environment.

For now, I believe Redis offers the best solution to implement a cross machine lock. Especially since Redis is very common in Django for caching or as a result backend for both Celery and Dramatiq.

Therefore, I'd suggest updating the documentation unless you maybe have another solution in mind?

Best,
Joe

[paweljasinski]

I can imagine using "select … for update no wait" as alternative implementation for non blocking lock.
It works as long as there is a row to lock on. As far as I can tell, the locking granularity is process, so there is already a perfect candidate:

select id from joeflow_process where id=process_pk for update no wait;

Quick check, and oracle, MySQL, postgres and mssql have it. No luck for SQLite.
There is also Django native support for it. https://docs.djangoproject.com/en/3.0/ref/models/querysets/#select-for-update
In my use case, it is difficult to justify use of redis just for the locking. At the same time, oracle is a hard requirement.
Are you ok with PR with configurable alternative locking based on "select for update"?

[codingjoe]

Hi @paweljasinski interesting idea. That does seem to be the better solution, though I haven't fully thought it through yet.
Yes, we lock on process level, so tasks never can truly be executed in parallel to avoid race conditions. I presume an update lock, might be ever better suited

That being said, I guess where are only getting anywhere if we try. Feel free to open a PR the replaces the current lock. We only need one that works and the DB is always there.

I am really excited, this is a great idea!
Best, Joe

[paweljasinski]

Does override acquire lock?

[codingjoe]

It doesn't I guess, yet it should. Feel free to add it.