Causes app to fail startup on AWS Lambda

[deceze]

I'm trying to use django-q to enqueue tasks to AWS SQS from an AWS Lambda function. Versions involved:

• Python 3.7
• Django 2.2.5
• django-q 1.0.2
• zappa 0.48.2

This works fine in a local development environment, but putting it on Lambda using Zappa, the app fails with a somewhat cryptic log:

    raise RuntimeError("populate() isn't reentrant") in populatelicationi
[1569572571263] Instancing..
    unlink_now)ng/lib/python3.7/multiprocessing/synchronize.py", line 59, in __init___

(Yes, it's somewhat cut off somewhere.)

So, populate() isn't reentrant is some generic unhelpful error triggered by Django in case some component fails to initialise, and the following cut-off line points somewhere to the multiprocessing module. When removing django_q from the list of installed apps, the error goes away.

It would seem that django-q is trying to import something from multiprocessing that doesn't work on AWS Lambda, which causes the entire thing to fail.

I don't know whether multiprocessing is strictly required for django-q to send tasks to SQS. If possible this should be decoupled somehow. django-q would be ideal for my use case, as it allows the simple use of async_task in code to enqueue tasks, which can be configured to use orm locally during development and SQS in production. FYI, when using SQS the qcluster functionality wouldn't be used, and the SQS queue would be an event source for Lambda workers.

[lordkev]

Just hit this same issue. Were you able to figure out a resolution @deceze?

It seems that Lambda doesn't have /dev/shm and therefore doesn't support multiprocessing.Queue or multiprocessing.Pool.

https://aws.amazon.com/blogs/compute/parallel-processing-in-python-with-aws-lambda/

[deceze]

@lordkev I did not. I switched to dramatiq and dramatiq-sqs instead.

[lordkev]

Ah, thanks for the reply. I just put a PR in with a really simple fix if you ever want to check out django-q again.

…

On Tue, Jan 21, 2020 at 1:24 AM deceze ***@***.***> wrote: @lordkev <https://github.com/lordkev> I did not. I switched to dramatiq and dramatiq-sqs instead. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#384?email_source=notifications&email_token=AAEGDEBHSYDYB42GOSK2U6TQ62PLHA5CNFSM4I3DPEJ2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEJOXZHY#issuecomment-576552095>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAEGDEFEOJ7CY3WVNGMUX73Q62PLHANCNFSM4I3DPEJQ> .