Can't run jobs on mac os catalina: process has forked and you cannot use this CoreFoundation functionality safely

[davisums]

Hey,

I'm having this issue on mac OS catalina, and I think it's django-rq related. Not having the same errors by using rq directly (though I haven't been able to completely set it up because it's so much more complicated than using django-rq)

This setup was working fine on Ubuntu but my queue won't process any jobs on mac os.

07:10:41 default: djpager.jobs.pager.generate_source(331) (f799a0ae-1500-4483-a1ce-818995c93bed)
The process has forked and you cannot use this CoreFoundation functionality safely. You MUST exec().
Break on __THE_PROCESS_HAS_FORKED_AND_YOU_CANNOT_USE_THIS_COREFOUNDATION_FUNCTIONALITY___YOU_MUST_EXEC__() to debug.
objc[26643]: +[NSNumber initialize] may have been in progress in another thread when fork() was called.
objc[26643]: +[NSNumber initialize] may have been in progress in another thread when fork() was called. We cannot safely call it or ignore it in the fork() child process. Crashing instead. Set a breakpoint on objc_initializeAfterForkError to debug.
07:10:41 Moving job to FailedJobRegistry (work-horse terminated unexpectedly; waitpid returned 6)

Googling around, the only suggestion I found was executing this command:

export OBJC_DISABLE_INITIALIZE_FORK_SAFETY=YES

That makes the error message slightly shorter, but still no jobs are run:

07:23:26 default: djpager.jobs.pager.generate_source(362) (33fd1d39-5bc4-49b9-a1ee-c5fb0218fe95)
The process has forked and you cannot use this CoreFoundation functionality safely. You MUST exec().
Break on __THE_PROCESS_HAS_FORKED_AND_YOU_CANNOT_USE_THIS_COREFOUNDATION_FUNCTIONALITY___YOU_MUST_EXEC__() to debug.
07:23:26 Moving job to FailedJobRegistry (work-horse terminated unexpectedly; waitpid returned 11)

Any suggestions on how to get this to work on mac os? Thanks a lot for all the great work on this project ♥️

[thomasmatecki]

Members of my team have encountered this, and I think a reinstall of the project virtual environment did the trick.

Anyone else have a similar experience?

[k0t3n]

Having same issue using decorated function

[thomasmatecki]

@k0t3n https://turtlemonvh.github.io/python-multiprocessing-and-corefoundation-libraries.html

Perhap upgrading your python version will do the trick? (https://bugs.python.org/issue8713)

[k0t3n]

@thomasmatecki the problem reproduces on Python 3.7.6

[thomasmatecki]

Only when using the decorator? Queue#enqueue works fine?

[k0t3n]

No, the problem is not with the @job decorator. I'm talking about using my own decorator. The error occurs both using @job & delay(), and enqueue(). By the way, running worker through RQ works fine (probably the problem is in the django-rq).

[k0t3n]

Starting the worker using sudo solves the problem, but the error message is still present.

[selwin]

Summarizing the discussion on this bug: https://bugs.python.org/issue33725 (correct me if I'm wrong):

1. fork() has always been unsafe on Mac
2. spawn* functions should be instead

This means that the change should be done on RQ instead of this library. However, I don't think this process is something that can be rushed. I've never used spawn functions before and would need to properly investigate the differences.

Can @k0t3n confirm that running manage.py rqworker with sudo can workaround this problem? (i.e jobs are executed properly even though error message is still present)

[k0t3n]

@selwin yes that's right

[asennoussi]

I'm facing the same issue, but I decided to remove the faulty function to the main thread instead:
Contacts.objects.filter(user=request.user).delete()

[zachsiegel-capsida]

I have this same issue with Python 3.9.5 on a Mac, the following Python packages installed:

rq==1.10.1
rq-scheduler==0.11.0
django-redis==5.2.0
redis==4.1.3

and redis installed by Homebrew with brew info redis giving: stable 6.2.6 (bottled), HEAD

For me, running the python manage.py rqscheduler and python manage.py rqworker commands with sudo does not fix the issue. I will note that not ALL my jobs fail. I have three scheduled jobs in my app and only one of them fails; they are all pretty similar.

Note that I do not experience this issue in my Windows/Linux (WSL VM) environment(s), only Mac.

[codeAshu]

Anyone planning anything around it to fix? Maybe in rq?

[asennoussi]

It doesn't happen to me anymore with the new OSX