New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

celery4 worker fails to start on Windows #3383

Closed
AnkurDedania opened this Issue Aug 10, 2016 · 5 comments

Comments

Projects
None yet
6 participants
@AnkurDedania
Copy link
Contributor

AnkurDedania commented Aug 10, 2016

trying to execute https://github.com/celery/celery/blob/master/examples/app/myapp.py

λ python myapp.py worker

 -------------- celery@C001683697 v4.0.0rc3 (0today8)
---- **** -----
--- * ***  * -- Windows-7-6.1.7601-SP1 2016-08-10 16:12:28
-- * - **** ---
- ** ---------- [config]
- ** ---------- .> app:         myapp:0x26c8c88
- ** ---------- .> transport:   amqp://guest:**@localhost:5672//
- ** ---------- .> results:     disabled://
- *** --- * --- .> concurrency: 12 (prefork)
-- ******* ----
--- ***** ----- [queues]
 -------------- .> celery           exchange=celery(direct) key=celery


[2016-08-10 16:12:28,153: CRITICAL/MainProcess] Unrecoverable error: AttributeError("Can't pickle local object 'Pool.__init__.<locals>.Process'",)

Traceback (most recent call last):
  File "C:\Miniconda3\lib\site-packages\celery\worker\__init__.py", line 201, in start
    self.blueprint.start(self)
  File "C:\Miniconda3\lib\site-packages\celery\bootsteps.py", line 125, in start
    step.start(parent)
  File "C:\Miniconda3\lib\site-packages\celery\bootsteps.py", line 376, in start
    return self.obj.start()
  File "C:\Miniconda3\lib\site-packages\celery\concurrency\base.py", line 134, in start
    self.on_start()
  File "C:\Miniconda3\lib\site-packages\celery\concurrency\prefork.py", line 117, in on_start
    **self.options)
  File "C:\Miniconda3\lib\site-packages\billiard\pool.py", line 1015, in __init__
    self._create_worker_process(i)
  File "C:\Miniconda3\lib\site-packages\billiard\pool.py", line 1124, in _create_worker_process
    w.start()
  File "C:\Miniconda3\lib\site-packages\billiard\process.py", line 122, in start
    self._popen = self._Popen(self)
  File "C:\Miniconda3\lib\site-packages\billiard\context.py", line 383, in _Popen
    return Popen(process_obj)
  File "C:\Miniconda3\lib\site-packages\billiard\popen_spawn_win32.py", line 72, in __init__
    reduction.dump(process_obj, to_child)
  File "C:\Miniconda3\lib\site-packages\billiard\reduction.py", line 97, in dump
    ForkingPickler(file, protocol).dump(obj)
AttributeError: Can't pickle local object 'Pool.__init__.<locals>.Process'

Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "C:\Miniconda3\lib\site-packages\billiard\spawn.py", line 159, in spawn_main
    new_handle = steal_handle(parent_pid, pipe_handle)
  File "C:\Miniconda3\lib\site-packages\billiard\reduction.py", line 119, in steal_handle
    _winapi.PROCESS_DUP_HANDLE, False, source_pid)
OSError: [WinError 87] The parameter is incorrect
@berkerpeksag

This comment has been minimized.

Copy link
Collaborator

berkerpeksag commented Aug 14, 2016

Thanks for the report. According to http://docs.celeryproject.org/en/master/whatsnew-4.0.html#removed-features, Windows is no longer supported by Celery 4.0.

@zhangqunshi

This comment has been minimized.

Copy link

zhangqunshi commented Jun 9, 2017

@berkerpeksag why don't support the windows?

@katrinleinweber

This comment has been minimized.

Copy link

katrinleinweber commented Jul 23, 2018

@auvipy

This comment has been minimized.

Copy link
Member

auvipy commented Jul 30, 2018

now support added. try 4.2.x

@sparrowt

This comment has been minimized.

Copy link

sparrowt commented Jan 15, 2019

Although the pickling issue is fixed (presumably by celery/billiard#225) I'm still seeing the 2nd error shown above:

WindowsError: [Error 87] The parameter is incorrect

on celery 4.2.1 when running celery worker with --pool prefork argument.

I know Windows is not technically supported but @auvipy your comment suggests that perhaps Windows support has returned in some form, is that correct?

In my case it seems the reason _winapi.OpenProcess(_winapi.PROCESS_DUP_HANDLE, False, source_pid) is failing is because, despite source_pid being set to a valid non-zero int which appears to be the parent process ID, the parent celery worker process has quit (shortly after starting) with this error:

  ...
  File "...\python2\lib\site-packages\billiard\popen_spawn_win32.py", line 64, in __init__
    _winapi.CloseHandle(ht)
TypeError: argument 1 must be an integer, not _subprocess_handle

Seems I'm not alone: celery/billiard#239

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment