Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Some operations (e.g. `applymap`, `groupby`) are not working with Dask #977

Closed
devin-petersohn opened this issue Jan 9, 2020 · 2 comments
Closed
Milestone

Comments

@devin-petersohn
Copy link
Member

@devin-petersohn devin-petersohn commented Jan 9, 2020

Describe the problem

Dask has a serialization issue with some of the way we submit the functions. It appears as if the serializer is attempting to pickle the whole package and send that to the workers. It fails because we start a LocalCluster if there isn't one already, and the workers are not authorized to start their own cluster. See logs below.

Source code / logs

UserWarning: 
Port 8787 is already in use. 
Perhaps you already have a cluster running?
Hosting the diagnostics dashboard on a random port instead.
Task exception was never retrieved
future: <Task finished coro=<_wrap_awaitable() done, defined at /Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/asyncio/tasks.py:530> exception=AssertionError('daemonic processes are not allowed to have children',)>
Traceback (most recent call last):
  File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/asyncio/tasks.py", line 537, in _wrap_awaitable
    return (yield from awaitable.__await__())
  File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/distributed/nanny.py", line 251, in start
    response = await self.instantiate()
  File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/distributed/nanny.py", line 334, in instantiate
    result = await self.process.start()
  File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/distributed/nanny.py", line 522, in start
    await self.process.start()
  File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/distributed/process.py", line 34, in _call_and_set_future
    res = func(*args, **kwargs)
  File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/distributed/process.py", line 202, in _start
    process.start()
  File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/multiprocessing/process.py", line 103, in start
    'daemonic processes are not allowed to have children'
AssertionError: daemonic processes are not allowed to have children
Task exception was never retrieved
future: <Task finished coro=<_wrap_awaitable() done, defined at /Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/asyncio/tasks.py:530> exception=AssertionError('daemonic processes are not allowed to have children',)>
Traceback (most recent call last):
  File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/asyncio/tasks.py", line 537, in _wrap_awaitable
    return (yield from awaitable.__await__())
  File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/distributed/nanny.py", line 251, in start
    response = await self.instantiate()
  File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/distributed/nanny.py", line 334, in instantiate
    result = await self.process.start()
  File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/distributed/nanny.py", line 522, in start
    await self.process.start()
  File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/distributed/process.py", line 34, in _call_and_set_future
    res = func(*args, **kwargs)
  File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/distributed/process.py", line 202, in _start
    process.start()
  File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/multiprocessing/process.py", line 103, in start
    'daemonic processes are not allowed to have children'
AssertionError: daemonic processes are not allowed to have children
Task exception was never retrieved
future: <Task finished coro=<_wrap_awaitable() done, defined at /Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/asyncio/tasks.py:530> exception=AssertionError('daemonic processes are not allowed to have children',)>
Traceback (most recent call last):
  File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/asyncio/tasks.py", line 537, in _wrap_awaitable
    return (yield from awaitable.__await__())
  File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/distributed/nanny.py", line 251, in start
    response = await self.instantiate()
  File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/distributed/nanny.py", line 334, in instantiate
    result = await self.process.start()
  File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/distributed/nanny.py", line 522, in start
    await self.process.start()
  File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/distributed/process.py", line 34, in _call_and_set_future
    res = func(*args, **kwargs)
  File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/distributed/process.py", line 202, in _start
    process.start()
  File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/multiprocessing/process.py", line 103, in start
    'daemonic processes are not allowed to have children'
AssertionError: daemonic processes are not allowed to have children
Task exception was never retrieved
future: <Task finished coro=<_wrap_awaitable() done, defined at /Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/asyncio/tasks.py:530> exception=AssertionError('daemonic processes are not allowed to have children',)>
Traceback (most recent call last):
  File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/asyncio/tasks.py", line 537, in _wrap_awaitable
    return (yield from awaitable.__await__())
  File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/distributed/nanny.py", line 251, in start
    response = await self.instantiate()
  File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/distributed/nanny.py", line 334, in instantiate
    result = await self.process.start()
  File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/distributed/nanny.py", line 522, in start
    await self.process.start()
  File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/distributed/process.py", line 34, in _call_and_set_future
    res = func(*args, **kwargs)
  File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/distributed/process.py", line 202, in _start
    process.start()
  File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/multiprocessing/process.py", line 103, in start
    'daemonic processes are not allowed to have children'
AssertionError: daemonic processes are not allowed to have children
Task exception was never retrieved
future: <Task finished coro=<_wrap_awaitable() done, defined at /Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/asyncio/tasks.py:530> exception=AssertionError('daemonic processes are not allowed to have children',)>
Traceback (most recent call last):
  File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/asyncio/tasks.py", line 537, in _wrap_awaitable
    return (yield from awaitable.__await__())
  File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/distributed/nanny.py", line 251, in start
    response = await self.instantiate()
  File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/distributed/nanny.py", line 334, in instantiate
    result = await self.process.start()
  File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/distributed/nanny.py", line 522, in start
    await self.process.start()
  File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/distributed/process.py", line 34, in _call_and_set_future
    res = func(*args, **kwargs)
  File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/distributed/process.py", line 202, in _start
    process.start()
  File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/multiprocessing/process.py", line 103, in start
    'daemonic processes are not allowed to have children'
AssertionError: daemonic processes are not allowed to have children
Task exception was never retrieved
future: <Task finished coro=<_wrap_awaitable() done, defined at /Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/asyncio/tasks.py:530> exception=AssertionError('daemonic processes are not allowed to have children',)>
Traceback (most recent call last):
  File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/asyncio/tasks.py", line 537, in _wrap_awaitable
    return (yield from awaitable.__await__())
  File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/distributed/nanny.py", line 251, in start
    response = await self.instantiate()
  File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/distributed/nanny.py", line 334, in instantiate
    result = await self.process.start()
  File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/distributed/nanny.py", line 522, in start
    await self.process.start()
  File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/distributed/process.py", line 34, in _call_and_set_future
    res = func(*args, **kwargs)
  File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/distributed/process.py", line 202, in _start
    process.start()
  File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/multiprocessing/process.py", line 103, in start
    'daemonic processes are not allowed to have children'
AssertionError: daemonic processes are not allowed to have children
Task exception was never retrieved
future: <Task finished coro=<_wrap_awaitable() done, defined at /Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/asyncio/tasks.py:530> exception=AssertionError('daemonic processes are not allowed to have children',)>
Traceback (most recent call last):
  File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/asyncio/tasks.py", line 537, in _wrap_awaitable
    return (yield from awaitable.__await__())
  File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/distributed/nanny.py", line 251, in start
    response = await self.instantiate()
  File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/distributed/nanny.py", line 334, in instantiate
    result = await self.process.start()
  File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/distributed/nanny.py", line 522, in start
    await self.process.start()
  File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/distributed/process.py", line 34, in _call_and_set_future
    res = func(*args, **kwargs)
  File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/distributed/process.py", line 202, in _start
    process.start()
  File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/multiprocessing/process.py", line 103, in start
    'daemonic processes are not allowed to have children'
AssertionError: daemonic processes are not allowed to have children
Task exception was never retrieved
future: <Task finished coro=<_wrap_awaitable() done, defined at /Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/asyncio/tasks.py:530> exception=AssertionError('daemonic processes are not allowed to have children',)>
Traceback (most recent call last):
  File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/asyncio/tasks.py", line 537, in _wrap_awaitable
    return (yield from awaitable.__await__())
  File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/distributed/nanny.py", line 251, in start
    response = await self.instantiate()
  File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/distributed/nanny.py", line 334, in instantiate
    result = await self.process.start()
  File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/distributed/nanny.py", line 522, in start
    await self.process.start()
  File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/distributed/process.py", line 34, in _call_and_set_future
    res = func(*args, **kwargs)
  File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/distributed/process.py", line 202, in _start
    process.start()
  File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/multiprocessing/process.py", line 103, in start
    'daemonic processes are not allowed to have children'
AssertionError: daemonic processes are not allowed to have children
@modin-bot

This comment has been minimized.

Copy link

@modin-bot modin-bot commented Jan 10, 2020

This issue has been mentioned on Modin Discuss. There might be relevant details there:

https://discuss.modin.org/t/modin-engine-dask-or-ray/99/1

@devin-petersohn

This comment has been minimized.

Copy link
Member Author

@devin-petersohn devin-petersohn commented Jan 15, 2020

This looks isolated to MacOSX 10.12, which is no longer supported. I'm going to close as wontfix because the OS is too old, but please reopen if the issue occurs again.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

None yet
2 participants
You can’t perform that action at this time.