Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

multiporcessing in task throwd daemonic processes are not allowed to have children #13298

Closed
mtaohuang opened this issue Dec 24, 2020 · 3 comments
Labels
affected_version:2.0 Issues Reported for 2.0 kind:bug This is a clearly a bug

Comments

@mtaohuang
Copy link

Apache Airflow version:
2.0

Environment:
Python3.6

What happened:
I used mutiprocessing Process in PythonOperator callable function, but it reports an error.

File "/home/xxx/airflow/dags/Gen/SecData.py", line 88, in gen
w.start()
File "/usr/lib64/python3.6/multiprocessing/process.py", line 103, in start
'daemonic processes are not allowed to have children'
AssertionError: daemonic processes are not allowed to have children

What you expected to happen:
In airflow1.10, I could use multiprocessing in task function, it works well,
I'm not sure if is't a bug or there are other ways to use multiporcessing in PythonOperator task function.

@mtaohuang mtaohuang added the kind:bug This is a clearly a bug label Dec 24, 2020
@boring-cyborg
Copy link

boring-cyborg bot commented Dec 24, 2020

Thanks for opening your first issue here! Be sure to follow the issue template!

@mtaohuang
Copy link
Author

export PYTHONOPTIMIZE=1
solved the problem

@vikramkoka vikramkoka added the affected_version:2.0 Issues Reported for 2.0 label Feb 9, 2021
@raphaelauv
Copy link
Contributor

explanation there -> https://stackoverflow.com/a/57493983

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
affected_version:2.0 Issues Reported for 2.0 kind:bug This is a clearly a bug
Projects
None yet
Development

No branches or pull requests

3 participants