Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Airflow standalone command does not exit gracefully #19260

Closed
2 tasks done
dstandish opened this issue Oct 27, 2021 · 11 comments · Fixed by #23274
Closed
2 tasks done

Airflow standalone command does not exit gracefully #19260

dstandish opened this issue Oct 27, 2021 · 11 comments · Fixed by #23274
Labels
area:core kind:bug This is a clearly a bug

Comments

@dstandish
Copy link
Contributor

Apache Airflow version

2.2.0 (latest released)

Operating System

macOS

Versions of Apache Airflow Providers

No response

Deployment

Virtualenv installation

Deployment details

No response

What happened

run airflow standalone

enter ctrl+c

hangs here:

webserver | 127.0.0.1 - - [27/Oct/2021:09:30:57 -0700] "GET /static/pin_32.png HTTP/1.1" 304 0 "http://localhost:8080/home" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/95.0.4638.54 Safari/537.36"
^Cstandalone | Shutting down components

What you expected to happen

No response

How to reproduce

No response

Anything else

No response

Are you willing to submit PR?

  • Yes I am willing to submit a PR!

Code of Conduct
@dstandish dstandish added kind:bug This is a clearly a bug area:core labels Oct 27, 2021
@dstandish
Copy link
Contributor Author

OK looks like it may have something to do with problems shutting down the scheduler.

I had no tasks running but still it hangs here:

[2021-10-27 09:41:21,617] {local_executor.py:388} INFO - Shutting down LocalExecutor; waiting for running tasks to finish.  Signal again if you don't want to wait.

Logs attached.
kill-logs-scheduler.txt

@potiuk potiuk mentioned this issue Apr 26, 2022
Merged
potiuk added a commit to potiuk/airflow that referenced this issue Apr 26, 2022
@potiuk
Remove custom signal handling in Triggerer
3a73320 
There is a bug in CPython (fixed in March 2022 but not yet released) that
makes async.io handle SIGTERM improperly by using async unsafe
functions and hanging the triggerer receive SIGPIPE while handling
SIGTERN/SIGINT and deadlocking itself. Until the bug is handled
we should rather rely on standard handling of the signals rather than
adding our own signal handlers. Seems that even if our signal handler
just run exit(0) - it caused a race condition that led to the hanging.

More details:
   * https://bugs.python.org/issue39622
   * python/cpython#83803

Fixes: apache#19260
potiuk added a commit that referenced this issue Apr 27, 2022
@potiuk
Remove custom signal handling in Triggerer (#23274)
6bdbed6 
There is a bug in CPython (fixed in March 2022 but not yet released) that
makes async.io handle SIGTERM improperly by using async unsafe
functions and hanging the triggerer receive SIGPIPE while handling
SIGTERN/SIGINT and deadlocking itself. Until the bug is handled
we should rather rely on standard handling of the signals rather than
adding our own signal handlers. Seems that even if our signal handler
just run exit(0) - it caused a race condition that led to the hanging.

More details:
   * https://bugs.python.org/issue39622
   * python/cpython#83803

Fixes: #19260
ephraimbuddy pushed a commit that referenced this issue May 8, 2022
@potiuk @ephraimbuddy
Remove custom signal handling in Triggerer (#23274)
9ee62df 
There is a bug in CPython (fixed in March 2022 but not yet released) that
makes async.io handle SIGTERM improperly by using async unsafe
functions and hanging the triggerer receive SIGPIPE while handling
SIGTERN/SIGINT and deadlocking itself. Until the bug is handled
we should rather rely on standard handling of the signals rather than
adding our own signal handlers. Seems that even if our signal handler
just run exit(0) - it caused a race condition that led to the hanging.

More details:
   * https://bugs.python.org/issue39622
   * python/cpython#83803

Fixes: #19260
(cherry picked from commit 6bdbed6)
ephraimbuddy pushed a commit that referenced this issue May 21, 2022
@potiuk @ephraimbuddy
Remove custom signal handling in Triggerer (#23274)
a0a2ddf 
There is a bug in CPython (fixed in March 2022 but not yet released) that
makes async.io handle SIGTERM improperly by using async unsafe
functions and hanging the triggerer receive SIGPIPE while handling
SIGTERN/SIGINT and deadlocking itself. Until the bug is handled
we should rather rely on standard handling of the signals rather than
adding our own signal handlers. Seems that even if our signal handler
just run exit(0) - it caused a race condition that led to the hanging.

More details:
   * https://bugs.python.org/issue39622
   * python/cpython#83803

Fixes: #19260
(cherry picked from commit 6bdbed6)
@ephraimbuddy ephraimbuddy mentioned this issue May 21, 2022
Closed
61 tasks
@potiuk potiuk reopened this Jun 11, 2022
@potiuk
Copy link
Member

potiuk commented Jun 11, 2022

Seems that the fix was not effective and broke Triggerer job killing: #23274 (comment)

Reverting it, until we find a better fix (or wait fo python to fix it).

@dstandish
Copy link
Contributor Author

Ok @potiuk maybe we should just reopen and merge #23271?

@potiuk
Copy link
Member

potiuk commented Jun 13, 2022

I think there is a PR opened - or maybe just merged about it.

@potiuk
Copy link
Member

potiuk commented Jul 14, 2022

I am not sure what has changed - but I cannot reproduce it any more @dstandish - if you can't as well, I propose to close it.

@SergiyKolesnikov
Copy link

@potiuk I can still reproduce it with Airflow 2.5.2. Standalone hangs after Ctrl+C.

@potiuk
Copy link
Member

potiuk commented Mar 26, 2023

Yes woudl be nice if somoene diagnoses and fixes it. Maybe you @SergiyKolesnikov ?

@SergiyKolesnikov
Copy link

I have looked at the running processes after hitting Ctrl-C. The webserver and scheduler processes were killed successfully, but the airflow triggerer process was still running. After I manually killed the triggerer process with kill PID the standalone command terminated normally.

I have also tested Airflow v2.6.2 and with this version Ctrl-C terminates the standalone command normally without any problems.

@potiuk
Copy link
Member

potiuk commented Jul 7, 2023

So I guess we should close that one.

@potiuk potiuk closed this as completed Jul 7, 2023
@DannyLee12
Copy link
Contributor

I have this issue on 2.6.3. FWIW.

@potiuk
Copy link
Member

potiuk commented Aug 22, 2023 •
edited

So I heartily encourage you to post a new issue with all the details. commenting on a closed issue does not move a needle. And likely your issue @DannyLee12 is similar but not necessarily the same.

Posting an issue with clear reproduction steps and logs showing the problem is the best way you can help those who would look at it to reproduce and maybe fix it.

On the other hand, commenting on a closed issuse that apparently have been closed without providing all those details does not help anyone in diagnosing and fixing the issue.

I heartily encourage you to open a new issue and provide as much details and reproducible steps as you can.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
area:core kind:bug This is a clearly a bug
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Remove custom signal handling in Triggerer potiuk/airflow
4 participants
@potiuk @SergiyKolesnikov @dstandish @DannyLee12