-
Notifications
You must be signed in to change notification settings - Fork 150
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Interrupt SASSession and clean up properly #592
Comments
Thanks for opening this as an issue. From the discussion, just to have my response too: Hey, yes, I can help with this. Actually, this would be an enhancement request, which would be good to be an Issue. Do you mind changing it to an Issue, or opening it as one? There's a choice to 'reference in new issue' but I think if I click that I'll be the one opening the issue. The IOM access method uses Java since there's no Python IOM client, so I use the Java IOM client and have an interface between Python and Java to do that. So, the subprocess in this case is the Java process. IOM is an asynchronous interface, so it seems like I may need to be doing something extra that I'm not, to terminate the SAS side, when you terminate the Python side abnormally. I have an idea but need to investigate. Thanks, |
I believe I was able to reproduce this. It only seems to happen if there's a running code submission in SAS that hasn't returned yet. If saspy is terminated when the workspace server is idle, the SAS process terminates when the Java process (IOM client) terminates. Also, I see that if the Java process terminates while there is a running job, once that job finished, the SAS server sees the Client had terminated and the workspace server terminates then. So, it is being cleaned up for this case, but not until whatever code was submitted finishes. Also, what version of saspy are you running, a current one (I don't think it matters, just checking). Thanks! |
Yes that’s the same behavior I’m seeing, I’m hoping we can make sure the
running code interrupts immediately and is fully cleaned up when I
cancel the run.
I’m running saspy v4.5 I believe, I looked yesterday but not at my desk at
the moment.
…On Fri, Mar 8, 2024 at 1:35 PM Tom Weber ***@***.***> wrote:
I believe I was able to reproduce this. It only seems to happen if there's
a running code submission in SAS that hasn't returned yet. If saspy is
terminated when the workspace server is idle, the SAS process terminates
when the Java process (IOM client) terminates. Also, I see that if the Java
process terminates while there is a running job, once that job finished,
the SAS server sees the Client had terminated and the workspace server
terminates then. So, it is being cleaned up for this case, but not until
whatever code was submitted finishes.
I believe that I may be able to preempt that job in SAS so that in this
case SAS will terminate when Java is terminating, same as when there's no
running job. That's what I will look into next. Would you mind seeing if
you see the same situation, where your workspace server does terminate once
the code it was running finishes, just to be sure you're seeing the same
behavior that I am?
Also, what version of saspy are you running, a current one (I don't think
it matters, just checking).
Thanks!
Tom
—
Reply to this email directly, view it on GitHub
<#592 (comment)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/AYCJ6Y7ZYQVZTTXFPOB7YBLYXIHGTAVCNFSM6AAAAABENJA72SVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTSOBWGI4TGMZUGY>
.
You are receiving this because you authored the thread.Message ID:
***@***.***>
--
This e-mail, any and all files transmitted with it, and all corresponding
e-mail messages are confidential and intended solely for the use of the
individual or entity to whom they are addressed. If you have received this
email in error, please notify the sender immediately by return e-mail. If
you are not the named addressee, you should not disseminate, distribute or
copy this e-mail. Disclosing, copying, distributing or taking any action in
reliance on the contents of this information is strictly prohibited.
|
Great thanks! Yes, I believe I will be able to do this. Trying to code it up now. Fingers crossed :) |
Yep, so much for positivity. So, the submit call (how you submit code to the server) is synchronous, not asynch. Still, I have handlers that try to close the connection when the java side takes an exception, but that's not actually making any difference. This isn't going to be what I was thinking. This is gonna talk longer than I thought, but I'm going to see what I can do! |
Hey @nbuck1234 I've been working on this and not having much luck catching the termination and trying to cancel the submission to the server. But, one of my attempts, having a separate thread, to try to catch this since the other is blocked in a synchronous wait has allowed me to implement canceling the submission proactively. This is really the implementation I've been missing in both IOM and HTTP access methods. In STDIO I implement the attention handler in SAS (interactive cancel or terminate based upon user response). In the API interfaces (IOM Workspace server and Compute API in HTTP), there is no way to do attention handling like in interactive SAS 9. These API protocols have a 'CANCEL' API call that will terminate whatever code had been submitted. This would be what you would really want in the first place. Currently if you try to interrupt, you see this (for IOM): As an example, this is what I'm seeing with a simple test case. You can see the proc print ran and we got that output, and the data step that was sleeping 100 seconds got canceled, as I didn't hit ctl-c until after the proc ran and the DS was still running. And, the server was ready to execute the next things it was sent :)
When I get this solid, I can push it to a branch for you to try out, if you can do that. This ought to provide you with what you're looking for! Thanks, |
Fantastic- let me know, and I’ll give it a shot!
…On Wed, Mar 13, 2024 at 8:42 AM Tom Weber ***@***.***> wrote:
Hey @nbuck1234 <https://github.com/nbuck1234> I've been working on this
and not having much luck catching the termination and trying to cancel the
submission to the server. But, one of my attempts, having a separate
thread, to try to catch this since the other is blocked in a synchronous
wait has allowed me to implement canceling the submission proactively. This
is really the implementation I've been missing in both IOM and HTTP access
methods. In STDIO I implement the attention handler in SAS (interactive
cancel or terminate based upon user response). In the API interfaces (IOM
Workspace server and Compute API in HTTP), there is no way to do attention
handling like in interactive SAS 9. These API protocols have a 'CANCEL' API
call that will terminate whatever code had been submitted.
This would be what you would really want in the first place. Currently if
you try to interrupt, you see this (for IOM):
SAS attention handling is not yet supported over IOM. Please enter (T) to
terminate SAS or (C) to continue.
But with this new implementation of the separate thread in Java waiting on
a separate socket to communicate with it, I've been able to get this CANCEL
interface to work. I still have more to do on it; just a prototype at the
moment. I'm seeing something strange on the windows side too, but I need
more testing on this to be sure it works in all cases. If I can get this
working production quality, it'll get you what you need and provide
something that's been missing the whole time!
As an example, this is what I'm seeing with a simple test case. You can
see the proc print ran and we got that output, and the data step that was
sleeping 100 seconds got canceled, as I didn't hit ctl-c until after the
proc ran and the DS was still running. And, the server was ready to execute
the next things it was sent :)
>>> sas.submitLST('proc print data=sashelp.class;run;data a; x=1; call sleep(100,1); run;', 'text', method='listandlog')
^CException caught!
Please enter (T) to Terminate SAS or (C) to Cancel submitted code or (W) continue to Wait.c
Canceled submitted statements
The SAS System Tuesday, March 12, 2024 03:35:00 PM 1
Obs Name Sex Age Height Weight
1 Alfred M 14 69.0 112.5
2 Alice F 13 56.5 84.0
3 Barbara F 13 65.3 98.0
4 Carol F 14 62.8 102.5
5 Henry M 14 63.5 102.5
6 James M 12 57.3 83.0
7 Jane F 12 59.8 84.5
8 Janet F 15 62.5 112.5
9 Jeffrey M 13 62.5 84.0
10 John M 12 59.0 99.5
11 Joyce F 11 51.3 50.5
12 Judy F 14 64.3 90.0
13 Louise F 12 56.3 77.0
14 Mary F 15 66.5 112.0
15 Philip M 16 72.0 150.0
16 Robert M 12 64.8 128.0
17 Ronald M 15 67.0 133.0
18 Thomas M 11 57.5 85.0
19 William M 15 66.5 112.0
7 The SAS System Tuesday, March 12, 2024 03:35:00 PM
34
35 proc print data=sashelp.class;run;
NOTE: There were 19 observations read from the data set SASHELP.CLASS.
NOTE: The PROCEDURE PRINT printed page 1.
NOTE: PROCEDURE PRINT used (Total process time):
real time 0.03 seconds
cpu time 0.01 seconds
35 ! data a; x=1; call sleep(100,1); run;
NOTE: The DATA step has been abnormally terminated.
WARNING: The data set WORK.A may be incomplete. When this step was stopped there were 0 observations and 1 variables.
NOTE: DATA statement used (Total process time):
real time 3.01 seconds
cpu time 0.00 seconds
8 The SAS System Tuesday, March 12, 2024 03:35:00 PM
36
When I get this solid, I can push it to a branch for you to try out, if
you can do that. This ought to provide you with what you're looking for!
Thanks,
Tom
—
Reply to this email directly, view it on GitHub
<#592 (comment)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/AYCJ6Y3NOAU5ZXCBB7QNRVLYYBJVDAVCNFSM6AAAAABENJA72SVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTSOJUGQZTMMRVG4>
.
You are receiving this because you were mentioned.Message ID:
***@***.***>
--
This e-mail, any and all files transmitted with it, and all corresponding
e-mail messages are confidential and intended solely for the use of the
individual or entity to whom they are addressed. If you have received this
email in error, please notify the sender immediately by return e-mail. If
you are not the named addressee, you should not disseminate, distribute or
copy this e-mail. Disclosing, copying, distributing or taking any action in
reliance on the contents of this information is strictly prohibited.
|
ok, I've pushed it to a branch named
to get it and try it out. It's behaving pretty well on both linux and wiindows. I need to do more testing and validation, of course, but it's looking pretty promising. Let me know what you see. Thanks, |
I installed the new version and tried running a script that runs a loop for
several minutes.
I'm running from jupyter notebook, when i interrupt the kernel I get the
prompt exception caught enter C to cancel. When i choose cancel it comes
back with "cancelled statements" , but actually the process appears to
continue running and I see on the sas server that the process ID remains
open indefinitely, even after the script should have finished.
[image: image.png]
I restart the kernel and run again, this time i get the expected behavior
when I choose cancel C.
[image: image.png]
Try again without restarting the kernel:
It hangs and the process remains open...
[image: image.png]
*I've found when I restart the kernel it will reliably cancel and work as
expected.*
*Could this behavior be related to Batch mode?*
*How will this work without user interaction?* It will not be feasible for
us to press "C", there won't even be a console window open when this runs in
production.
Thanks,
Nick
…On Wed, Mar 13, 2024 at 10:49 AM Tom Weber ***@***.***> wrote:
ok, I've pushed it to a branch named cancel, go figure :) You can simply
pip uninstall -y saspy
pip install ***@***.******@***.***
to get it and try it out. It's behaving pretty well on both linux and
wiindows. I need to do more testing and validation, of course, but it's
looking pretty promising.
Let me know what you see.
Thanks,
Tom
—
Reply to this email directly, view it on GitHub
<#592 (comment)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/AYCJ6Y4BOWJROCBOER3FK6TYYBRQTAVCNFSM6AAAAABENJA72SVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTSOJUGU3TMMJVGE>
.
You are receiving this because you were mentioned.Message ID:
***@***.***>
--
This e-mail, any and all files transmitted with it, and all corresponding
e-mail messages are confidential and intended solely for the use of the
individual or entity to whom they are addressed. If you have received this
email in error, please notify the sender immediately by return e-mail. If
you are not the named addressee, you should not disseminate, distribute or
copy this e-mail. Disclosing, copying, distributing or taking any action in
reliance on the contents of this information is strictly prohibited.
|
How are you running scripts in 'batch' from jupyter? I didn't try this when running scripts (file interpreter), just interactively. Running batch scripts, you're correct, there wouldn't be a way to interact with this functionality. I don't see either of the pics it looks like you tried to attach. So I can't see what that shows. The cancel, doesn't shut down the SAS process. It cancels the current code that was submitted. The process stays up to continue to be used. I haven't found a way that allows me to cancel the running code when you abnormally kill the Python process. But, even when you do that, the SAS process terminates on it's own once it finishes whatever code you were running in it after Python and Jupyter have been killed. It's just that I can't catch that and cancel the running code so it shuts down right away. If you shutdown Python normally, even while there is code running, then it will stop and shutdown right away. Ok, I tried this running as a batch script. If you just kill the python process this won't get caught and there's no chance of trying to clean up. If you set prompt=False (in config def or on SASsession) and you interrupt the process instead of killing it (kill -SIGINT pid or ctl-c, but not kill [-9] pid or restart kernel in Jupyter) then my code will catch that and do the 'cancel' implicitly for you which will get you what I expect you want. Here's my example case of this. program t2.py
batch submitting it: finding and interrupting (not KILLing) it lets it cancel what's running and finish, coming down clean.
and the result from the job submission:
And, you can get the same from hitting ctl-c if you're running the script in the foreground:
Set prompt=False for your configuration and try this out with your use cases. You won't get prompted and it will do the cancel on it's own. Tom |
Looking back at your original example code, you're trying to catch an interrupt and issue sas.endsas(). I did add the CANCEL code to endsas() as well, so that it would cancel anything that might be running and then shutdown the SAS session. I tried this case, interactively, not the same as how you had it - I don't know how you're 'killing' the process. But, I was able to run a test case, hit ctl-c and get my prompt, hit ctl-c again and get back to the python prompt, while the submitted code was still running. I then submitted sas.endsas() and the SAS server was immediately terminated; meaning the cancel worked and then the server was shut down cleanly. So, I'm kinda expecting that first case you showed, which, given the output you showed, seemed as though you did catch that and issue sas.endsas(), which should now actually cancel the running code and shut down the server. Again, set prompt=False in your config since you're gonna be running batch and prompting won't work. Then I think this should do what you want. Again, that's based on what you showed actually executing that sas.endsas() in the catch block. Tom |
Thanks for looking at this and following up Tom.
I'll do some testing today and see how this goes, and let you know later
this afternoon.
…On Thu, Mar 14, 2024 at 11:55 AM Tom Weber ***@***.***> wrote:
Looking back at your original example code, you're trying to catch an
interrupt and issue sas.endsas(). I did add the CANCEL code to endsas() as
well, so that it would cancel anything that might be running and then
shutdown the SAS session. I tried this case, interactively, not the same as
how you had it - I don't know how you're 'killing' the process. But, I was
able to run a test case, hit ctl-c and get my prompt, hit ctl-c again and
get back to the python prompt, while the submitted code was still running.
I then submitted sas.endsas() and the SAS server was immediately
terminated; meaning the cancel worked and then the server was shut down
cleanly.
So, I'm kinda expecting that first case you showed, which, given the
output you showed, seemed as though you did catch that and issue
sas.endsas(), which should now actually cancel the running code and shut
down the server. Again, set prompt=False in your config since you're gonna
be running batch and prompting won't work. Then I think this should do what
you want. Again, that's based on what you showed actually executing that
sas.endsas() in the catch block.
Tom
—
Reply to this email directly, view it on GitHub
<#592 (comment)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/AYCJ6Y3ZXOXRV6IY7ILCYSTYYHCAJAVCNFSM6AAAAABENJA72SVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTSOJXG44DINZTGM>
.
You are receiving this because you were mentioned.Message ID:
***@***.***>
--
This e-mail, any and all files transmitted with it, and all corresponding
e-mail messages are confidential and intended solely for the use of the
individual or entity to whom they are addressed. If you have received this
email in error, please notify the sender immediately by return e-mail. If
you are not the named addressee, you should not disseminate, distribute or
copy this e-mail. Disclosing, copying, distributing or taking any action in
reliance on the contents of this information is strictly prohibited.
|
Awesome, thanks! I do want to get this working as best it can! |
Tom from what I can tell in my testing, this is working as intended!
…On Thu, Mar 14, 2024 at 12:04 PM Tom Weber ***@***.***> wrote:
Awesome, thanks! I do want to get this working as best it can!
—
Reply to this email directly, view it on GitHub
<#592 (comment)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/AYCJ6Y6N34PH46JDYFAFLF3YYHDBVAVCNFSM6AAAAABENJA72SVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTSOJXHAYDGNJXGE>
.
You are receiving this because you were mentioned.Message ID:
***@***.***>
--
This e-mail, any and all files transmitted with it, and all corresponding
e-mail messages are confidential and intended solely for the use of the
individual or entity to whom they are addressed. If you have received this
email in error, please notify the sender immediately by return e-mail. If
you are not the named addressee, you should not disseminate, distribute or
copy this e-mail. Disclosing, copying, distributing or taking any action in
reliance on the contents of this information is strictly prohibited.
|
That's great! How are you 'killing' the python processes? As I mentioned, if you 'kill -9 pid' then my code can't issue the cancel or try to then shutdown, as the process just terminates out from under me. But, if you interrupt the process, that would give my code a chance to do this, which is working, so that's great. You must be terminating it in a 'normal' as opposed to 'abnormal' way so the interrupting is happening prior to process shutdown. So that's great! |
You’re right Tom. It’s a “normal” interrupt. Dagster has a “terminate run”
button that requests a normal interrupt and allows this to work.
I’ll have to think of additional separate layers to handle abnormal closing
of python processes. But I believe you’ve done all you can and I’m so happy
you did!
Thanks again,
Nick
…On Thu, Mar 14, 2024 at 2:41 PM Tom Weber ***@***.***> wrote:
That's great! How are you 'killing' the python processes? As I mentioned,
if you 'kill -9 pid' then my code can't issue the cancel or try to then
shutdown, as the process just terminates out from under me. But, if you
interrupt the process, that would give my code a chance to do this, which
is working, so that's great. You must be terminating it in a 'normal' as
opposed to 'abnormal' way so the interrupting is happening prior to process
shutdown. So that's great!
—
Reply to this email directly, view it on GitHub
<#592 (comment)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/AYCJ6Y4AUCXZZ35JTSTSZJDYYH4PLAVCNFSM6AAAAABENJA72SVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTSOJYGI4TMNZQGA>
.
You are receiving this because you were mentioned.Message ID:
***@***.***>
--
This e-mail, any and all files transmitted with it, and all corresponding
e-mail messages are confidential and intended solely for the use of the
individual or entity to whom they are addressed. If you have received this
email in error, please notify the sender immediately by return e-mail. If
you are not the named addressee, you should not disseminate, distribute or
copy this e-mail. Disclosing, copying, distributing or taking any action in
reliance on the contents of this information is strictly prohibited.
|
Cool! I'll need to spend some more time validating this out, but I ought to be able to merge it in and build a new release with it after finish testing it. For the abnormal term, there's nothing I can do, but the workspace server still goes away as soon as it sees the client terminated. It just doesn't see that while it's executing code that was submitted. As soon as that code finishes, it notices the client is gone and it terminates. So, it's not necessarily the case that there's always some long running job at the point you terminate the client. And, now for any normal case, it'll stop and shut down right away. Thanks! |
Found a couple problems that I need to address. Just better coordination and error handling between the two java threads than I have in the prototype. I've worked that out and pushed those changes. So back to testing and validating again. Just FYI. I'll let you know when I merge this in and build a new release; once I feel it's production quality. Tom |
ok, I've pushed all of this and built a new release, v5.7.0, which containers this enhancement. It's here on github, on PyPI and on Conda-forge now. Go ahead and grab it and, if you don't mind, verify it's working as expected. Assuming so, we'll close this one! Thanks! |
@nbuck1234 I typed all this in but didn't submit it. So, adding it back just so it's here for explanation. I ended up making one last change to this to limit it to the submit methods (submit(), suibmitLST(), submitLOG()), and, of course, endsas(), which was your concern. My concern was that all of the saspy methods generate and submit SAS code. Many have to generate some code, submit it and get the output from that, generate more, and submit that, ... So for these multi-sequence methods, allowing a user to just kill my code in the middle of what I'm trying to do, isn't helpful to anyone and I really don't want to have to debug those problems that aren't 'real' saspy problems. But, killing a single user submit with user code, is straight forward and a helpful feature. So, that's what I changed since last you tried this. Again, it shouldn't make any different for what you're trying to do. This is just to document it here since it's what I ended up doing. Tom |
closing this. Let me know if you need anything else! |
Discussed in #591
Originally posted by nbuck1234 March 7, 2024
I am using winiom configuration , from a remote machine to our sas server. I happen to be using this with dagster if you're curious.
My scenario is in the case where the job is kicked off, I want to be able to cancel it from the remote machine, rather than tracking it down in the SAS server.
My code looks like this :
When I kill the running job I get the following in my stdout, which is what I expect:
In this specific test case above i got the SAS process id from the session object :
When I logged into the SAS server to see if it was really ended, I see that it is in fact still running, and I must manually end it from there.
Whats the difference between the subprocess id from the stdout and the SAS process id property of the sassession?
How can i be sure that the process is cleaned up properly in this scenario?
I hope this question is useful to the community!
The text was updated successfully, but these errors were encountered: