can not make calls to Python on a separate thread

[keowang]

The following (copies from Example) worked fine
using (Py.GIL())
{
dynamic test = Py.Import("programs.test_program");
Console.WriteLine(test.func(args));
}

However, the below ends up hanging forever even though it's the same one line program as above. Without the ability to run on a separate thread how can I kill an execution if it takes too long. We deal with massive amount of python scripts at runtime it's gonna be a pain to modify them to terminated the costly computation.

using (Py.GIL())
{
      dynamic test = Py.Import("programs.test_program");

      Thread t = new Thread(delegate () {
              Console.WriteLine(test.func(args));
    });
}

[den-run-ai]

what happens if thread is launched outside of Py.GIL() block?

[keowang]

This is what happens:

An unhandled exception of type 'System.AccessViolationException' occurred in Python.Runtime.dll

Additional information: Attempted to read or write protected memory. This is often an indication that other memory is corrupt.

[keowang]

Try to do something inside the python script by following https://stackoverflow.com/questions/492519/timeout-on-a-function-call. It's running fine in python but once again pythonnet hangs forever. I wonder what is the solution.

import sys
import threading
import _thread as thread 

def quit_function(fn_name):
    print('{0} took too long'.format(fn_name), file=sys.stderr)
    sys.stderr.flush()
    thread.interrupt_main() # raises KeyboardInterrupt

def exit_after(s):
    def outer(fn):
        def inner(*args, **kwargs):
            timer = threading.Timer(s, quit_function, args=[fn.__name__])
            timer.start()
            try:
                result = fn(*args, **kwargs)
            finally:
                timer.cancel()
            return result
        return inner
    return outer

@exit_after(.00001)
def testWrapper():
    while True : 
        i = 0

[keowang]

Hi can I get an update, It's a blocker on our end. I would appreciate if you can advice. Thanks a lot!

[den-run-ai]

The reason why interrupt does not work in pythonnet is due to this issue and corresponding WIP pull request:

#449
#450

[den-run-ai]

The code in pull request is actually OK, but the tests for it are not good.

[keowang]

so what's the fix?

[den-run-ai]

The fix is in the pull request, just look at the diff:

https://github.com/pythonnet/pythonnet/pull/450/files

In short pythonnet should call Py_InitializeEx() instead of Py_Initialize().

[keowang]

Thanks for the pointers. Does this mean I have to get the source (after the fix) and then build it myself is there a binary I can just directly embed in my .Net application.

[keowang]

Or if this is easier. Would you mind advising me how to create a infinity loop detection when I am calling python code.

[den-run-ai]

@keowang I tried with master branch and I don't see any issue:

scriptcs (ctrl-c to exit or :help for help)

> #r "C:\Python\Python35_64b\Lib\site-packages\Python.Runtime.dll"
> using Python.Runtime;
> dynamic intr;
> dynamic sys;
> using (Py.GIL()) {sys=Py.Import("sys");}
> sys.path.ToString()
['C:\\Python\\Python35_64b\\python35.zip', 'C:\\Python\\Python35_64b\\Lib', 'C:\\Python\\Python35_64b\\DLLs', 'C:\\ProgramData\\chocolatey\\lib\\ScriptCs\\tools', 'C:\\Python\\Python35_64b', 'C:\\Python\\Python35_64b\\lib\\site-packages', 'C:\\Python\\Python35_64b\\lib\\site-packages\\win32', 'C:\\Python\\Python35_64b\\lib\\site-packages\\win32\\lib', 'C:\\Python\\Python35_64b\\lib\\site-packages\\Pythonwin', 'C:\\Windows\\Microsoft.NET\\Framework64\\v4.0.30319\\']
> sys.path.append(@"C:\Users\denis\Dropbox\coding\pythonnet\issue511");
> using (Py.GIL()) {intr=Py.Import("interrupt_main");}
> intr.testWrapper()
start inf loop
testWrapper took too long
KeyboardInterrupt :

[den-run-ai]

@rmadsen-ks do you see any issue?

[keowang]

@denfromufa

This is my code running the python script above and it hangs every time.
using (Py.GIL())
{
dynamic testModule = Py.Import("programs.test_program");
output = testModule.testWrapper();
}

Pythonnet installed using following wheel:
pythonnet-2.3.0-cp36-cp36m-win_amd64.whl

Python:3.6 64-bit

Would you please let me know if you can run this code

[den-run-ai]

@keowang ok, i did some testing with pythonnet 2.3.0, but I still don't see your exact issue here. Show me the problem again. Here is my testing:

//#r "C:\Python\Python35_64b\Lib\site-packages\Python.Runtime.dll"
using Python.Runtime;
using System;

namespace interrupt
{
    class Program
    {
        static void Main(string[] args)
        {
            dynamic intr;
            dynamic sys;
            PythonEngine.Initialize(true); //, true); second argument if for @rmadsen-ks branch
            using (Py.GIL()) {sys=Py.Import("sys");}
            sys.path.append(System.AppDomain.CurrentDomain.BaseDirectory);
            using (Py.GIL()) 
            {
                intr=Py.Import("interrupt_main");
            }
            using (Py.GIL()) 
            {
                try 
                {
                	intr.testWrapper();
            	}
            	catch {
            		Console.WriteLine("some exception here");
            	}
            }
        }
    }
}

import sys
import threading
import _thread as thread 

def quit_function(fn_name):
    print('{0} took too long'.format(fn_name), file=sys.stderr)
    sys.stderr.flush()
    thread.interrupt_main() # raises KeyboardInterrupt

def exit_after(s):
    def outer(fn):
        def inner(*args, **kwargs):
            timer = threading.Timer(s, quit_function, args=[fn.__name__])
            timer.start()
            try:
                result = fn(*args, **kwargs)
            finally:
                timer.cancel()
            return result
        return inner
    return outer

@exit_after(0.1)
def testWrapper():
    print('start inf loop')
    while True : 
        i = 0

if __name__=="__main__":
    print('before threads launched')
    try:
        testWrapper()
    except KeyboardInterrupt:
        print("forced to interrupt")

Then if I call from python:

python interrupt_main.py
before threads launched
start inf loop
testWrapper took too long
forced to interrupt

If I call from C# compiled app:

interrupt.exe
start inf loop
testWrapper took too long
some exception here

[keowang]

@denfromufa I appreciate your response. The issues is you are running the binary and surprisingly it worked. However, in Visual Studio debug mode intr.testWrapper() will hang every time. There is no way we can get around this by skipping the debug mode and directly monitoring the program behavior via logs which is hugely inconvenient. Would you mind looking into that see if there is some solution that we can have the consistent behavior in both debug and release mode. Thanks

[den-run-ai]

By debug mode do you mean debug build of the binary and pythonnet or debugging (stepping) through this code in VS?

…

On Sun, Jul 16, 2017, 1:27 PM keowang ***@***.***> wrote: @denfromufa <https://github.com/denfromufa> I appreciate your response. The issues is you are running the binary and surprisingly it worked. However, in Visual Studio debug mode intr.testWrapper() will hang every time. There is no way we can get around this by skipping the debug mode and directly monitoring the program behavior via logs which is hugely inconvenient. Would you mind looking into that see if there is some solution that we can have the consistent behavior in both debug and release mode. Thanks — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#511 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AHgZ5frzEFYfVN0sXlWxnx7DoiX_HRcHks5sOlX3gaJpZM4OXua6> .

[keowang]

stepping through the code in VS

[den-run-ai]

Do you debug with attach mode or start the debugger with F5? Is the debugger engine only "Managed"?

[den-run-ai]

Also what Exception Settings in VS have you enabled for debugging session?

…

On Sun, Jul 16, 2017, 2:15 PM keowang ***@***.***> wrote: stepping through the code in VS — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#511 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AHgZ5UqrevMLQ_MVJHFLYPjWPI-7JWq6ks5sOmFtgaJpZM4OXua6> .

[keowang]

I hit F5 and it's a managed only. The exception setting is by default you want me to list them all? Or maybe you can let me know what exception I should have ticked that I did not

[keowang]

@denfromufa Hello, can I get an update?

[den-run-ai]

can you show your exceptions settings:

https://docs.microsoft.com/en-us/visualstudio/debugger/managing-exceptions-with-the-debugger

[den-run-ai]

i can debug in vs 2017 with no problems:

[keowang]

[keowang]

That's all

[keowang]

[keowang]

This is how it hangs

[keowang]

Can I please ask what's your exception setting? Thanks!

[den-run-ai]

you can see my exception settings in my screenshot

[keowang]

yes I tried and it's the same thing

[den-run-ai]

did you try break all: ctrl+alt+break

[keowang]

press the key manually?

[den-run-ai]

no, break in the debugger

[keowang]

how do I do that?

[den-run-ai]

debug -> break all

[keowang]

[keowang]

I do not see it here

[keowang]

Hey I fixed it by enabling the native code which makes the debugging very slow. Can I do it without the native code?

[den-run-ai]

"break all" has to be triggered during debugging. i do not have native code debugger enabled. please contact VS team @ msft about debugger issues.