Returned value is set to null when code contains multiple lines

[hadim]

Consider the following code :

package org.scijava.jupyterkernel.sandbox;

import javax.script.ScriptEngine;
import javax.script.ScriptException;
import org.scijava.Context;
import org.scijava.script.ScriptLanguage;
import org.scijava.script.ScriptService;

/**
 *
 * @author Hadrien Mary
 */
public class TestScriptEngine {

    public static void main(String[] args) throws ScriptException {
        // Only for testing purpose

        Context context = new Context();
        ScriptService scriptService = context.getService(ScriptService.class);
        ScriptLanguage scriptLanguage = scriptService.getLanguageByName("python");
        ScriptEngine engine = scriptLanguage.getScriptEngine();

        Object result = engine.eval("p=999\n555");
        System.out.println(result);

        scriptService = context.getService(ScriptService.class);
        scriptLanguage = scriptService.getLanguageByName("python");
        engine = scriptLanguage.getScriptEngine();
        
        result = engine.eval("555");
        System.out.println(result);

        context.dispose();
    }

}

It returns:

null
555

While it should returns :

555
555

[hadim]

It seems to be a deep bug :-(

https://github.com/jythontools/jython/issues/72

[hadim]

Introduction

In Jupyter when an executed line ends by a variable such as:

p = 999
p

The intended behavior is to return the variable (p here). The Jupyter client can then display it as a special Out cell (different from the UI dedicated to print() statement. See the screenshot below:

The Scijava Groovy scripting language follows this behavior while the Scijava Python scripting language does not and returns null.

---

Possible solution

This bug is very annoying when using Jupyter so I have opened an issue on the Jython repo: https://github.com/jythontools/jython/issues/72.

I don't think the Jython devs are going to add this feature but one of the dev (thanks to him) provided me a hack that works nicely:

public class TestJython {

    /**
     * @param args the command line arguments
     */
    public static void main(String[] args) throws PyException {

        PythonInterpreter interp = new PythonInterpreter() {
            @Override
            public PyObject eval(String s) {
                int splt = s.lastIndexOf('\n');
                super.exec(s.substring(0, splt + 1));
                return super.eval(s.substring(splt + 1));
            }
        };

        Object result = interp.eval("p=999\np");
        
        // The next line prints `999` which is not the case with the default PythonInterpreter
        System.out.println(result);

    }
}

I would like to integrate this to Scijava but I need your input @ctrueden for that:

• First, are you ok with this fix?
• PythonInterpreter needs to be overridden but it's one layer deep in the code: It's used by PyScriptEngine and so we would also need to override it. I could do that in the Scijava Jython repo.
• Last, I don't know how to tell the Script framework to use the overridden PyScriptEngine. Is there is a kind of priority to choose which ScriptEngine is picked up?

[ctrueden]

I did some digging with a debug session into Jython. And I tried a few hacky things (e.g. wrapping the script in a Python function). It does seem like a tough nut to crack. I guess this hack is the best we can hope for, so I'm fine with adding it to scripting-jython.

However, looking at the Jython code, it's very tricky to slip in there. The PyScriptEngine has a private final PythonInterpreter interp. So I think it will work better to override the PyScriptEngine's eval(String script, ScriptContext context) and eval(Reader reader, ScriptContext context) to call the PyInterpreter as above. I will try soon but bus time is over for today.

[hadim]

Well, gonna be tricky since PyScriptEngine constructor is private. I can't extend it (unless I am missing a Java trick here).

[ctrueden]

Actually, the constructor is package-private, and the jython JAR is not sealed, so we can implement the subclass in package org.python.jsr223. Very dirty, but it works. Better would be to make a request upstream to make the PyScriptEngine constructor public instead, of course. There are several other changes that should ideally be made in PyScriptEngine to make it subclassing-friendly.

I started trying to make the changes here. I added a new unit test JythonTest#testReturnValues which fails with the old JythonScriptLanguage and passes with the updated one using the new JythonScriptEngine subclass. But then the existing JythonTest#testEval starts erroring out. I did not have time to figure out why yet. Feel free to try it if you want. Otherwise I might let this sit for a while, since I want to focus on the SciJava Common #@ preprocessor stuff.

[hadim]

I had a quick look but I can't figure out what's wrong...