Join GitHub today
GitHub is home to over 40 million developers working together to host and review code, manage projects, and build software together.Sign up
Using Jupyter kernels for code execution? #79
Although most commonly known as an interactive notebook serving environment, the Jupyter ecosystem may be more usefully thought of as a set of protocols that, among other things, support remote code execution in dynamically launched "kernels" which are now available for a wide variety of programming languages.
One of the issues with using different code environments for teaching (eg using Jupyter notebooks) and assessment (eg using CodeRunner) is the need to maintain consistency across code environments if students are expected to create assessed code fragments that run the same way in the test server as their own Jupyter powered coding environments.
Which got me wondering: how well defined is the API to the sandbox environment which CodeRunner executes code in and is the sandbox server (
If the coupling is handled via a well defined API, then could a middleware shim be used to provide sandboxed code execution either via remotely launched Jupyter kernels or via the current sandboxed environment server?
This would allow Jupyter users to make use of the same code environments for running coding labs using Jupyter UIs as well as CodeRunner based assessments, as well as extending the range of code environments supported by CodeRunner.
(As an aside, Jupyter has several candidate autograding solutions, such as
Internally CodeRunner is designed to support multiple sandboxes, implemented as subclasses of the abstract class qtype_coderunner_sandbox - see sandbox.php. Sandboxes are essentially plugins to CodeRunner. Several different ones have been used over the years but the only current ones are the jobe sandbox (file jobesandbox.php) and the ideone sandbox. The latter interfaces to the Sphere On-line judge server but is now more-or-less defunct. Both of those sandboxes run as services. CodeRunner can support multiple sandboxes at the same time and questions can be configured to select a particular sandbox (if desired). By default the first available sandbox that supports the language required by the question is used.
It would certainly be feasible to use a Jupyter server as a sandbox for CodeRunner by writing another sandbox plugin. I did look into this briefly but the Jupyter protocol is much more complicated than the Jobe protocol (which is documented here). Because I don't use Jupyter myself it wasn't a priority.
I take your point that if you're teaching with Jupyter and wanted to assess with CodeRunner it would be comforting to know that jobs were being run with exactly the same language software in both cases. I'm not sure that the two would play well together, however. I don't quite see how a notebook style of programming, which (as I understand it) seems to place a heavy dependence on global state as multiple cells are run, would fit with the CodeRunner model of submitting a complete working standalone program (possibly with support files).
@trampgeek The Jupyter ecosystem is not just about notebooks. A Jupyter server launches an environment within which you can execute code, returning the any response from the last line of code in the code chunk executed. (So yes, I can run individual code cells in a notebook UI, in any order, persisting state between code cell executions, and return a response from each code cell execution). State is maintained within the environment as long as the kernel process exists.
But that doesn't stop me running a single block of code. If I wanted to execute a single chunk of code in Jupyter, I could launch a kernel, execute that code chunk there, get the response, and kill the kernel.
The way that
I need to dig into CodeRunner a bit more deeply, I think, to get a feel for what gets passed where, to see how a Jupyter sandbox might fit...
Interesting. It sounds to me like writing a Jupyter sandbox for CodeRunner isn't the right approach here.
CodeRunner has a "combinator template grader" facility that allows question authors to write just about any code they like, in any language, to grade a student's submission. The output from the grader is the student's mark and the feedback to them. And I've recently added to CodeRunner the capability of allowing students to attach files to their answer. So ...
Are you are able to write a program, preferably in Python or C/C++, that reads a student's notebook file (and any other support files such as a grader notebook if you like) and prints the mark the student should get? The program could connect to an outside Jupyter kernel in order to do so. If you can write such a program, I can show you how to turn it into a CodeRunner question without any modification to the CodeRunner or Jobe systems. More generally, you would probably want to make it into a question type that could be used as a starting point for grading arbitrary Jupyter notebooks (again using only built-in CodeRunner capabilities).
A couple of caveats: you would need to configure the Jobe server firewall to allow access to the Jupyter kernel server, and you would need to guarantee to grade a submission in under, say, 30 seconds.
If you're interested in pursuing this idea, I suggest we move the discussion to the CodeRunner question author's forum. Note that if you wish to use student attachments, you need to be running the very latest version of CodeRunner from github; the feature is still marked Experimental but I'll be using it with a class of several hundred students next week, so it should be stable Real Soon Now :-)
I'll try to set up a test Moodle/CodeRunner and have a play with that to get a feel for how it works.
There are possibly several different ways of imagining how Jupyter integration may work.
Certainly my naive initial thought was: could we just execute a code blob in a jupyter mediated py envt and return its output to CodeRunner. If jobe and ideone both implement a similar sandbox interface/api, then why not a jupyter server extension that can speak it too?
The sandbox server just receives requests from CodeRunner to run a particular job and return its output. Sandboxes are stateless. You can already run Python jobs (or other languages) via Jobe, so the only benefit you'd get from a Jupyter server is guaranteed consistency with regard to the Python version, installed libraries etc.
But I'm perhaps not understanding what you're trying to achieve, so I think your idea of setting up a test Moodle/CodeRunner system and having a play is the right way to go.
I'm going to close this now as it's not really an issue with CodeRunner. If you'd like to continue the discussion would you mind posting to either the Developers' Forum or the Question Authors' Forum on coderunner.org.nz, please?
Okay, thanks... will try to take a look.
Just for my own notes, this gist provides a simple Python example of launching a MyBinder container and executing a simple code chunk on it. The request and response would need wrapping as per the CodeRunner sandbox API and equivalent PHP modules found to handle the connection streams / sockets.