Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Using Jupyter kernels for code execution? #79

Closed
psychemedia opened this issue Aug 5, 2019 · 6 comments

Comments

@psychemedia
Copy link

commented Aug 5, 2019

Although most commonly known as an interactive notebook serving environment, the Jupyter ecosystem may be more usefully thought of as a set of protocols that, among other things, support remote code execution in dynamically launched "kernels" which are now available for a wide variety of programming languages.

Jupyter servers such as the Jupyter kernel gateway or the Jupyter enterprise gateway provide headless access to individually launched remote kernels.

One of the issues with using different code environments for teaching (eg using Jupyter notebooks) and assessment (eg using CodeRunner) is the need to maintain consistency across code environments if students are expected to create assessed code fragments that run the same way in the test server as their own Jupyter powered coding environments.

Which got me wondering: how well defined is the API to the sandbox environment which CodeRunner executes code in and is the sandbox server (Jobe?) expected to run as service? (Indeed, can it be run as a remote service?)

If the coupling is handled via a well defined API, then could a middleware shim be used to provide sandboxed code execution either via remotely launched Jupyter kernels or via the current sandboxed environment server?

This would allow Jupyter users to make use of the same code environments for running coding labs using Jupyter UIs as well as CodeRunner based assessments, as well as extending the range of code environments supported by CodeRunner.

(As an aside, Jupyter has several candidate autograding solutions, such as nbgrader as well as integration with okpy, but nothing that I am aware of that is integrated with Moodle explicitly.)

@trampgeek

This comment has been minimized.

Copy link
Owner

commented Aug 6, 2019

Internally CodeRunner is designed to support multiple sandboxes, implemented as subclasses of the abstract class qtype_coderunner_sandbox - see sandbox.php. Sandboxes are essentially plugins to CodeRunner. Several different ones have been used over the years but the only current ones are the jobe sandbox (file jobesandbox.php) and the ideone sandbox. The latter interfaces to the Sphere On-line judge server but is now more-or-less defunct. Both of those sandboxes run as services. CodeRunner can support multiple sandboxes at the same time and questions can be configured to select a particular sandbox (if desired). By default the first available sandbox that supports the language required by the question is used.

It would certainly be feasible to use a Jupyter server as a sandbox for CodeRunner by writing another sandbox plugin. I did look into this briefly but the Jupyter protocol is much more complicated than the Jobe protocol (which is documented here). Because I don't use Jupyter myself it wasn't a priority.

I take your point that if you're teaching with Jupyter and wanted to assess with CodeRunner it would be comforting to know that jobs were being run with exactly the same language software in both cases. I'm not sure that the two would play well together, however. I don't quite see how a notebook style of programming, which (as I understand it) seems to place a heavy dependence on global state as multiple cells are run, would fit with the CodeRunner model of submitting a complete working standalone program (possibly with support files).

@psychemedia

This comment has been minimized.

Copy link
Author

commented Aug 6, 2019

@trampgeek The Jupyter ecosystem is not just about notebooks. A Jupyter server launches an environment within which you can execute code, returning the any response from the last line of code in the code chunk executed. (So yes, I can run individual code cells in a notebook UI, in any order, persisting state between code cell executions, and return a response from each code cell execution). State is maintained within the environment as long as the kernel process exists.

But that doesn't stop me running a single block of code. If I wanted to execute a single chunk of code in Jupyter, I could launch a kernel, execute that code chunk there, get the response, and kill the kernel.

The way that nbgrader (the Jupyter autograder) works is essentially to define a marking notebook that has question cells and a test/answer cell associated with each one. The notebook is stripped of test/answer cells, provided to the student, completed by them, returned, and then has the test cells added back in. If a test cell raises an exception, the student gets no marks (or a partial mark in a recent PR) for that question.

I need to dig into CodeRunner a bit more deeply, I think, to get a feel for what gets passed where, to see how a Jupyter sandbox might fit...

One thing I have noticed is that there doesn't seem to be a PHP package for managing Jupyter servers. That said, there are various javascript packages (thebelab, juniper.js) that can launch remote kernels using MyBinder, execute code there, and retrieve the result. (I mention those because they can also help you think away from Jupyter as just a notebook service...)

@trampgeek

This comment has been minimized.

Copy link
Owner

commented Aug 6, 2019

Interesting. It sounds to me like writing a Jupyter sandbox for CodeRunner isn't the right approach here.

CodeRunner has a "combinator template grader" facility that allows question authors to write just about any code they like, in any language, to grade a student's submission. The output from the grader is the student's mark and the feedback to them. And I've recently added to CodeRunner the capability of allowing students to attach files to their answer. So ...

Are you are able to write a program, preferably in Python or C/C++, that reads a student's notebook file (and any other support files such as a grader notebook if you like) and prints the mark the student should get? The program could connect to an outside Jupyter kernel in order to do so. If you can write such a program, I can show you how to turn it into a CodeRunner question without any modification to the CodeRunner or Jobe systems. More generally, you would probably want to make it into a question type that could be used as a starting point for grading arbitrary Jupyter notebooks (again using only built-in CodeRunner capabilities).

A couple of caveats: you would need to configure the Jobe server firewall to allow access to the Jupyter kernel server, and you would need to guarantee to grade a submission in under, say, 30 seconds.

If you're interested in pursuing this idea, I suggest we move the discussion to the CodeRunner question author's forum. Note that if you wish to use student attachments, you need to be running the very latest version of CodeRunner from github; the feature is still marked Experimental but I'll be using it with a class of several hundred students next week, so it should be stable Real Soon Now :-)

@psychemedia

This comment has been minimized.

Copy link
Author

commented Aug 6, 2019

I'll try to set up a test Moodle/CodeRunner and have a play with that to get a feel for how it works.

There are possibly several different ways of imagining how Jupyter integration may work.

Certainly my naive initial thought was: could we just execute a code blob in a jupyter mediated py envt and return its output to CodeRunner. If jobe and ideone both implement a similar sandbox interface/api, then why not a jupyter server extension that can speak it too?

@trampgeek

This comment has been minimized.

Copy link
Owner

commented Aug 10, 2019

The sandbox server just receives requests from CodeRunner to run a particular job and return its output. Sandboxes are stateless. You can already run Python jobs (or other languages) via Jobe, so the only benefit you'd get from a Jupyter server is guaranteed consistency with regard to the Python version, installed libraries etc.

But I'm perhaps not understanding what you're trying to achieve, so I think your idea of setting up a test Moodle/CodeRunner system and having a play is the right way to go.

I'm going to close this now as it's not really an issue with CodeRunner. If you'd like to continue the discussion would you mind posting to either the Developers' Forum or the Question Authors' Forum on coderunner.org.nz, please?

@trampgeek trampgeek closed this Aug 10, 2019

@psychemedia

This comment has been minimized.

Copy link
Author

commented Aug 12, 2019

Okay, thanks... will try to take a look.

Just for my own notes, this gist provides a simple Python example of launching a MyBinder container and executing a simple code chunk on it. The request and response would need wrapping as per the CodeRunner sandbox API and equivalent PHP modules found to handle the connection streams / sockets.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
2 participants
You can’t perform that action at this time.